I have a quick question. Can dagster work with Flink?
02/15/2023, 8:13 PM
hi @Jun Ying! the quick answer is that dagster does not have any pre-built components/libraries for interacting with Flink, and is generally not designed for streaming use cases, so you might have trouble integrating Flink into a dagster project if you want to use it in a streaming context.
hi @Jun Ying! there's no prebuilt integration for that functionality, but a similar operation could be implemented in dagster
it'd just be a matter of defining your own custom op that functions similarly to the airflow operator (but yeah unfortunately no out-of-the-box solution here)
02/15/2023, 11:58 PM
i see, Thank you!
04/26/2023, 2:58 PM
@owen Do you think Dagster will be able to stream data from Azure Event Hub or Apache Kafka though it does not have any integrated library currently but by using Azure event hub python client library along with sensor in Dagster?