https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jun Ying

02/15/2023, 7:32 PM
Hi everyone, I have a quick question. Can dagster work with Flink? Thank you@@
o

owen

02/15/2023, 8:13 PM
hi @Jun Ying! the quick answer is that dagster does not have any pre-built components/libraries for interacting with Flink, and is generally not designed for streaming use cases, so you might have trouble integrating Flink into a dagster project if you want to use it in a streaming context.
j

Jun Ying

02/15/2023, 8:35 PM
i see, how about basic Flink operator like submit@@? I see there is one for airflow here: https://github.com/apache/airflow/issues/9134 but i can not find information for dagster…
o

owen

02/15/2023, 11:08 PM
hi @Jun Ying! there's no prebuilt integration for that functionality, but a similar operation could be implemented in dagster
it'd just be a matter of defining your own custom op that functions similarly to the airflow operator (but yeah unfortunately no out-of-the-box solution here)
j

Jun Ying

02/15/2023, 11:58 PM
i see, Thank you!
h

Hetal Movaliya

04/26/2023, 2:58 PM
@owen Do you think Dagster will be able to stream data from Azure Event Hub or Apache Kafka though it does not have any integrated library currently but by using Azure event hub python client library along with sensor in Dagster?
2 Views