19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi everyone,
I have a quick question. Can dagster ...
# ask-communityj
Jun Ying
02/15/2023, 7:32 PMHi everyone,
I have a quick question. Can dagster work with Flink?
Thank you@@
o
owen
02/15/2023, 8:13 PMhi @Jun Ying! the quick answer is that dagster does not have any pre-built components/libraries for interacting with Flink, and is generally not designed for streaming use cases, so you might have trouble integrating Flink into a dagster project if you want to use it in a streaming context.
j
Jun Ying
02/15/2023, 8:35 PMi see, how about basic Flink operator like submit@@? I see there is one for airflow here: https://github.com/apache/airflow/issues/9134 but i can not find information for dagster…
o
owen
02/15/2023, 11:08 PMhi @Jun Ying! there's no prebuilt integration for that functionality, but a similar operation could be implemented in dagster
owen
02/15/2023, 11:09 PMit'd just be a matter of defining your own custom op that functions similarly to the airflow operator (but yeah unfortunately no out-of-the-box solution here)
j
Jun Ying
02/15/2023, 11:58 PMi see, Thank you!
h
Hetal Movaliya
04/26/2023, 2:58 PM@owen Do you think Dagster will be able to stream data from Azure Event Hub or Apache Kafka though it does not have any integrated library currently but by using Azure event hub python client library along with sensor in Dagster?
5 Views