19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
dagster-de
dagster-cube
gigs-freelance
dagster-wandb
dagster-machine-learning
dagster-bigquery
tools
dagster-dask
dagster-snowflake
dagster-releases
data-platform-design
events
dagster-serverless
dagster-noteable
introductions
dagster-airflow
dagster-cloud
random
dagster-dbt
faq-read-me-before-posting
dagster-airbyte
dagster-spatial
dagster-support
announcements
github-discussions
豆瓣酱帮
dagster-feedback
dagstereo
dagster-ecs
dagster-kubernetes
jobs
dagster-showcase
Title
j
Jun Ying
02/15/2023, 7:32 PMHi everyone,
I have a quick question. Can dagster work with Flink?
Thank you@@
o
owen
02/15/2023, 8:13 PMhi @Jun Ying! the quick answer is that dagster does not have any pre-built components/libraries for interacting with Flink, and is generally not designed for streaming use cases, so you might have trouble integrating Flink into a dagster project if you want to use it in a streaming context.
j
Jun Ying
02/15/2023, 8:35 PMi see, how about basic Flink operator like submit@@? I see there is one for airflow here: https://github.com/apache/airflow/issues/9134 but i can not find information for dagster…
o
owen
02/15/2023, 11:08 PMhi @Jun Ying! there's no prebuilt integration for that functionality, but a similar operation could be implemented in dagster
it'd just be a matter of defining your own custom op that functions similarly to the airflow operator (but yeah unfortunately no out-of-the-box solution here)
j
Jun Ying
02/15/2023, 11:58 PMi see, Thank you!
h
Hetal Movaliya
04/26/2023, 2:58 PM@owen Do you think Dagster will be able to stream data from Azure Event Hub or Apache Kafka though it does not have any integrated library currently but by using Azure event hub python client library along with sensor in Dagster?