19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
s
Srini R
02/02/2022, 9:58 AMHi, Currently we are running airflow and looking forward to migrate to dagster.. evaluation of options going on right now.
What's the best possible route to migrate airflow dags to dagster?
Is there any tool available? Any document available to follow best practices..?
Adding few more questions..
If I want multi tenant architecture for dagster, by setting up clusters in different regions.. if I do that, would I be able to establish a connectivity from one region to another region in order to start a job..
For example: two setup of dagster. One at Tokyo and one at Phillipines. Once the job is completed in Tokyo, Phillipines job must be triggered.. but these are not in same clusters. Different deployments and different platform.. but both use dagster.. would it be possible
d
daniel
02/02/2022, 3:52 PMHi Srini- there are some docs here on integrating dagster with airflow: https://docs.dagster.io/integrations/airflow#using-dagster-with-airflow - don't hesitate to ask any followups if they don't answer your questions though
Dagster sensors would be a good fit for your second question: https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#sensors
You can kick off runs using sensors based on any external state change (which could include a job that was completed in another Dagster deployment)
If you're specifically interested in having multiple independent Dagster deployments, that could also be a good fit for our cloud offering, which was designed with that use case in mind and alleviating the hosting burden there - happy to share more details about that if that sounds interesting
s
Srini R
02/02/2022, 4:03 PMHi Daniel, thanks for providing such good info.. having multiple deployment independent installations and establishing a connectivity to pass messages is a key requirements for us.. would like to more about how to go on in that area
d
daniel
02/02/2022, 4:04 PMWe don't have a way to pass messages directly per se, but sensors would allow you to have one deployemtn polling for state changes in another deployment - do the examples in that sensors doc around things like asset sensors look like a good match for what you have in mind?
s
Srini R
02/02/2022, 4:05 PMMigration from airflow to dagster- i will check out the links and come back on it. Thanks..
Is dagster cloud offering would be the way to go for multiple cloud deployment..? Then I would like to know more about it. Can you pls assist on it..?
Also interested in knowing, what kind of efforts required in doing the multiple dagster cloud deployment? From source code movement and devops perspective
Sorry to bother with so many questions.. it's just we are exploring and observed that we have many unknown which we need to fill up..
d
daniel
02/02/2022, 4:19 PMNo problem. There's more info on the cloud offering here as well as docs and a signup for early access: https://dagster.io/cloud
s
Srini R
02/02/2022, 4:20 PMThanks much.. will check it