I'm new to Dagster and just managed to put a simple pipeline in place. I'm looking to use it on one of our local VM. It works brilliantly locally. I'm facing some internal pushback in my org regarding the use of Dagster.
The main counterargument is we(opposition) will be moving to Azure, and we don't see how this orchestrator complements, is better than azure pipelines or any other orchestrator tool in their ecosystem. I've mentioned dagster ability to translate pipelines as code (my team is very keen on this), simple concepts used as building blocks and ease of deployment.
Are there any other points I should refer to? How Dagster complements Azure and areas where it shines deploying python code. Any help is much appreciated. Thank you in advance.
10/26/2021, 5:36 PM
Hi @FranciscoGS. If your org is looking to use Azure native products, the closest thing to Dagster is Azure Data Factory. Data Factory is Azure's code-free ETL service that I think is a big pain in the butt to use. You've already mentioned some pros to Dagster over it which is Dagster's ability to both use python and document pipelines as code.
I'd recommend your org build out a Dagster version and an Azure Data Factory proof of concept using 1 small use case. That will help them see the pros and cons to both. That will also involve the whole team in the decision making process when they choose Dagster 😉
10/27/2021, 8:44 AM
Thank you Marcos for the useful pointers and very sensible approach. I will suggest the proof of concept as it seems the best path to make a decision.