Hey all, I’ll soon have to do an objective comparison between Dagster, Airflow, and Prefect (in other words at least 3 options) before bringing in an orchestration framework into a new org, but am obviously quite biased (although I’d guess most people here will similarly be). Has anyone done something similar in the past and could share a few ideas? It’s clear where my preference lies, but I would like the comparison to be objective.

Had to do one in the past. Was honest and put previous work with Dagster as a plus in the comparison and gave credit to other orchestrators in terms of how easy they were to spin up (this was before the Dagster Cloud offering). Pretty much the argument came down to, Dagster may take a little longer to get going but the investment was worth it.

Yes, I’ll have no choice but to wear my bias on my sleeve given my involvement and previous work, but I still want the comparison to be fair, which would be very difficult given I never worked with Prefect beyond a quick PoC and Airflow beyond a few months (some of the worst months of my life tbh)

Let me know if we can support from our end. I would love to see your candid conclusions!

I've recently done a comparison between these three, although I only had to justify it to myself so I didn't do anything particularly formal. My overall conclusion was that the way dagster handled software-defined assets was a huge benefit over previous work we had done with airflow. The main points of differentiation: • airflow pushed as towards a workflow that pushed everything into a single task, so we really weren't using the DAG capabilities. This mostly stemmed from our need to pass large amounts of data between processes, which airflow doesn't handle - hence we would have to either handle intermediate storage ourselves or end up with behemoth tasks (we chose the latter). • resource usage - dagster easily used less resources than airflow - granted we had done only minimal tuning to airflow, but memory usage was a bit of an issue • there is a much better test workflow from within dagster - we found testing/debugging significantly harder with airflow • I only delved minimally into prefect, but comparing prefect and dagster, the way that dagster managed partitioning was more aligned with what we were looking for, and more capable. • the only point that dagster lost out on was the name - in Australia (and growing up on a sheep farm), the word dag has a slightly unsavoury meaning 😄

@Vinnie; We are just finishing up the exact same process -- good luck on your journey. Might I recommend reaching out to Dagster's sales team. They will likely assign you an engineer and give you trial credits. This will make your experience much better as there is a lot of mis-information out there.

@Steve Jackson I’ve been in contact with them already, but we’ll likely be going full OSS rather than cloud. I’m confident I can assess the pros and cons for Dagster since I’ve been working with it for a while already, it’s more about making this comparison fair to other tools that I’m less familiar with (and, to be completely honest, fairly biased against)

@Vinnie -- Gotcha, good luck on your journey.