Hi all - so i've gone through the all the current airflow, prefect and now dagster documentation as the first part of my evaluation for a large impl and honestly.... so far from what I'm seeing dagster is simply next level. The facilities around treating workflows as an application with proper typing, inputs/outputs, the iomanager abstraction, assets, events/metadata emitting etc.... wow. This has me excited. That said I'm going to have to start doing some prototyping on all 3 of these products.... one question i can see coming from above is: "well dagster hasn't been around as long as the others, smaller community, less mature, will this be around in 3 years or dead?..... etc etc" (in particular to prefect). Do you guys have any articles or comparisons w/ dagster/prefect? I can gleem the differences just reading the docs, but would be good to get your perspective. Also am interested in your cloud offerring/maturity pricing etc.

We don’t have a comparison with Prefect handy right now, but this comparison with Airflow could be helpful. https://dagster.io/blog/dagster-airflow

yeah i've seen that one, honestly im about to write off airflow

Got it. cc @max for information on Dagster Cloud

We compared the too a lot internally and the main points for us when choosing Dagster were (1) composability (the ability to wire graphs to graphs for example, easily create materialization events, great local UI, etc. (2) better documentation (3) celery integration On the composability point though, we're finding that Airflow does one thing better and it's composing pipelines programatically. Dagster has the configuration API and GraphDefinition but both are a bit awkward IMO.

meaning hard to use (not intuitive) or buggy?

I did a little comparing to Prefect, though I haven't used Prefect in production. There was definitely a lot of similarity, but here's my "cons" list for Prefect: * simple things are not laid out clearly in docs (UI/server separate from main pipeline run), so getting a local pipeline running with a UI is not simple * local UI requires docker + docker-compose, again complicating local development a bit * prefect.context does not error on bad lookups / tutorial uses string-key lookup / no friendly str, so that and other things make for some bad software patterns / more difficult debugging * flashy UI, slow and hard to navigate (big cards for single-line statuses; logs not obvious; nav bar at bottom; slow page transitions and log search) * no external triggers (just schedules / per-task run conditions)

I learned things about Kubernetes from reading the Dagster source code! It's really top-notch.