https://dagster.io/ logo
c

Caleb

11/16/2019, 3:33 PM
Hi all, I am new here and wondering what is the difference between Dagster and Prefect (https://docs.prefect.io/)?
👍 1
a

alex

11/19/2019, 4:22 PM
Unfortunately I’m not aware of any detailed comparison between the two systems. There are episodes of data engineering podcast from each of the founders so that could give you a high level sense of where each is coming from: https://www.dataengineeringpodcast.com/prefect-workflow-engine-episode-86/ https://www.dataengineeringpodcast.com/dagster-data-applications-episode-104/
s

schrockn

11/19/2019, 6:52 PM
Some additional commentary: currently as constituted they are definitely overlapping systems but with some different philosophies. I think they will diverge more as time goes forward, actually. Germane differences off the top of my head: 1) Dagster pipelines are more structured and constrained. This allows us have a lot of additional features (a type system, a config management system, a system-managed context object that flows through the compute, among other things). By constrast Prefect pipelines are more minimal and dynamic. Another way of framing this difference is that dagster is very interested in what the computations are doing rather than only how they are doing it. We consider ourselves the application layer for data applications (rich metadata, type system, structured events with semantic meaning etc), whereas prefect frames their software in terms of “negative” and “positive” engineering. This negative/positive framing is more exclusively about the “how” of pipelines: retries, operational matters etc. 2) We are 100% open source. Our GUIs and all associated tools are open source. There will be an eventual commercial product, but what is open source now will forever and always be open source. Prefect has a hosted, commercial cloud product. In order to get UIs and monitoring tools you must be a customer of that cloud product. That all being said, I have not used Prefect or done an in-depth comparison. I actually think it is more appropriate to have neutral community members do a deeper comparison and a “bakeoff”. I can only comment as to the top level things.
2 Views