04/13/2022, 10:21 AM
My two cents on this : Prefect provides tooling for running tasks on a schedule in a large variety of setups. It has few abstractions, which make it more like a swiss army knife for orchestration. Dagster on the other hand focusses more on data engineering issues like data integrity, testing, idempotency, etc. It has abstractions for solving data problems such as partitions, backfill, assets and resources (which do not exist as such in Prefect, to my knowledge). Both are great tools, and you can probably achieve the same result with both, but Dagster will probably be more suited for managing complex data pipelines and Prefect is probably more suited for simple, general purpose task management.
🎉 2