The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

But it's adding a lot of structural complexity that if you don't already know how to use, learning _both_ Dagster and K8s at the same time is probably counterproductive

Completely agree here.  If you don't already know K8s; just focus on getting single node Dagster to work (its much simpler to debug).

Only add in K8s when you need multiple nodes.

I'd go one step further and ask whether you really need to use Docker on a single machine.

Are your pipelines sufficiently different that they need more dependancy isolation that separate Python virtualenvs can provide?

<@U01HZ13NR2L> Installing things via docker is a lot easier than sorting out Python versions, pip versions, etc :slightly_smiling_face:

But yeah that assumes you don't need any fancy libraries for your tasks