Noah K

01/30/2021, 11:17 AM
But it's adding a lot of structural complexity that if you don't already know how to use, learning both Dagster and K8s at the same time is probably counterproductive
💯 1


01/31/2021, 10:25 AM
Completely agree here. If you don't already know K8s; just focus on getting single node Dagster to work (its much simpler to debug). Only add in K8s when you need multiple nodes.
I'd go one step further and ask whether you really need to use Docker on a single machine. Are your pipelines sufficiently different that they need more dependancy isolation that separate Python virtualenvs can provide?

Noah K

01/31/2021, 10:30 AM
@mrdavidlaing Installing things via docker is a lot easier than sorting out Python versions, pip versions, etc 🙂
👍 1
But yeah that assumes you don't need any fancy libraries for your tasks