Hi everyone and thanks for the amazing work you've done here with Dagster! I'm wondering if anyone thought of running solids as independent docker images? Like, for example, Gitlab runners do. That would give an ability to version control solids code and execution environment, dependencies. I'm currently playing with celery deployment and just getting started so would appreciate if someone could guide me where can I hack task execution and try it out And another thing that I'm really missing is a built-in abstraction for steps caching with your materializations: so that a solid could skip execution if it has already processed the same inputs and has a ready materialization for them. That would require to make materialization names based on inputs hash, not just run_id uuid, I've already made this for my homemade pipelines before I found dagster and it was quite usefull.

We agree that those two features would be useful and are building towards them. We are currently working on the foundational pieces, so it will still be a bit before they are ready (order of months not weeks)

May I be of any help? Is there a place to contribute? And still could you please point me if there is an easy way to quick-patch dagster and try docker runs for celery workers?

I think @nate has a prototype diff (doing DinD on K8s) you could reference. You could take that and fork the dagster-celery engine to do docker run, using likely

tags

on the solid to communicate which image to use. i

Oh, great, I'll look into that, thank you

hey @matas - are you deploying on k8s? I’ve also been working on https://dagster.phacility.com/D2609 which does the same thing, but launches step execution natively in k8s Jobs vs. the DinD approach in D2409. Happy to discuss both further!

No, we're not using k8s, I was looking for a DinD approach more. Thanks for your work, i'll try to launch your branch and give you a feedback on this

ok great. yes please keep me posted!