Run solids / workflow /whatever in a controlled python environment Hello, I still try to figure out all pros and cons using dagster. I have this question. Is dagster in production able to "conternize" easily at one of those levels: • solid • workflow • other ? My use case is that I want to run some ML models (any frameworks) into dagster as solid. Since dependencies can be difficult if you have to install multiple environment, what is the capacity of dagster for this ? I can find workaround but if it is already there... 🙂

Hi Thomas, containerizing both the solids and the pipelines are absolutely possible. Every dagster pipeline is always run in its own process, and you can use repositories to have different pipelines that use completely different python environments within the same Dagster deployment. There are also several ways to run each solid in its own container/process, depending on where you're deploying Dagster (e.g. in Kubernetes or not). One thing we don't support yet is running different solids within a single pipeline in different images/python environments yet, but we're doing some work now with the goal of enabling that.

It can be / will be based on Kuberenetes at the end. Should I look here for repo ? Can you point me to some exemples /vids ? https://docs.dagster.io/tutorial/advanced_repositories Thanks for your help 🙂

I'd suggest looking through our kubernetes docs here: https://docs.dagster.io/deploying/kubernetes Those demonstrate per-solid isolation.

Thanks for your suggestion. It's hard for me to see where it will be difficult. Can we do this at Docker/container level ?

Assuming you’re referring to running different solids in different python environments within the same pipeline - The difficulty comes from the fact that the system requires being able to load the pipeline definition and each of its solid definitions in code in the same process, which might not be possible if different solids in the pipeline require different imports or Python environments. This is a requirement that we’re planning on relaxing in the future in order to enable use cases like this one. Cc @alex who may have more to add here