https://dagster.io/ logo
f

Fran Sanchez

08/14/2020, 8:08 PM
I'm very glad to see see finally the support of docker containers for user code, thanks for this! One question related to this feature, is there any way to define solids in different containers and then have a pipeline using them? Put this in a different way, can I build a pipeline that uses solids from different docker images?
a

alex

08/14/2020, 8:16 PM
not yet
it’ll be a while before we get that one ironed out, at least a general solution
there may be some clever things that can be done with executors that already run each step in a docker image in the nearer term
just speculating, if * we added a way for solids to have lets say a tag to override which image they should be executed in * you still had a base image that could load the whole pipeline with the solid definitions in them for the “run coordinator” node. You would have to have all the
solid
functions do lazy imports or something - assuming thats the motivation for the separate images is conflicting deps
👍 2
might be able to cobble something together - but that would be pretty tricky
2 Views