https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jesse Myers

04/05/2023, 11:37 PM
Hi folks, I'm looking for a way to have the docker launcher start runs from the code container itself not the daemon, since I'd like to spread the load on several different nodes in our swarm. The docs on the subject suggest that this is possible, any clues on how to go about this?
d

daniel

04/05/2023, 11:53 PM
We don’t currently have a feature that does this - apologies if the docs are misleading. The default run launcher lets you make launch runs on the code container from the daemon, but I’m not aware of a way to launch runs from a different container besides the daemon.
j

Jesse Myers

04/05/2023, 11:56 PM
so I can have multiple code containers on different nodes, and the daemon can launch runs on those code containers, i suppose in separate processes, but not spin up a separate container, is that correct?
d

Danny Steffy

04/06/2023, 12:04 AM
Would it be possible to write a
DockerSwarmRunLauncher
? And how complicated of a process is that?
d

daniel

04/06/2023, 12:28 AM
Jesse I don’t think that would quite work out of the box right now to have multiple code containers at once on different nodes - but sounds possible with some changes
j

Jesse Myers

04/06/2023, 12:29 AM
What changes are we talking about, is that something we can do on our end, or it's a feature that needs to be implemented in the framework?
d

daniel

04/06/2023, 12:29 AM
Danny a DockerSwarmRunLauncher sounds like not a huge lift to me as long as it has an api for launching containers, you could take a look at the DockerRunLauncher as a reference, it’s not huge - https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-docker/dagster_docker/docker_run_launcher.py#L99
Jesse I think it would require a custom run launcher of sorts currently - which isn’t trivial but is a pluggable part of the system
j

Jesse Myers

04/06/2023, 12:32 AM
so if we want to scale horizontally we'd need to either build a custom DockerSwarmRunLauncher or another custom run launcher that spins up runs from distributed code containers?
d

daniel

04/06/2023, 12:33 AM
Or use k8s or ecs which are better equipped to scale horizontally out of the box, but I imagine you were already aware of those options
j

Jesse Myers

04/06/2023, 12:34 AM
yes, our infra runs on docker swarm, so that will be very inconvenient
d

daniel

04/06/2023, 12:35 AM
You could file a feature request issue for the docker swarm run launcher if writing one is daunting - there may be others out there who would be interested in such a thing and we could incorporate it into future prioritization
j

Jesse Myers

04/06/2023, 12:36 AM
alright, thanks Daniel
I've opened a feature request in github: https://github.com/dagster-io/dagster/issues/13406
🙏 1
2 Views