https://dagster.io/ logo
#dagster-plus
Title
# dagster-plus
z

Zach

08/10/2023, 5:32 PM
Is it possible to have different RunLaunchers for different code locations within a single deployment?
I'd assume this would require multiple agents to be deployed, but it's unclear how I would indicate what code locations would be handled by what agent
j

johann

08/10/2023, 5:52 PM
It’s not currently possible, though per-agent routing is in our queue
Some users have enabled it themselves by writing a custom run launcher that wraps 2 existing launchers, and routes to them based on run tag/code location/etc.
🌈 1
z

Zach

08/10/2023, 5:58 PM
Ah okay, yeah I had briefly considered that - seems like it would be more flexible anyway to just have a tag-based run launcher. Thanks!
m

Martin Peak

08/10/2023, 11:37 PM
this feature would be really useful for us. Some runs needs to run on a windows box due to a licensing constraint, whereas the rest would run on our standard linux container setup.
j

johann

08/11/2023, 2:52 PM
I thought someone had shared code from doing this previously but I’m not finding it. If one of you guys ended up making a GH discussion or something with a scaffold for it that would be lovely
Ah you’re specifically talking about agent routing actually Martin? That one will be on us 🙂
a

Alex Kan

08/11/2023, 8:53 PM
Would be really useful for us as well ^ • For our code locations running Python code, we want the jobs to execute in a k8s pod • For other code locations that just run dbt commands, we would like to avoid the overhead of starting a k8s pod and just run a subprocess that executes the dbt cli command Originally asked at https://dagster.slack.com/archives/C01U954MEER/p1691786342099089 but in the context of Dagster open source, not Cloud
m

Martin Peak

08/15/2023, 9:13 AM
Yes for us it would be multiple agents set up and the ability to tag a job so that it could get executed by a specific agent.
j

johann

08/15/2023, 3:26 PM
We appreciate the feedback! Agent routing is on our roadmap for q3.
❤️ 1
Alex this won’t affect you in OSS where we don’t have agents, but the custom run launcher on top of two existing launchers has worked for other users. I’d also recommend this for anyone on cloud hybrid who needs this immediately, or wants extra control over launching.
❤️ 1
a

Alex Kan

08/15/2023, 3:30 PM
Do you have a code pointer/slack link to this implementation?
j

johann

08/15/2023, 3:31 PM
I recall at least one but unfortunately can’t find it. I’d love to get this turned in to a GH discussion that we could reference
I wrote up https://github.com/dagster-io/dagster/discussions/15856 to give some details on the custom run launcher process
❤️ 1
2 Views