19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Is it possible to have different RunLaunchers for ...
# dagster-plusz
Zach
08/10/2023, 5:32 PMIs it possible to have different RunLaunchers for different code locations within a single deployment?
Zach
08/10/2023, 5:33 PMI'd assume this would require multiple agents to be deployed, but it's unclear how I would indicate what code locations would be handled by what agent
j
johann
08/10/2023, 5:52 PMIt’s not currently possible, though per-agent routing is in our queue
johann
08/10/2023, 5:53 PMSome users have enabled it themselves by writing a custom run launcher that wraps 2 existing launchers, and routes to them based on run tag/code location/etc.
🌈 1
z
Zach
08/10/2023, 5:58 PMAh okay, yeah I had briefly considered that - seems like it would be more flexible anyway to just have a tag-based run launcher. Thanks!
m
Martin Peak
08/10/2023, 11:37 PMthis feature would be really useful for us. Some runs needs to run on a windows box due to a licensing constraint, whereas the rest would run on our standard linux container setup.
j
johann
08/11/2023, 2:52 PMI thought someone had shared code from doing this previously but I’m not finding it. If one of you guys ended up making a GH discussion or something with a scaffold for it that would be lovely
johann
08/11/2023, 2:53 PMAh you’re specifically talking about agent routing actually Martin? That one will be on us 🙂
a
Alex Kan
08/11/2023, 8:53 PMWould be really useful for us as well ^
• For our code locations running Python code, we want the jobs to execute in a k8s pod
• For other code locations that just run dbt commands, we would like to avoid the overhead of starting a k8s pod and just run a subprocess that executes the dbt cli command
Originally asked at https://dagster.slack.com/archives/C01U954MEER/p1691786342099089 but in the context of Dagster open source, not Cloud
m
Martin Peak
08/15/2023, 9:13 AMYes for us it would be multiple agents set up and the ability to tag a job so that it could get executed by a specific agent.
j
johann
08/15/2023, 3:26 PMWe appreciate the feedback! Agent routing is on our roadmap for q3.
❤️ 1
johann
08/15/2023, 3:29 PMAlex this won’t affect you in OSS where we don’t have agents, but the custom run launcher on top of two existing launchers has worked for other users. I’d also recommend this for anyone on cloud hybrid who needs this immediately, or wants extra control over launching.
❤️ 1
a
Alex Kan
08/15/2023, 3:30 PMDo you have a code pointer/slack link to this implementation?
j
johann
08/15/2023, 3:31 PMI recall at least one but unfortunately can’t find it. I’d love to get this turned in to a GH discussion that we could reference
johann
08/15/2023, 3:55 PMI wrote up https://github.com/dagster-io/dagster/discussions/15856 to give some details on the custom run launcher process
❤️ 1
johann
08/15/2023, 4:06 PMAnd added a comment about this specific case https://github.com/dagster-io/dagster/discussions/15856#discussioncomment-6732093
3 Views