hello everybody, for a bit now Ive had a question on my mind with Dagster regarding scheduling runs or ops/assets to run on particular groups of dagster cloud agents. I did some looking around the docs but didnt find too much. thought someone on here might have some good suggestions or has dealt with this before. the idea is to ultimately be able to materialize assets or run jobs only on a specifically named pools of workers. jobs can be then best run or nodes that can support them. maybe something in the form of attaching labels/tags to agents via their configs used on various nodes; then some kind of preference for certainly tagged nodes can be expressed in the asset definitions, or something similar. any guidance is greatly appreciated. really enjoying learning dagster btw. 🙂 (edited)

There are some ways to configure per-run configuration, depending on what deployment type / agent you’re planning on running. For example, if you’re planning on running jobs on k8s using the

K8sRunLauncher

, you can specify per-job / per-op configuration using run tags: https://docs.dagster.io/dagster-cloud/deployment/agents/kubernetes/configuration-reference#per-job-and-per-op-configuration

hi @prha 🙂. Im already deploying a hybrid agent on a set of ec2 instances, running as a container. We're also not running Kubernetes as our cluster orchestration engine. The goal with these worker pools is to keep them really simple. With both k8/ecs and others there are ways to tag underlying cluster nodes and mark affinity with jobs/containers/services that are to be provisioned on those tagged nodes... I was hoping there was some way to simulate this kind of pattern with Dagster 🙂 Thank you for your guidance.

What user code launcher are you using? Are you running a single agent?

yep, running a single agent for now. are there benefits to running multiple agents? user code launcher used right now is Docker. 🙂

I think the only way to do this right now is to configure the agent to use a custom user code launcher that extends the

DockerUserCodeLauncher

and supplies a custom run launcher that extends the

DockerRunLauncher

. This run launcher could examine the tags on the run and change accordingly the options on the launched container.