How can I (ideally per code location or even more fine grained) control which agent is running an job/op/asset materialization? I see that for accelerator hardware i.e. GPUs https://dagster.slack.com/archives/C02LJ7G0LAZ/p1677528206949639?thread_ts=1677527782.079859&cid=C02LJ7G0LAZ kubernetes tags for specific jobs might be sufficient. But all jobs are still using the same dagster-cloud agent to launch. However, my problem is a bit different: I am in a hybrid cloud setup where some jobs need to launch on-premise and others in the cloud. Currently, I think it would be easiest to deploy separate agents and tag these (if possible) and somehow apply the tags on jobs/ops/assets so the right agent will be chosen. Is this a possibility? If not, how does the way forward look like?
03/14/2023, 8:52 PM
Currently the solution is to create a second deployment for on prem jobs and have the on prem agent point at that.
03/14/2023, 8:57 PM
Are there any roadmap plans to simplify this as indicated above?
@johann do you have any best practices i.e. for firewall changes vs. multi-agent setups?