Hi We need to execute jobs in different clusters due to secu

Question

Hi We need to execute jobs in different clusters due to security restrictions Can I use k8s job executor for that somehow

johann · Answer

Hi Nick can you expand on this Would the different jobs ideally all be visible within one Dagit instance

Mykola Palamarchuk · Answer

We have a couple of accounts with different resources Dagit runs in a k8s cluster in one of the accounts We want to collect data from databases in all the accounts but the direct cross account database access is prohibited So I think about possibility to run jobs in different clusters each under its own account

Mykola Palamarchuk · Answer

I see that worker should have access to the dagster database even if it runs on the remote cluster That is also a case for celery as I understand Is that correct

johann · Answer

Hmm if databases can t be accessed across accounts it s a little tricky to consolidate the data to a single Dagit Is running a separate Dagit instance in each account a possibility or would that defeat the purpose of the project

johann · Answer

Otherwise I think this likely calls for a custom run launcher or executor and you d provide whatever scheme for launching compute across clusters and getting events back

johann · Answer

It s worth mentioning that Dagster Cloud operates with an agent model that could facilitate something like this the agent in each cluster would just need outward access to the api endpoint You could reach out to < max> if you d like to chat