Marco Jacopo Ferrarotti

09/03/2022, 6:26 PM
Hi guys, I'm reading the docs since I'd like to propose Dagster as primary orchestrator for data pipelines at work. We have a mixed HPC/Cloud infrastructure where the Cloud partition is based on Open Stack and the HPC partition is managed by the PBS workload manager.. I would host dagit and the code workspaces/repositories on OpenStack VMs but I would like to spawn the jobs on the HPC partition. Has anyone experimented with a similar setup? From what I understood I would need to implement a custom RunLauncher to integrate with PBS.. Am I correct? If the community has something already backed for such use case I'd love to take a look at it..