The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I'm trying to run scripts in parallel on a cluster through `qsub -sync y` calls within solids, but it is only submitting and running one job at a time. When I omit the `-sync y` part, the pipeline completes immediately once all the jobs are submitted, which doesn't let me see the progress in `dagit`

My `dagster.yaml` config has `max_concurrent_runs: 4`

Ahh I just needed to add the execution config to my pipeline config...
```execution:
   multiprocess:
     config:
       max_concurrent: 4```
Now I have 4 jobs running in parallel