I m trying to run scripts in parallel on a cluster through `

Question

I m trying to run scripts in parallel on a cluster through `qsub sync y` calls within solids but it is only submitting and running one job at a time When I omit the ` sync y` part the pipeline completes immediately once all the jobs are submitted which doesn t let me see the progress in `dagit`

Justin Swaney · Answer

My `dagster yaml` config has `max concurrent runs 4`

Justin Swaney · Answer

Ahh I just needed to add the execution config to my pipeline config ```execution multiprocess config max concurrent 4``` Now I have 4 jobs running in parallel