you may have something in stdout/stderr where ever dagit is running - this should only happen if the process crashes

also having items just get stuck in the "starting" status

to be fair i am programmatically spawning about 50-100 pipeline runs at a time which i don't think is exactly a use case you guys are planning around

you may want to set

max_concurrent_runs

if they are executing via dagit https://docs.dagster.io/latest/deploying/instance/#dagit

so i assume you are using the celery engine which executes each step (based on the solid it came from) via celery

hm

i created a pipeline that spawns a bunch of new pipeline runs

using the RemoteDagitRunLauncher

well im guessing the issue is that even though actual execution isn’t happening in those processes - its too much for that one vcpu and 8g of ram to have hundreds of subprocesses going at the same time

some options are: * set

max_concurrent_runs

on the dagit instance settings - this will use an in memory queue so isnt the safest but should allow things to proceed without crashing * give the dagit box more resources * write your own run launcher that does whatever you can dream for where to handle these processes

i'll try max_concurrent_runs

what is the default for that?

so unbounded

since once the pipeline is running it just sleeping and checking on celery - the majority of contention will be at start up time

i'm gonna give both of those a shot