Moody Edghaim
05/09/2023, 5:41 PMOpExecutionContext.run_config
allows us to inspect the run configurations, and I'm able to inspect max_concurrent
in there, but only if it's provided as a value when launching the job via Dagit.Moody Edghaim
05/09/2023, 6:55 PMThe max_concurrent arg is optional and tells the execution engine how many processes may run concurrently. By default, or if you set max_concurrent to be 0, this is the return value of python:multiprocessing.cpu_count().
so, i can inspect the OpExecutionContext.run_config
, get the execution.config.max_concurrent
key.
if it's 0 or doesn't exist as a key in the run_config
, then i should just use the multiprocessing.cpu_count()
value?
is there a built-in alternative?sean
05/09/2023, 9:27 PMOpExecutionContext
. Would you mind opening a GH issue?Moody Edghaim
05/09/2023, 10:08 PMMoody Edghaim
05/09/2023, 10:19 PM