Stepan Dvoiak
07/13/2021, 11:16 AMdagster/priority
.
So I tried and thats works only for InProcessExecutor
and MultiprocessExecutor
with max_concurrent: 1
but for max_concurrent > 1
that will not work as well as for CeleryExecutor
with dagster-celery/priority
tag
So tag dagster-celery/priority
in case of CeleryExecutor with only one process (autoscale=1,1
) have no effect in that particular example case
Can anyone know how to achieve same effect on CeleryExecutor with one worker to execute all parts of composite_solid on first DynamicOutput and only after - on secondStepan Dvoiak
07/13/2021, 11:23 AMStepan Dvoiak
07/13/2021, 11:24 AMjohann
07/13/2021, 1:55 PMmax_concurrent > 1
because the engine tries to maximize the number of steps running at once. When deciding the next step to execute, it only considers steps that have had their inputs fulfilled (and then uses priorities to pick among that group).johann
07/13/2021, 1:58 PMStepan Dvoiak
07/14/2021, 11:01 AMStepan Dvoiak
07/14/2021, 11:11 AM