https://dagster.io/ logo
#ask-community
Title
# ask-community
y

Yang

08/05/2022, 6:40 PM
I have an op that spins out many dynamic ops. It seems that they don't run in parallel. How do I get them to run in parallel? I think each one doesn't use that many resources. Thanks!
I have a gke agent
a

alex

08/05/2022, 6:50 PM
which executor are you using on the job?
y

Yang

08/05/2022, 7:12 PM
What's an executor?
docker?
a

alex

08/05/2022, 7:22 PM
it can be set on your
@job
, the default is one that can be toggled between in process and multi process via config. An event early in the stream should indicate which is in use.
the multiprocess executor will by default choose its max concurrency based on the number of CPUs - it can be overridden by config
y

Yang

08/05/2022, 7:34 PM
hmmm I'll look around for these settings thanks!
Copy code
Executing steps using multiprocess executor: parent process (pid: 1)
I will try setting concurrent processes to a bigger number given that resources aren't being used up.
hmm, is there a way to set max_concurrent in the code? I'd like to apply it to jobs that are already queued.
a

alex

08/05/2022, 8:59 PM
you could set
executor_def
on the job directly to the
multiprocess_executor
and use
configured
to set the max concurrent config
y

Yang

08/05/2022, 9:15 PM
like this?
Copy code
multi_executor = configured(multiprocess_executor)(
    {"max_concurrent": 4})

asset_overview_zip_job = define_asset_job(
    "asset_overview_zip_job",
    selection=AssetSelection.assets(asset_overview_assets),
    executor_def=multiprocess_executor,
)
not sure how to use configured
a

alex

08/05/2022, 9:16 PM
i think that would work, also
multiprocess_executor.configured({"max_concurrent": 4})
. Two ways to do the same thing i think
y

Yang

08/05/2022, 9:16 PM
oohh ok thanks
It looks like it doesn't use the config for jobs that were already in the queue. Is that right?
a

alex

08/08/2022, 2:23 PM
if i recall correctly this is k8s, so it depends on how you manage your docker images. If you are updating in place an image tag they are using they will get updated. If you are using some sort of versioning strategy you would have to push updates to the image versions the queue jobs are set to use
y

Yang

08/08/2022, 5:48 PM
Oooh I see, yes existing jobs have the older image tagged, darn.
3 Views