I'm curious if this is a bug... My code location c...
# ask-community
c
I'm curious if this is a bug... My code location container has this
RUN
line at the end:
Copy code
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "--max_workers", "3", "--module-name", "my_module"]
If I materialize all of the assets in a job, dagster spins up 3 workers to handle concurrent processing. However, once it is all complete, if I again materialize all of the assets in the job (as I was doing as part of testing), dagster only spins up 2 workers (which is default). Is that expected or should I be specifying
--max_workers
elsewhere, too?
🤖 1
d
Hey clay - the "max-workers" parameter there is referring to a number of threads on the grpc server, it's not directly related to dagster execution. It sounds like you may be looking for the 'max_concurrent' parameter here, which controls how many ops execute at once in a given run: https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#default-job-executor
❤️ 1
(and is specified in configuration for the job)
c
ah, ok! Thanks!
So, in a docker multi-container deployment,
max_concurrent
in the job will determine how many code location containers are launched to complete the work?
d
max_concurrent will determine how many processes get spun up for a particular job. Where those processes are running will depend on which executor you pick
the default executor does each op in a subprocess
if you use the docker_executor, each op will execute in a different docker container
❤️ 1
c
ok, thanks for clarifying!
condagster 1