Hi all, i'm facing an issue that when I run a job of dagster lots of

/usr/bin/python3 -c from multiprocessing.spawn

multiprocessing.semaphore_tracker

processes start running and system got overload and with only 8 concurrent jobs my 16 cores/64 ram systemd overlods, in fact i'm not importing

multiprocessing

this library in my code please help on it.

by default dagster runs each op in its own subprocess, which is what uses multiprocessing. You can turn off multiprocessing by changing your job to use the in_process_executor instead of the default multiprocess_executor:

from dagster import job, in_process_executor

@job(executor_def=in_process_executor)
def my_job():
  ...

(but then your ops will execute serially within each job instead of in parallel)

OK, why on only 8 concurrent jobs so much multiprocessing threads?

how many multiprocessing.semaphore_tracker processes are there?

59

are there more semaphore_tracker processes than regular dagster subprocesses? The default behavior would be one subprocess per op

and 70

multiprocessing.spawn

.

i think there's probably about one semaphore_tracker process per subprocess - believe that's something spun up by the multiprocessing library

yeah, may be.

I'm not totally sure what exactly you should do, but I'm happy to answer questions about how dagster works to help you understand the problem

Dear I just have dagster on my server nothing else and i'm facing multiple issues since last week please help.

Is using the in_process_executor not an option for you? That is the first thing I would try to see if it improves your performance

Can you please tell how I can set this configurations?

execution:
  multiprocess:
    config:
      max_concurrent: 4

That's run configuration, you set that in the Launchpad in dagit: https://docs.dagster.io/concepts/configuration/config-schema

Thanks @daniel

in_process_executor

this worked for me.

nice - glad it worked out. We may want to do more on our side to make multiprocess_executor less likely to cause problems when using the default config

Can you please tell me one more thing that which one is more efficient logs in files (~/dagster_home) or logs in Postgress?

Postgres is generally more performant

Ok, thanks.