Qumber Ali
11/19/2021, 1:46 PM/usr/bin/python3 -c from multiprocessing.spawn
multiprocessing.semaphore_tracker
processes start running and system got overload and with only 8 concurrent jobs my 16 cores/64 ram systemd overlods, in fact i'm not importing multiprocessing
this library in my code please help on it.daniel
11/19/2021, 2:51 PMfrom dagster import job, in_process_executor
@job(executor_def=in_process_executor)
def my_job():
...
(but then your ops will execute serially within each job instead of in parallel)Qumber Ali
11/19/2021, 3:02 PMdaniel
11/19/2021, 3:04 PMQumber Ali
11/19/2021, 3:05 PMdaniel
11/19/2021, 3:06 PMQumber Ali
11/19/2021, 3:06 PMmultiprocessing.spawn
.daniel
11/19/2021, 3:07 PMQumber Ali
11/19/2021, 3:08 PMdaniel
11/19/2021, 3:08 PMQumber Ali
11/19/2021, 3:10 PMmultiprocessing.spawn
and multiprocessing.semaphore_tracker
thread also closed too.daniel
11/19/2021, 3:16 PMQumber Ali
11/19/2021, 5:02 PMexecution:
multiprocess:
config:
max_concurrent: 4
daniel
11/19/2021, 5:09 PMQumber Ali
11/19/2021, 7:32 PMin_process_executor
this worked for me.daniel
11/19/2021, 7:33 PMQumber Ali
11/19/2021, 7:33 PMdaniel
11/19/2021, 8:18 PMQumber Ali
11/21/2021, 2:51 PM