10/28/2021, 8:54 AM
Improve performance of jobs Hello, I wonder how we can improve jobs performance ? I experience big delay between each ops (10s in local dev mode with SQLite and with UI). I also see difference of 5s duration for 30s jobs between UI and CLI. How do I know where dagster spend most of the time ? I will share further information when I am permitted to do so.


10/28/2021, 2:25 PM
Hi Thomas - the first thing I would check here is whether there's a notable difference if the run is using the multiprocess_executor (which I believe is the default) and the in_process_executor - the former spins up a process for each op, so if you have a lot of small ops the process spinup time can become a bottleneck (10 seconds seems like a long time though) You can swap out the executors by changing the executor_def property on your job: I've also had some luck using pyspy to see what dagster processes are up to and what the slowest parts are:
It's also possible that writing to sqlite is the bottleneck here - you could try running against a postgres DB to see if that speeds things up (in the past when people have reported long delays between ops we were able to trace it back to writing to the event log being the issue)
@Dagster Bot docs Add tips for debugging performance issues with dagster jobs
👀 1
👍 1

Dagster Bot

10/28/2021, 2:27 PM