Improve performance of jobs Hello I wonder how we can improv

Question

Improve performance of jobs Hello I wonder how we can improve jobs performance I experience big delay between each ops 10s in local dev mode with SQLite and with UI I also see difference of 5s duration for 30s jobs between UI and CLI How do I know where dagster spend most of the time I will share further information when I am permitted to do so

daniel · Answer

Hi Thomas the first thing I would check here is whether there s a notable difference if the run is using the multiprocess executor which I believe is the default and the in process executor the former spins up a process for each op so if you have a lot of small ops the process spinup time can become a bottleneck 10 seconds seems like a long time though You can swap out the executors by changing the executor def property on your job I ve also had some luck using pyspy to see what dagster processes are up to and what the slowest parts are

daniel · Answer

It s also possible that writing to sqlite is the bottleneck here you could try running against a postgres DB to see if that speeds things up in the past when people have reported long delays between ops we were able to trace it back to writing to the event log being the issue

daniel · Answer

< Dagster Bot> docs Add tips for debugging performance issues with dagster jobs

Dagster Bot · Answer

Created issue at <https github com dagster io dagster issues 5433>