19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hey!
When I run the "dagster dev -f" command local...
# ask-communitya
Alexander Holberg
06/13/2023, 2:15 PMHey!
When I run the "dagster dev -f" command locally with 3 files included a lot of python processes with approximately 144MB memory consumption are created. The number of python processes is around 12 with 1-2 being closed and reopened at 30 sec intervals.
Each code location is fairly small with one 1 job and sensor included.
The issue started when switching computer last week.
Does anybody have an idea why this is the case?
d
daniel
06/13/2023, 2:17 PMHi Alexander - there's a fix to reduce the number of python processes that are spawned at once here: https://github.com/dagster-io/dagster/pull/14649
It would be surprising to me if this were an issue that started last week though, what version of dagster are you using? Did the version of dagster possibly change?
daniel
06/13/2023, 2:18 PMOne way to reduce the number of those 'dagster api grpc' subprocesses that are created is to run your own code server instead as described here, instead of relying on the daemon to run them for you: https://docs.dagster.io/concepts/code-locations/workspace-files#running-your-own-grpc-server
a
Alexander Holberg
06/13/2023, 2:23 PMhey daniel, thx for the quick response.
I got a new computer last week and thus updated dagster to the newest version. (1.3.9)
Might be why the issue suddenly occurred.
I will try some of the fixes you referenced above and get back to you with result. 👍
d
daniel
06/13/2023, 2:23 PMdo you recall what version you were on before?
daniel
06/13/2023, 2:27 PManother option - depending on how separate you want the definitions in those different python files to be, you could combine them into either a single file with a single Definitions object combining the definitions from all three files, or a single entry point that imports a repository from each of those three files (https://docs.dagster.io/concepts/code-locations#definitions-versus-repositories) - that would leave you with one code location and one process running instead of three different processes
a
Alexander Holberg
06/13/2023, 2:34 PMI will have to check the old computer later tonight for what version of Dagster it was running.
For it now it looks like your suggestion of combining the code spaces into one is the easiest way to fix my problem. 👍
🎉 1
5 Views