https://dagster.io/ logo
Title
a

Arun Kumar

09/15/2021, 7:41 PM
Hi team, still facing this memory issue when the sensors are turned on. Running profilers within the sensor code did not provide any useful information. Is there any other way to profile the overall user code process?
d

daniel

09/15/2021, 7:47 PM
Have you been able to check if all 3 of the sensors have the issue? (e.g. by turning them on one at a time and checking the memory usage). If it was only one of the three that could be a pretty useful clue
a

Arun Kumar

09/15/2021, 7:52 PM
Yes, even with 1 sensor, the memory is rising slowing. More the number of sensors faster the memory rise. I am currently just trying to comment different parts of a sensor to find where the memory is leaking.
d

daniel

09/15/2021, 7:52 PM
'even with 1 sensor' as in each of the three has the rpoblem?
i'd be curious if a sensor that does almost nothing has the problem - that would help rule out if its truly specific to your sensor code
a

Arun Kumar

09/15/2021, 7:56 PM
Yesterday I tried running 3 empty sensors (sensors that did nothing and had only a print statement and each with minimum_interval_seconds set to 30 s). This was the memory trend.
d

daniel

09/15/2021, 7:57 PM
got it - so they stabilized in a way that the 'real' sensors do not?
or is that still a bad graph
(if you have a similar graph for the real sensors that would be helpful)
a

Arun Kumar

09/15/2021, 8:06 PM
I would not call it a good graph. Not sure why the memory was initially rising given that we only run 3 empty sensors. Also, I see that in the stable part of the graph the memory was not completely constant and increasing very slowly
This is the graph for real sensors
😮 1
d

daniel

09/15/2021, 8:07 PM
i guess good is all relative 🙂 at least it seems much more stable than this new graph
1
a

Arun Kumar

09/15/2021, 8:16 PM
Ah, yes its surely much much better than the real sensor graph.