https://dagster.io/ logo
#announcements
Title
# announcements
g

Gaetan DELBART

03/26/2020, 10:54 AM
Hey, another related k8s question. When I start my deployments, dagster-celery worker take about 200 MiB of RAM. When a job get executed on the worker, the memory get used (this is normal as a process is running on the worker). But, when the worker is done with the process, the memory never goes back to its original value. I'm trying to investigate what happen after the end of the process. It's pretty annoying, because, each time a new "job" has to run on the worker, the RAM increase a little, and, at some point, kubernete kill the worker because there is not enough Memory Have you ever got something similar to it ?
Memory at startup
Memory after execution
a

alex

03/26/2020, 3:11 PM
interesting - definitely possible we have a memory leak. Though I am surprised anything is happening on the workers if you havent been selecting the celery engine in your pipeline confgig
n

nate

03/26/2020, 3:38 PM
yeah this is interesting, thanks for flagging this @Gaetan DELBART — not something we’ve seen yet but will investigate further. agree with @alex it’s surprising given your other thread, without that configuration nothing should’ve been happening on the workers
g

Gaetan DELBART

03/26/2020, 3:40 PM
Yeah, my other thread is for another configuration. K8sRunLauncher and celery launcher are two separated "issues" :)
a

alex

03/26/2020, 3:41 PM
how are you sending work to the celery workers if not the celery engine? the K8sRunLauncher will just do the execution in the job pod if your not using the celery engine.
g

Gaetan DELBART

03/26/2020, 3:43 PM
Copy code
execution:
  celery:
    config:
      broker: <pyamqp://user>:pwd@dagster-rabbitmq:5672//
I'm using this configuration, and, in the interface, I'm not using the k8sRunLauncher, I'm using the "start execution"
a

alex

03/26/2020, 3:43 PM
ah i see i see thanks for the clarification
thanks a lot for the report! well have to dig in to figure out why we’re bleeding memory
https://github.com/dagster-io/dagster/issues/2334 will keep posting updates on this issue - but so far it seems that the issue is the loaded code for doing the processing staying in memory