Hey, another related k8s question. When I start my deployments, dagster-celery worker take about 200 MiB of RAM. When a job get executed on the worker, the memory get used (this is normal as a process is running on the worker). But, when the worker is done with the process, the memory never goes back to its original value. I'm trying to investigate what happen after the end of the process. It's pretty annoying, because, each time a new "job" has to run on the worker, the RAM increase a little, and, at some point, kubernete kill the worker because there is not enough Memory Have you ever got something similar to it ?

interesting - definitely possible we have a memory leak. Though I am surprised anything is happening on the workers if you havent been selecting the celery engine in your pipeline confgig

yeah this is interesting, thanks for flagging this @Gaetan DELBART — not something we’ve seen yet but will investigate further. agree with @alex it’s surprising given your other thread, without that configuration nothing should’ve been happening on the workers

Yeah, my other thread is for another configuration. K8sRunLauncher and celery launcher are two separated "issues" :)

how are you sending work to the celery workers if not the celery engine? the K8sRunLauncher will just do the execution in the job pod if your not using the celery engine.

execution:
  celery:
    config:
      broker: <pyamqp://user>:pwd@dagster-rabbitmq:5672//

I'm using this configuration, and, in the interface, I'm not using the k8sRunLauncher, I'm using the "start execution"

ah i see i see thanks for the clarification