Hi everyone! I've updated my dagster setup to 0.8.2 and found that there is no reload button anymore and at the same time dagit instance doesn't watch files so I have to redeploy my containers or run dagit and celery workers upder pm2 or any other watching manager alternative. Is this correct or am I missing something? Why have you eliminated the reload functionality?

Hey matas we had to temporarily remove that function to account for the new architecture. We are re-adding it this week. The backend is already implemented (see https://dagster.phacility.com/D3519) and the frontend will be done on Thursday.

Just trying to figure out how to organize code updates. We're not using dagster in prod yet but wanted to start this week after you've released 0.8.0 So what was your initial idea about it? should dagit watch files or should it be manualy restarted \ redeployed?

so with any of the run launchers we always launch a fresh process

and what happens with workers then? Imagine I have a long running pipeline with several solids. I start it with runlauncher and it generates an indeendent process for communication with workers, right? but what happens if I change last solid's code under worker while the pipeline is running? will it immediately take the updated version or does it have to be restarted to update?

i’m not sure precisely what configuration you have running. and @nate is more intimately familiar with how the workers are structured

what do you mean by "engine"?

hey @matas yes by default in the current implementation, the celery workers execute user-code in process (this function is what is executed on the workers), which means they need to be redeployed on code change right now

and if I restart a running worker it would break the pipeline, right?

only if there is a task currently executing on that particular worker - if you use

TERM

this should happen gracefully: https://docs.celeryproject.org/en/stable/userguide/workers.html#restarting-the-worker

so If I tell my pm2-watcher to restart worker process gracefully with TERM on code changes - it should make the deal?

yes I think that should do the trick, but let me know if you run into any issues

sure, thank you!

🎉

bad news @nate it was working because of my misconfiguration - all the jobs were executing not in celery but in the same process. Now I've made a simple example pipeline of 5 solids and a celery executor with 2 workers. Each solid has a sleep(x) statement inside, I change x and redeploy while running. That led me to a very strange state: I deployed 3, 6 and 10 seconds sleep solids, and it is predictable that when I redeploy while the pipeline is running I get the freshest code when a new solid starts (though of course it is not the desired behavior) but! after my last deploy (6s sleep) both workers restarted but I still have random code executing O_o. I even waited for all the pipelines to complete, redeployed again (with 1s sleep) and still having 2 and 3 versions of code executing in one pipeline randomly