Hey y'all! I'm running into a weird problem in regards to some of the Dagster pipelines I've set up. I have a workspace that generates pipelines by reading a

config.yml

file. Last week I added a new pipeline to the

config.yml

and I could see it show up in the corresponding workspace. I manually ran it, making sure it worked, which it did. However, the next time I checked on that workspace a couple days later, the new pipeline was no longer there. I can see the runs for it in my run history from last week when I manually ran it, but I don't know what would make it completely disappear from the workspace if nothing has changed in terms of the

config.yml.

Let me know if I can clarify the problem in any way

Hi Ryan, did you mean you dynamically generate pipelines based on a config.yml file?

Yup, there's the word I was looking for 😅

is the

config.yml

(the one responsible for creating pipeline definitions) up-to-date in your workspace? the only place i can think of may be outdated is that the workspace didn’t pick up the changes in

config.yml

— which you ran manually but might not be deployed to prod

When I say I ran it manually, I mean that instead of letting the scheduled pipeline run in prod, I manually executed the pipeline. But to be able to do that, I had to see it appear in my prod workspace. That's what I'm confused about, it was in the prod workspace one day and then not the next day. Is there another way to make sure the workspace is all up-to-date other than hitting refresh in Dagit?

Hi Ryan - the only way I can imagine this happening is if you had multiple dagit replicas and only reloaded the workspace in one of them - each dagit keeps its own copy of the workspace in memory. So to switch back to an old version of the repository it would need to either have reloaded using the old config.yaml file, or been a different process that never picked up the change (unless there's a bug that somehow caches old versions of repositories, but I haven't seen other reports of that)