https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Ryan Bell

05/09/2022, 4:39 PM
Hey y'all! I'm running into a weird problem in regards to some of the Dagster pipelines I've set up. I have a workspace that generates pipelines by reading a
config.yml
file. Last week I added a new pipeline to the
config.yml
and I could see it show up in the corresponding workspace. I manually ran it, making sure it worked, which it did. However, the next time I checked on that workspace a couple days later, the new pipeline was no longer there. I can see the runs for it in my run history from last week when I manually ran it, but I don't know what would make it completely disappear from the workspace if nothing has changed in terms of the
config.yml.
Let me know if I can clarify the problem in any way
y

yuhan

05/09/2022, 10:27 PM
Hi Ryan, did you mean you dynamically generate pipelines based on a config.yml file?
r

Ryan Bell

05/09/2022, 11:41 PM
Yup, there's the word I was looking for 😅
y

yuhan

05/09/2022, 11:52 PM
is the
config.yml
(the one responsible for creating pipeline definitions) up-to-date in your workspace? the only place i can think of may be outdated is that the workspace didn’t pick up the changes in
config.yml
— which you ran manually but might not be deployed to prod
r

Ryan Bell

05/09/2022, 11:55 PM
When I say I ran it manually, I mean that instead of letting the scheduled pipeline run in prod, I manually executed the pipeline. But to be able to do that, I had to see it appear in my prod workspace. That's what I'm confused about, it was in the prod workspace one day and then not the next day. Is there another way to make sure the workspace is all up-to-date other than hitting refresh in Dagit?
Just realized I've also been liberal with my use of what a workspace is vs a repository. The pipeline that disappeared was part of a repository in my prod workspace, but then disappeared from that repository (though the repository itself is still there and so are all other pipelines within that repository)
d

daniel

05/10/2022, 3:42 AM
Hi Ryan - the only way I can imagine this happening is if you had multiple dagit replicas and only reloaded the workspace in one of them - each dagit keeps its own copy of the workspace in memory. So to switch back to an old version of the repository it would need to either have reloaded using the old config.yaml file, or been a different process that never picked up the change (unless there's a bug that somehow caches old versions of repositories, but I haven't seen other reports of that)