https://dagster.io/ logo
Join Slack
Powered by
What do you think about it?
# announcements
d
What do you think about it?
d
The closest thing we have to this now is the python_package option in the workspace, would that work for this use case? If you put all your definitions in the python module it would automatically load them in without a restart
d
I am using repositories as subproyects, each one with its pipelines
So if i want to add a new repository for new pipelines I need to restart dagit website
How the python module works? Is it loaded from pip?
p
@David Lacalle Castillo You could do a local install (e.g. 
pip install -e <path/to/local_module>
, which treats that folder as an installed package
d
The problem is that I woud like the following behaviour: • Dinamically add folders of files(repositories) • In those folder dinamically add files (pipelines)
I managed to create new repositories and pipelines dinamically. But all of them in same file and using load from file
If in that file I edit and add pipelines or repositories they are reloaded @prha
p
You can add repositories and pipelines and they will be reloaded without restarting dagit. You do have to point your 
workspace.yaml
to load a “repository location”
This is a pointer to a python file that imports all of the repositories you want
d
This approach looks a little tricky but I think that it will do the trick
p
Yep… I’m sure you’ve already seen this, but there’s some documentation on workspaces here: https://docs.dagster.io/overview/repositories-workspaces/workspaces
d
I can reload this folder with git-sync
p
It might not exactly have all the steps to do what you want, but try it out and let us know!
d
Okey, thank you very much @prha
Actually I was pointing to a big file and all repositories and pipelines inside it
But it gets harder to maintain as it gets bigger
p
Is it sufficient to have the repository/pipeline definitions in different files and then just import them in this one file?
d
Yes, I think that it will do the trick. I have tried before in a single file and worked.
I can use docker git-sync for syncing this folder.
I asked before if versioning is allowed. But i think that this will be the better approach
p
What do you mean when you say pipeline versioning?
d
If you modify a pipeline with the same name, the previous versions and changes are stored.
p
I see. And what would you want to do with those previous versions? Just view the structure, or also be able to run new executions with them?
d
To see the definition and previous runs
p
Ah, great. Yes, we already support that. We snapshot the structure of the pipelines. Historical runs are preserved, even if the pipelines change.
d
is the old structure is shown?
p
Yep! You should see a banner indicating that it’s a historical pipeline snapshot:
(clicked through to this from a historical run)
d
Is easy to know when the change was produced?
p
No, we don’t keep track of that. We just compare the snapshot to what code is currently loaded now, and know that it is not current.
d
Thank you for all your help
Open in Slack
PreviousNext
Powered by