19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
d
David Lacalle Castillo
09/09/2020, 3:52 PMWhat do you think about it?
d
daniel
09/09/2020, 3:53 PMThe closest thing we have to this now is the python_package option in the workspace, would that work for this use case? If you put all your definitions in the python module it would automatically load them in without a restart
d
David Lacalle Castillo
09/09/2020, 3:55 PMI am using repositories as subproyects, each one with its pipelines
So if i want to add a new repository for new pipelines I need to restart dagit website
How the python module works? Is it loaded from pip?
p
prha
09/09/2020, 4:00 PM@David Lacalle Castillo You could do a local install (e.g.
pip install -e <path/to/local_module>d
David Lacalle Castillo
09/09/2020, 4:02 PMThe problem is that I woud like the following behaviour:
• Dinamically add folders of files(repositories)
• In those folder dinamically add files (pipelines)
I managed to create new repositories and pipelines dinamically. But all of them in same file and using load from file
If in that file I edit and add pipelines or repositories they are reloaded @prha
p
prha
09/09/2020, 4:05 PMYou can add repositories and pipelines and they will be reloaded without restarting dagit. You do have to point your
workspace.yamlThis is a pointer to a python file that imports all of the repositories you want
d
David Lacalle Castillo
09/09/2020, 4:07 PMThis approach looks a little tricky but I think that it will do the trick
p
prha
09/09/2020, 4:07 PMYep… I’m sure you’ve already seen this, but there’s some documentation on workspaces here: https://docs.dagster.io/overview/repositories-workspaces/workspaces
d
David Lacalle Castillo
09/09/2020, 4:07 PMI can reload this folder with git-sync
p
prha
09/09/2020, 4:08 PMIt might not exactly have all the steps to do what you want, but try it out and let us know!
d
David Lacalle Castillo
09/09/2020, 4:08 PMOkey, thank you very much @prha
Actually I was pointing to a big file and all repositories and pipelines inside it
But it gets harder to maintain as it gets bigger
p
prha
09/09/2020, 4:10 PMIs it sufficient to have the repository/pipeline definitions in different files and then just import them in this one file?
d
David Lacalle Castillo
09/09/2020, 4:14 PMYes, I think that it will do the trick. I have tried before in a single file and worked.
I can use docker git-sync for syncing this folder.
I asked before if versioning is allowed. But i think that this will be the better approach
p
prha
09/09/2020, 4:16 PMWhat do you mean when you say pipeline versioning?
d
David Lacalle Castillo
09/09/2020, 4:30 PMIf you modify a pipeline with the same name, the previous versions and changes are stored.
p
prha
09/09/2020, 4:34 PMI see. And what would you want to do with those previous versions? Just view the structure, or also be able to run new executions with them?
d
David Lacalle Castillo
09/09/2020, 4:38 PMTo see the definition and previous runs
p
prha
09/09/2020, 4:39 PMAh, great. Yes, we already support that. We snapshot the structure of the pipelines. Historical runs are preserved, even if the pipelines change.
d
David Lacalle Castillo
09/09/2020, 4:42 PMis the old structure is shown?
p
prha
09/09/2020, 4:45 PMYep! You should see a banner indicating that it’s a historical pipeline snapshot:
(clicked through to this from a historical run)
d
David Lacalle Castillo
09/09/2020, 4:49 PMIs easy to know when the change was produced?
p
prha
09/09/2020, 4:50 PMNo, we don’t keep track of that. We just compare the snapshot to what code is currently loaded now, and know that it is not current.
d
David Lacalle Castillo
09/09/2020, 4:52 PMThank you for all your help