paul.q
11/24/2020, 10:11 AMmy_package:
workspace.yaml
subfolder:
repository.py
The workspace.yaml
file under my_package
looks like this:
load from:
- relative_path: subfolder/repository.py
We start dagit from a folder with another workspace.yaml
in it that contains:
load_from:
- python_package: my_package
We get this error:
dagster.core.errors.DagsterUserCodeProcessError: (DagsterInvariantViolationError) - dagster.core.errors.DagsterInvariantViolationError: No pipelines or repositories found in "my_package".
I've changed some names and paths to simplify this. Also, this loads properly when we do it from our local environments where my_package
is not packaged (i.e. we set our PYTHONPATH
to the my_package
parent folder and our workspace.yaml
in the folder where dagit starts points at relative path my_package/subfolder/repository.py
So, is this meant to work? We'd like to be able to have a workspace.yaml
file in our deployment target that has pointers to packages as they're developed and these packages will have repositories inside them. In that way we can allow different teams to develop their own dagster packages and deploy them to a central dagit server.alex
11/24/2020, 3:40 PM@respository
objects in the top level __init__.py
of my_package
? We don’t crawl the whole package, just the entry point. For example, i am guessing if you did python_pacakge: my_package.subfolder.repository
it would workpaul.q
11/24/2020, 9:28 PM@repository
object up to the top level of the package (same folder as __init__.py
without luck. Tried your idea and still no good, BUT, I must not have tested the first idea properly. Package structure is now:
my_package:
workspace.yaml
repository.yaml
subfolder:
...
The workspace.yaml file looks like:
load_from:
- python_file:
relative_path: repository.py
And everything works!
Thanksalex
11/24/2020, 9:31 PM__init__.py
file itselfpaul.q
11/24/2020, 9:33 PM