narom
06/29/2020, 9:04 AMv0.7.13
and I'm currently trying to update it into v0.8.5
. I'm encountering an error where the scheduler for v0.8.5
can't find my pipelines
directory.
For context, here's how v0.7.13
was structured:
dagster
- solids
- test.py
- pipelines
- test.py
- scheduler.py
- repository.py
- repository.yaml
- dagster.yaml
And here's how I structured v0.8.5
:
dagster
- solids
- test.py
- pipelines
- test.py
- repository.py
- dagster.yaml
- workspace.yaml
Everything's working fine except for the scheduled run which is raising ModuleNotFoundError: No module named 'pipelines'
. It can access the pipelines
folder when ran manually (using dagster run and by running the generated script) but not when scheduled.
Added a simplified version of my repository.py
here: https://gist.github.com/santosnarom/7f42ba94d29fb8395f06ae2b12b8cb85alex
06/29/2020, 3:02 PMcurrent working directory
is your folder, and when dagster executes from cron it does not change directories.
You can:
* make it a proper package
and pip install -e
it.
* add the directory to the python path with some shenanigans like https://github.com/dagster-io/dagster/blob/master/.buildkite/step_builder.py#L7-L9alex
06/29/2020, 3:02 PMprha
06/29/2020, 3:53 PMnarom
06/30/2020, 1:27 AM