https://dagster.io/ logo
Powered by
b

Basil V

10/13/2020, 8:19 PM
Hello, I'm running into a 
ModuleNotFoundError: No module named '<my_module>'
error when I try to run Dagit. I recently upgraded my project to use the 
workspace.yaml
structure. I can provide more details but does anyone have an idea what may be causing the issue?
d

daniel

10/13/2020, 8:23 PM
hi Basil - I bet we can sort this out, could you paste the contents of your workspace.yaml?
b

Basil V

10/13/2020, 8:24 PM
Thanks for the help! The structure of my project is basically:
Copy code
top_level_project_folder
    my_module
        __init__.py
        repo.py
    workspace.yaml
and my workspace.yaml file contains:
Copy code
load_from:
  - python_module:
      module_name: my_module.repo
      attribute: my_repository
I'm running 
dagit
from the same directory as 
workspace.yaml
. I've tried importing/running 
find_packages
from 
setuptools
in the same directory and 
my_module
shows up.
And 
my_repository
is a function in my_module/repo.py decorated with @repository
Also a related question I've seen both 
python_module
and 
python_package
used in the 
workspace.yaml
file—which one is correct? (currently tried both and getting same error)
One other thing to note—I'm running 
dagit
from inside a pipenv shell
a

alex

10/13/2020, 8:32 PM
cc @prha we default the working directory to the location of 
workspace.yaml
so since you depend on working directory to load the module, you may be best off using 
python_file
instead of 
python_module
p

prha

10/13/2020, 8:33 PM
that’s right…. both 
python_package
and 
python_module
expect the specified package to be installed
b

Basil V

10/13/2020, 8:36 PM
When I use 
python_file
I get this error:
Copy code
ImportError: `attempted relative import with no known parent package` while importing module repo from file /path/to/my_module/repo.py. Consider using the module-based options `-m` for CLI-based targets or the `python_package` workspace.yaml target.
and 
workspace.yaml
in this case:
Copy code
load_from:
  - python_file:
      relative_path: my_module/repo.py
      attribute: my_repository
I don't understand where I'm going wrong I feel like I've paralleled the examples in your docs pretty closely.
Appreciate the help!
p

prha

10/13/2020, 8:44 PM
sure, yes, this is kind of an annoying issue that we haven’t found a good workaround for
python doesn’t like relative imports in non-packages, but we need to be able to reproducibly load your code
is it possible to do a local installation of your module / package?
b

Basil V

10/13/2020, 8:47 PM
Yeah its possible but does that mean I need to reinstall locally every time I change my package? There isn't a development workflow where I can run dagit locally?
I was able to do this before same project structure when you guys used the 
repository.yaml
file workflow
p

prha

10/13/2020, 8:49 PM
We use 
pip install -e path/to/module
to do an editable install for local development
yeah, it’s a headache, but before workspaces, we were running dagster code and user-provided pipeline logic code all together
doing this code separation cleaned up a lot to separate out those environments, but caused some code loading annoyances like 
attempted relative import with no known parent package
errors
b

Basil V

10/13/2020, 8:52 PM
So in my case I'd run 
pipenv install -e path/to/module
so I have my own module as a dependency for my project? How does that translate when I move things to production?
p

prha

10/13/2020, 8:53 PM
We may try to find another workaround for 
0.10.0
to resolve these headaches, but any change to code loading results in some breaking changes so we’re reluctant to do that in one of the weekly releases.
i’m not sure what your production deployment process is, but you would still check out your code as it currently is
b

Basil V

10/13/2020, 8:59 PM
Ok. No problem. I think that solved the issue for now. Thanks for your help and explaining things. I'll keep my eye out for changes in 
0.10.0
Thanks!
Powered by