https://dagster.io/ logo
Join Slack
Powered by
any ideas why a python package would not load in d...
# ask-community
d
any ideas why a python package would not load in dagit? I verified that it is installed in the venv, I can go to that venv and import it, and that if I do a 
dir(my_package)
I do see things that are of type 
RepositoryDefinition
The only error I see in dagit is 
dagster.core.errors.DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, asset groups, or asset definitions found in "my-package"
The error is coming from the right venv
p
Hi Daniel… how are you loading your workspace in dagit? Are you using a workspace.yaml? Can you share what that looks like? If a package is loaded which relies on the current directory being on the python path, you might want to specify the 
working_directory
argument. By default, dagit uses its own working directory as the base path to resolve local imports. Also, pasting this doc section in case you haven’t seen it: https://docs.dagster.io/concepts/repositories-workspaces/workspaces#workspaces
d
Yes. My workspace yaml looks like:
Copy code
load_from:
  - python_package: my_package
    enexecutable: /absolute/path/to/venv/bin/python
p
Yeah, I guess it’s not having an issue finding your package…
d
When that happens I got the module not found error. There was also a time where I go the same error as above in dagit, but the issue in the logs was a missing 
pkg_resources
(which for some reason did not get created as part of the 
venv
, so I just patched it)
p
Just digging into the workspace discovery code… we’re just using 
inspect.getmembers
calls on the module and doing isinstance checks to look for `RepositoryDefinition`…. I’m curious what it’s able to find in your loaded module
d
those checks pass
isinstance(inspect.getmembers(my_module)[-6][1],RepositoryDefinition)
returns true
p
And if you run:
Copy code
from dagster.core.workspace.autodiscovery import loadable_targets_from_python_package
loadable_targets_from_python_package("my-package")
do you get the invariant error?
d
The error I get is: 
missing 1 required positional argument: 'working directory'
Specifying 
/full/path/to/dagster/home
as the working directory (it should not matter for this package) in the venv returns a list of 
LoadableTarget
with what looks like all the repos
doing the same in the workspace.yaml does not change the error message
@prha any idea what it could be?
p
I am at a loss… my next guess would be to see what happens when you pass in 
None
to the working directory arg. Other than that, are you able to run this code through a debugger and see what the difference is in the environment?
d
I know how to run pdb, but I don't know how to run dagster that way
Ok, I finally figured it out, it was a misunderstanding on my part, but might happen to others as well. I have a package named 
my-package
, and that is what I put in the workspace.yaml file. When I import it manually (and what importlib does as well) is pass it 
my_package
. If you pass importlib 
my-package
, 
inspect.getmembers
returns an object that has a 
NamespaceLoader
and not the package fully loaded already. Since the package isn't "fully loaded", there are no assets to see there yet. Fixing the entry in the workspace.yaml file to 
my_package
fixes the issue
p
Wow, this is a good find. I’m glad you were able to find an answer for it. Will have to rememember this for when it will come up again.
3 Views
Open in Slack
PreviousNext
Powered by