Hey everyone, I am running into a weird ModuleNotF...
# ask-community
r
Hey everyone, I am running into a weird ModuleNotFound error when launching a workflow in a docker_user_code container. My code lives at
/app/module
inside the docker container, and I am able to successfully run
python -m module.pipeline
inside it, and the user_code rpc server is successfully able to find the module, but the Dagster execution fails. I have tried setting
-d
and
--working-directory
to
/app
but neither of them help, any ideas?
d
Hi Roshan - would you mind posting the full text of the error?
and maybe the exact import line that's failing if that's possible?
r
Yup! The line is
zeus.jobs.dagster_repo
<- this module contains 2x repos, and its called via
dagster api -d /app -m zeus.jobs.dagster_repo
Copy code
dagster.core.errors.DagsterImportError: Encountered ImportError: `No module named 'zeus'` while importing module zeus.jobs.dagster_repo. Local modules were resolved using the working directory `/app`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.
  File "/usr/local/lib/python3.9/site-packages/dagster/grpc/impl.py", line 77, in core_execute_run
    recon_pipeline.get_definition()
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstructable.py", line 150, in get_definition
    defn = self.repository.get_definition().get_pipeline(self.pipeline_name)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstructable.py", line 72, in get_definition
    return repository_def_from_pointer(self.pointer)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstructable.py", line 605, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstructable.py", line 534, in def_from_pointer
    target = pointer.load_target()
  File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 194, in load_target
    module = load_python_module(self.module, self.working_directory)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 130, in load_python_module
    raise DagsterImportError(
The above exception was caused by the following exception:
ModuleNotFoundError: No module named 'zeus'
  File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 125, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
d
oh, one thing that is possibly confusing here - this import error could be the result of another import statement within the code, it's not neccesarily specifically the result of being able to not import the 'zeus.jobs.dagster_repo' module
are there any other places that you have a local import of zeus within the module where it might be having trouble?
r
I just skimmed through my codebase and couldn't find any other place where this import exists, only in the Dockerfile for launching the user_code container
I can execute the code locally, but can't seem to via grpc/docker run launcher
d
would you mind sharing your workspace.yaml file?
Ah OK, sorry, this is specifically when launching the run? i assume you're using the DockerRunLauncher - is your gRPC server setting the DAGSTER_CURRENT_IMAGE variable correctly so that the image that is used when the run launches is the same as the image powering the gRPC server?
the relevant line from the deploy_docker example is here: https://github.com/dagster-io/dagster/blob/master/examples/deploy_docker/docker-compose.yml#L32 - I would double check that the container that was spun up for the run has the exact same image as the user code gRPC server
🎉 1
r
Yup! I set it in the docker compose
d
cool, and the image/tag of the run container is exactly the same as the image of the gRPC server?
is it possible to paste the full command/args that was passed in to the run container? It should be a big JSON blob
(the one that was unable to load the code)
r
Yes. I am just developing locally, and rebuild before each deployment of the grpc server. Can you give me a bit? I am off my computer atm, but I'll send the command here asap
d
sure, no rush
quick question - you said earlier that you 'can't seem to execute the code via grpc' - i thought it was working in the gRPC server, could you clarify what you meant here?
r
Oh, that's my bad - poor choice of words. I meant that I tried running the code locally without docker/the gRPC server and I was able to execute it. The gRPC server itself is up and running successfully.
these are the debug logs, is this what you are looking for?
d
If you run docker inspect on the two containers (the gRPC server and the container that the run launcher spins up) are there any differences? For example, the code in the gRPC server isn't mounted a a volume or anything like that by any chance?
r
I can take a look at the container that the run launcher spins up, the gRPC server has the correct mount though. Sec
d
Oh, is the code mounted as a volume? That would explain everything if it is
If that's it, you'll need to include the volumes to mount on the DockerRunLauncher config as well in your dagster.yaml
r
Oh yeah it is! Could you point me to an example?
d