I have created 3 python packages: - common utilit...
# ask-community
I have created 3 python packages: - common utilities - team A: analytics - team B: ML These teams are all basic scaffolds and look similar to https://github.com/slopp/dagteam/blob/main/ml_project/ml_project_tests/test_project_loads.py (only the shared utilities package) is added. Problem: Either the classpath works fine for dagster - or for pytest. But not for both. I have a call to a function from the utilities package in team A:
Copy code
from utilities.utilities import common_function
Dagster is only happy with this version above.
Copy code
from utilities import common_function
Pytest only with this version here. XOR in either case one of thm (dasger, pytest throws a module not found error) or
ImportError: cannot import name 'common_function' from 'utilities' (unknown location)
NOTICE: all 3 packages have been installed with
pip install -e .
and should be natively callable.
@geoHeil I updated the dagteam example to include a utilities package You should use
from utilities import common_function
in both cases Dagster's complaint suggests that the utilities package is either incorrectly setup or perhaps not installed when Dagster is calling it The steps that are required (and shown in the example): 1. Include a
in the utilities package that exports the common_function: https://github.com/slopp/dagteam/blob/main/utils/utils/__init__.py 2. Ensure the utilities package has a
that includes
which registers the modules within the package: https://github.com/slopp/dagteam/blob/main/utils/setup.py 3. Install the utilities package into the virtualenv for the project 4. Run dagster and pytest from the project virtualenv I also added some details to the bottom of the README. Let me know if this helps you get it sorted out!
For transitioning this to docker - would it be NOT enough to simply copy the packages? I would need to first copy and then install right? How does this though then work for a dev workflow inside docker?
When 1) passing the directory
"-d", "/opt/dagster/dagster_home/src/teamA_analytics"
and 2) installing the common package dagster is starting in docker. But I wonder how well this handles a dev cycle around reloads? Do you think when installing in editable mode the volume mapping should work fine?