Sebastian Charrier
03/01/2023, 11:29 PMlocations:
- location_name: data_analytics
code_source:
package_name: analytics
- location_name: ml
code_source:
package_name: machine_learning
and my folder structure
dagster_project
|--- shared_resources
| |--- resources
|--- analytics
| |--- assets
| |--- __init__.py
|--- machine_learning
| |--- assets
| |--- __init__.py
and for the defs in dagster_project/analytics/__init__.py and dagster_project/machine_learning/__init__.py I am importing the resources from shared_resources folder
from ..shared_resources.resources import RESOURCES_LOCAL, RESOURCES_PROD, RESOURCES_STAGING
from .assets import source_system_assets
deployment_name = os.environ.get("DAGSTER_DEPLOYMENT", "local")
resources_by_deployment_name = {
"local" : RESOURCES_LOCAL,
"staging": RESOURCES_STAGING,
"prod" : RESOURCES_PROD
}
all_assets = [*source_system_assets]
defs = Definitions(
assets = all_assets,
resources = resources_by_deployment_name[deployment_name],
)
But I am having this error. Is it the right way to organise code locations? how do I add more than 1 code location and share some common elements like resources or assets?
Thank you!!!daniel
03/02/2023, 12:51 AMSebastian Charrier
03/02/2023, 1:03 AMdagster._core.errors.DagsterImportError: Encountered ImportError: `attempted relative import beyond top-level package` while importing module analytics. Local modules were resolved using the working directory `/Users/charrier/Repos/dagster_project`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.
Zach
03/02/2023, 2:51 AMdagster_cloud.yaml
file inside of dagster_project
or is it in the same directory as dagster_project
?Sebastian Charrier
03/02/2023, 2:52 AMZach
03/02/2023, 2:52 AMdagster_cloud.yaml
and other top-level stuff like your README etc, then having dagster_project
be just your source code. with that setup I think you would change the package_name
attributes in your dagster_cloud.yaml
file to dagster_project.analytics
and dagster_project.machine_learning
. Daniel's right that packages that reference each other need to share a common root with a init.py file, but I think because the dagster_cloud.yaml file is inside the project it's not seeing the common rootSebastian Charrier
03/02/2023, 2:59 AMZach
03/02/2023, 3:01 AMSebastian Charrier
03/02/2023, 9:15 AMZach
03/02/2023, 4:15 PM