Hi guys, Is there any way to share resources and/or assets in more than 1 code location? For example I have this in dagster_cloud.yml

locations:
  - location_name: data_analytics
    code_source:
      package_name: analytics
  - location_name: ml
    code_source:
      package_name: machine_learning

and my folder structure

dagster_project
|--- shared_resources
|   |--- resources
|--- analytics
|   |--- assets
|   |--- __init__.py
|--- machine_learning
|   |--- assets
|   |--- __init__.py

and for the defs in dagster_project/analytics/__init__.py and dagster_project/machine_learning/__init__.py I am importing the resources from shared_resources folder

from ..shared_resources.resources import RESOURCES_LOCAL, RESOURCES_PROD, RESOURCES_STAGING
from .assets import source_system_assets

deployment_name = os.environ.get("DAGSTER_DEPLOYMENT", "local")
resources_by_deployment_name = {
    "local" : RESOURCES_LOCAL,
    "staging": RESOURCES_STAGING,
    "prod" : RESOURCES_PROD
}

all_assets = [*source_system_assets]

defs = Definitions(
    assets = all_assets,
    resources = resources_by_deployment_name[deployment_name],
)

But I am having this error. Is it the right way to organise code locations? how do I add more than 1 code location and share some common elements like resources or assets? Thank you!!!

Hi Sebastian - this looks like more of a Python import problem when it’s trying to load your code than a Dagster-specific issue. I suspect you would hit the same error if you ran “python -m analytics” in the base folder. What if you add an ___init__ _.py to the shared_resources folder like the other two folders have so that Python knows that it’s a Python package? You may also need to add one to the dagster_project folder.

Thank you Daniel, I have added the init files inside shared_resources folder and the dagster_project but still having the same error.

dagster._core.errors.DagsterImportError: Encountered ImportError: `attempted relative import beyond top-level package` while importing module analytics. Local modules were resolved using the working directory `/Users/charrier/Repos/dagster_project`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.

is your

dagster_cloud.yaml

file inside of

dagster_project

or is it in the same directory as

dagster_project

?

it is inside of dagster_project

I would also try writing a test that just tests importing objects from the shared_resources file from one of the machine_learning files outside of dagster to make sure the project structure is working outside of dagster

That make sense. Thank you, I will give a try

not completely sure it'll work but worth a shot. like I said though I would get it working outside of dagster first, then add in the dagster cloud deployment

Thanks a lot for you help. It is finally working as I expected. I have created a small demo in case someone else need it https://github.com/sebastiancharrier/dagster_multi_code_locations

that's cool, I actually hadn't seen multiple code locations organized in a single repo like that but have been looking to do something similar, this will be a great example to follow!