How to load multiple modules in workspace yml I ve looked ar dagster #ask-community

How to load multiple modules in workspace.yml? I’v...

Dayten Sheffar

05/02/2023, 7:57 PM

How to load multiple modules in workspace.yml? I’ve looked around, nothing here or in docs has helped (and I’ve tried everything I could find)

Copy code

root/
  __init__.py
  dagster_modules/
    __init__.py
    module_1/
      __init__.py
      ops/ assets/ etc...
    module_2/
      __init__.py
      ops/ assets/ etc...
  workspace.yaml
  shared_stuff_module/
    __init__.py
    some_constants.py
    some_helper_functions.py
  other_stuff_like_dbt_project/
  ...

where

workspace.yaml

is (among tons of other efforts):

Copy code

load_from:
  - python_package:
      package_name: module_1
  - python_package:
      package_name: module_2
  - python_package:
      package_name: shared_stuff_module

and the modules show up in UI but with error loading saying module cannot be found If I leave out the 2nd and 3rd entries under

load_from

, then

module_1

loads, but it can’t find relative imports from

shared_stuff_module

. Any help appreciated

Dayten Sheffar

05/02/2023, 7:59 PM

image.png,image.png

Dayten Sheffar

05/02/2023, 8:00 PM

shared_stuff_module

(i.e.,

helpers/

) doesnt really need to be under

load_from

I guess since there are no dagster assets, ops, etc… in there, but I can’t resolve the relative imports in say module_1

Dayten Sheffar

05/02/2023, 8:03 PM

If I move the modules up to the root (outside of dagster_modules) then they load, but I’d like to put them in the same folder (and not floating in the root folder)

Dayten Sheffar

05/02/2023, 8:04 PM

but then i get the relative import error

Odette Harary

05/02/2023, 8:12 PM

See if this helps

https://www.youtube.com/watch?v=GhD3i2pOp4A▾

Dayten Sheffar

05/02/2023, 8:42 PM

@Odette Harary yeah thats kind of helpful, but misses some of my main points. What would you do if

my_ml_pipeline

and

my_data_pipeline

were further nested in a folder called

dagster_modules

, can you leave

workspace.yaml

in the root and just change the contents of

workspace.yaml

to be like,

python_module:

<http://dagster_modules.my|dagster_modules.my>_ml_pipeline

Dayten Sheffar

05/02/2023, 8:43 PM

I did manage to solve my issue with relative imports from the

helpers

folder that contains no dagster assets etc.

Odette Harary

05/02/2023, 8:55 PM

working_directory should help as well

4 Views

Open in Slack

Previous Next