Hey Dagsters and Early Adopters, Thank you for the awesome work - I am in the early stages of exploring Dagster, so far loving it but not without bumps 🙂 I am having trouble with properly importing pipelines from its respective modules when using Dagit and

workspace.yaml

with 2 pipelines under the same directory. Reading documentation was of no help since it covers only simple use cases. Here is the structure of the folder from where I run `dagit`:

- local_repo.py
- workspace.yaml
- dagster_baby_pipeline (folder with pipeline definition)
- dagster_try (folder with pipeline definition)

In

local_repo.py

I try to import

baby_pipeline.py

from

dagster_baby_pipeline

and

hello_cereal.py

from

dagster_try

to return from

@repository

function. From the logs I am suspecting that those module aren’t getting loaded into Dagster process hence the relative imports don’t work. If I specify

working_directory

for ONLY

dagster_baby_pipeline

folder in

workspace.yaml

and don’t import

dagster_try

then it works. It seems like if I create a repository for each pipeline in its respective modules and import them separately in

workspace.yaml

then it will achieve what I want but will require boilerplate overhead. Can you please clarify what is happening under the hood with imports in this scenario and what is the best practice on using these abstractions? Is there a way to load both

dagster_baby_pipeline

and

dagster_try

modules so they can be accessed by a single

local_repo.py

?

Can you share more about the error you are seeing? Unless you make this a package you can’t do “relative” imports but working directory based imports should work

I do have all of them as a package with

__init__.py

file present. If we omit

dagster_try

package for now and only try to deal with

dagster_baby_pipeline

for the sake of simplicity, here is what we get: Happy Path:

(dagster_baby_pipeline) danil@dk dagster-pipelines % cat local_repo.py
from dagster import repository
import baby_pipeline

@repository
def repository():
    return [baby_pipeline.pipeline]%                                                                                                                                                                                                       
(dagster_baby_pipeline) danil@dk dagster-pipelines % cat workspace.yaml 
load_from:
  - python_file:
      relative_path: local_repo.py
      working_directory: /Users/danil/desktop/dagster-pipelines/dagster_baby_pipeline/

(dagster_baby_pipeline) danil@dk dagster-pipelines %

Sad Path:

(dagster_baby_pipeline) danil@dk dagster-pipelines % cat local_repo.py
from dagster import repository
from dagster_baby_pipeline import baby_pipeline

@repository
def repository():
    return [baby_pipeline.pipeline]%                                                                                                                                                                                                       
(dagster_baby_pipeline) danil@dk dagster-pipelines % cat workspace.yaml 
load_from:
  - python_file:
      relative_path: local_repo.py
      working_directory: .

(dagster_baby_pipeline) danil@dk dagster-pipelines %

Error log:

UserWarning: Error loading repository location local_repo.py:dagster.core.errors.DagsterUserCodePro
cessError: dagster.core.errors.DagsterImportError: Encountered ImportError: `No module named 'io_solids'` while importing module local_repo from file /Users/danil/Desktop/dagster-pipelines/local_repo.py. Local modules were resolved usi
ng the working directory `/Users/danil/Desktop/dagster-pipelines`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `workin
g_directory` configuration option for `python_file`-based workspace.yaml targets.

io_solids

is inside `dagster_baby_pipeline`:

(dagster_baby_pipeline) danil@dk dagster-pipelines % tree
.
├── Pipfile
├── Pipfile.lock
├── dagster_baby_pipeline
│   ├── Pipfile
│   ├── Pipfile.lock
│   ├── README.md
│   ├── __init__.py
│   ├── baby_pipeline.py
│   ├── configs
│   │   └── pipeline_config.yaml
│   ├── data
│   │   ├── forestfires.csv
│   │   └── processed_df.csv
│   ├── io_solids.py
│   └── logic.py
├── dagster_try
│   ├── Pipfile
│   ├── Pipfile.lock
│   ├── __init__.py
│   ├── cereal.csv
│   ├── config.py
│   ├── config_env.yml
│   ├── hello_cereal.py
│   ├── hello_dagster.py
│   ├── inputs.py
│   └── inputs_config.yml
├── local_repo.py
└── workspace.yaml

4 directories, 24 files
(dagster_baby_pipeline) danil@dk dagster-pipelines %

are you doing

import io_solids

or

from .io_solids import ...

edit: correct relative import

io_solids

gets imported from

baby_pipeline.py

as

import io_solids

.

import .io_solids

throws a syntax error. Yes, I can confirm

python local_repo.py

throws same issue. What would be the best practice to organize the pipeline so that they can in their our modules and successfully ran from Dagit?

oh my bad - you have to do

from .io_solids import ...

Hmm that doesn’t work either but the error isn’t helpful either:

(dagster_baby_pipeline) danil@dk dagster-pipelines % python local_repo.py
Traceback (most recent call last):
  File "local_repo.py", line 2, in <module>
    from dagster_baby_pipeline import baby_pipeline
  File "/Users/danil/Desktop/dagster-pipelines/dagster_baby_pipeline/baby_pipeline.py", line 2, in <module>
    from .io_solids import io_solids as io
ImportError: cannot import name 'io_solids' from 'dagster_baby_pipeline.io_solids' (/Users/danil/Desktop/dagster-pipelines/dagster_baby_pipeline/io_solids.py)

from .io_solids import io_solids as io

- do you have a attribute named

io_solids

in the file

io_solids

?

@danil If you’re interested, I’m currently working on a CLI command

dagster new-repo

that would generate a working Dagster project skeleton with a single repository. You can view the skeleton code here on Phabricator. If you want to try out the CLI command, you can

git checkout

my feature branch with instructions here

thanks @bob, I will check it out - that sounds pretty much what I am doing: setting up local dev environment to play with multiple pipelines using Dagit.

@danil thanks for closing the loop on this. I am just getting my feet wet but have been reading through posts that relate to “structure” and this helped a ton.