Hey Dagsters and Early Adopters Thank you for the awesome wo

Question

Hey Dagsters and Early Adopters Thank you for the awesome work I am in the early stages of exploring Dagster so far loving it but not without bumps slightly smiling face I am having trouble with properly importing pipelines from its respective modules when using Dagit and `workspace yaml` with 2 pipelines under the same directory Reading documentation was of no help since it covers only simple use cases Here is the structure of the folder from where I run `dagit` ``` local repo py workspace yaml dagster baby pipeline folder with pipeline definition dagster try folder with pipeline definition ``` In `local repo py` I try to import `baby pipeline py` from `dagster baby pipeline` and `hello cereal py` from `dagster try` to return from ` repository` function From the logs I am suspecting that those module aren t getting loaded into Dagster process hence the relative imports don t work If I specify `working directory` for ONLY `dagster baby pipeline` folder in `workspace yaml` and don t import `dagster try` then it works It seems like if I create a repository for each pipeline in its respective modules and import them separately in `workspace yaml` then it will achieve what I want but will require boilerplate overhead Can you please clarify what is happening under the hood with imports in this scenario and what is the best practice on using these abstractions Is there a way to load both `dagster baby pipeline` and `dagster try` modules so they can be accessed by a single `local repo py`

danil · Answer

Let me know if you d like me to create a ticket in GitHub issues or paste the code here What ever makes it easier to debug

alex · Answer

Can you share more about the error you are seeing Unless you make this a package you can t do relative imports but working directory based imports should work

alex · Answer

the quick test i did ``` dagenv38 tmp project $ tree folder a init py folder b init py repo py workspace yaml 2 directories 4 files dagenv38 tmp project $ cat workspace yaml load from python file relative path repo py working directory dagenv38 tmp project $ cat repo py from dagster import repository from folder a import my pipeline from folder b import other pipeline repository def my repo return my pipeline other pipeline ```

danil · Answer

I do have all of them as a package with ` init py` file present If we omit `dagster try` package for now and only try to deal with `dagster baby pipeline` for the sake of simplicity here is what we get Happy Path ``` dagster baby pipeline danil dk dagster pipelines cat local repo py from dagster import repository import baby pipeline repository def repository return baby pipeline pipeline dagster baby pipeline danil dk dagster pipelines cat workspace yaml load from python file relative path local repo py working directory Users danil desktop dagster pipelines dagster baby pipeline dagster baby pipeline danil dk dagster pipelines ``` Sad Path ``` dagster baby pipeline danil dk dagster pipelines cat local repo py from dagster import repository from dagster baby pipeline import baby pipeline repository def repository return baby pipeline pipeline dagster baby pipeline danil dk dagster pipelines cat workspace yaml load from python file relative path local repo py working directory dagster baby pipeline danil dk dagster pipelines ``` Error log ```UserWarning Error loading repository location local repo py dagster core errors DagsterUserCodePro cessError dagster core errors DagsterImportError Encountered ImportError `No module named io solids ` while importing module local repo from file Users danil Desktop dagster pipelines local repo py Local modules were resolved usi ng the working directory ` Users danil Desktop dagster pipelines` If another working directory should be used please explicitly specify the appropriate path using the ` d` or ` working directory` for CLI based targets or the `workin g directory` configuration option for `python file` based workspace yaml targets ``` `io solids` is inside `dagster baby pipeline` ``` dagster baby pipeline danil dk dagster pipelines tree Pipfile Pipfile lock dagster baby pipeline Pipfile Pipfile lock README md init py baby pipeline py configs pipeline config yaml data forestfires csv processed df csv io solids py logic py dagster try Pipfile Pipfile lock init py cereal csv config py config env yml hello cereal py hello dagster py inputs py inputs config yml local repo py workspace yaml 4 directories 24 files dagster baby pipeline danil dk dagster pipelines ```

danil · Answer

So my understading is that dagster is failing to load recursively everything under the current working directory

danil · Answer

< alex> Thanks for getting back to me so quickly appreciate it

alex · Answer

are you doing `import io solids` or `from io solids import ` edit correct relative import

alex · Answer

the former wont work unless `dagster baby pipeline` is in the python path i believe you need to do relative imports once you are inside the module

alex · Answer

gt dagster is failing to load recursively everything under the current working directory this is less of a dagster specific issue and more to do with how python works <https docs python org 3 tutorial modules html> you should see the same behavior if you just do `python local repo py`

danil · Answer

`io solids` gets imported from `baby pipeline py` as `import io solids` `import io solids` throws a syntax error Yes I can confirm `python local repo py` throws same issue What would be the best practice to organize the pipeline so that they can in their our modules and successfully ran from Dagit

danil · Answer

``` File Users danil Desktop dagster pipelines dagster baby pipeline baby pipeline py line 2 import io solids as io ^ SyntaxError invalid syntax```

alex · Answer

oh my bad you have to do `from io solids import `

danil · Answer

Hmm that doesn t work either but the error isn t helpful either ``` dagster baby pipeline danil dk dagster pipelines python local repo py Traceback most recent call last File local repo py line 2 in lt module gt from dagster baby pipeline import baby pipeline File Users danil Desktop dagster pipelines dagster baby pipeline baby pipeline py line 2 in lt module gt from io solids import io solids as io ImportError cannot import name io solids from dagster baby pipeline io solids Users danil Desktop dagster pipelines dagster baby pipeline io solids py ```

alex · Answer

`from io solids import io solids as io` do you have a attribute named `io solids` in the file `io solids`

alex · Answer

you can do `from io solids import ` but otherwise i think you want to just import the things from that file manually `from io solids import solid x solid y `

bob · Answer

< danil> If you re interested I m currently working on a CLI command `dagster new repo` that would generate a working Dagster project skeleton with a single repository You can If you want to try out the CLI command you can `git checkout` my feature branch with

danil · Answer

thanks bob I will check it out that sounds pretty much what I am doing setting up local dev environment to play with multiple pipelines using Dagit

danil · Answer

< alex> ` from io solids import ` seems to be the remedy for the this problem Thanks so much this is resolved

danil · Answer

< alex> < bob> Following up on this again currently I am able to launch `baby pipeline` from Dagit UI alongside others which is a big win thanks guys However once I get inside `dagster baby pipeline` module and try to run the pipeline separately getting the import error the other way ``` dagster baby pipeline danil dk dagster pipelines tree Pipfile Pipfile lock init py dagster baby pipeline Pipfile Pipfile lock README md init py baby pipeline py configs py constants py data forestfires csv processsed df csv io solids py logic py dagster try Pipfile Pipfile lock init py cereal csv config py config env yml hello cereal py hello dagster py inputs py inputs config yml local repo py workspace yaml 3 directories 26 files dagster baby pipeline danil dk dagster pipelines cd dagster baby pipeline dagster baby pipeline danil dk dagster baby pipeline dagster pipeline execute f baby pipeline py preset local Traceback most recent call last File Users danil local share virtualenvs dagster baby pipeline Vr5WxBjI lib python3 8 site packages dagster core code pointer py line 94 in load python file return import module from path module name python file File Users danil local share virtualenvs dagster baby pipeline Vr5WxBjI lib python3 8 site packages dagster seven init py line 50 in import module from path spec loader exec module module File lt frozen importlib bootstrap external gt line 783 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File baby pipeline py line 2 in lt module gt from io solids import ImportError attempted relative import with no known parent package```

danil · Answer

Are you aware of Python wizardry to have it both ways be able to run as a module from a higher level directory using Dagit and being able to run it standalone It is important to be able to do both for seamless dev UX My use case is I want to share `dagster baby pipeline` with my coworker via a GitHub repo hence running on its own is a must while being able to load multiple pipelines in Dagit UI at the same time

danil · Answer

I was able to figure this out by using imports like this ` from dagster baby pipeline import constants` inside `baby pipeline` for Dagit to work and `dagster pipeline execute f baby pipeline py d preset local` for running from within the module `` d ` saved the day

Michael Lynton · Answer

< danil> thanks for closing the loop on this I am just getting my feet wet but have been reading through posts that relate to structure and this helped a ton