We have our dagster code layout where our jobs resources and

Question

We have our dagster code layout where our jobs resources and sensors are all their own modules However we need to access resources and jobs inside of sensors How can I reference a resource defined in the resources module inside of the job module so our ops have access to them When I try to just import them I get an error saying we can t do a relative import above the top level module do I have a misunderstanding on how python references and imports are handled

Danny Steffy · Answer

i e we have a job that needs to access a sql server resource that we have dynamically generated when I define in my sensor job op the required resource key it says it isn t defined when I try to reference it I get the error regarding imports Is there a different way I should be laying out my code to remove this hurdle

Jean Gonzalez · Answer

are you getting circular dependency issues if so you can always do the imports inside the sensor method

Danny Steffy · Answer

no I m trying to define the resources used for a job by just accessing the resources we ve already defined in the `resources` folder I just noticed in the `Definitions` API information that ```Jobs created with job must already have resources bound at job creation time They do not respect the resources argument here ``` This means that we would need to import the resources in the `resources` folder but I don t see a way to do that in our current folder structure do to not being able to do a relative import above the current module correct Is there some other workaround or folder layout we should be using Defining the resources a second time inside the `sensors` module would mean duplicate code

Adam Bloom · Answer

We have our user code structured similarly How are you invoking dagster daemon dagit You need your top level module to be dagster code and I suspect you re a bit deeper down the tree right now

Danny Steffy · Answer

we have dagster daemon dagit and our dagster code container The working directory in dagster code is ` opt dagster app ` which is where we copy over our dagster code structure to

Danny Steffy · Answer

we have an ` init py` in each folder in our structure Should we not be doing that

Adam Bloom · Answer

doesn t hurt but if they re empty also doesn t help

Adam Bloom · Answer

that changed back in python 3 3

Danny Steffy · Answer

they re not empty they have the definitions that we want to export to our repo py at the base level

Danny Steffy · Answer

how do you reference resources from your sensor folder do you do a relative import

Adam Bloom · Answer

one sec let me dig up some examples from our user code I ll note we re using absolute imports and not relative but either should work

Danny Steffy · Answer

our containers are all using python 3 10 slim as the base image maybe it s an out of date version of python

Adam Bloom · Answer

no don t think that s it you still haven t answered what your entrypoint is and I m more suspicious of that

Adam Bloom · Answer

as an example we launch dagster dagit etc with the following argument ```dagit m dagster user code``` usually managed by the helm chart but an example from local development

Danny Steffy · Answer

`ENTRYPOINT dagit h 0 0 0 0 p 80 w workspace yaml ` `CMD dagster api grpc h 0 0 0 0 p 4000 f repo py working directory opt dagster app ` `CMD dagster daemon run `

Danny Steffy · Answer

those are our entry points for dagit dagster code gRPC container and the daemon in order

Adam Bloom · Answer

oh they re all different interesting

Adam Bloom · Answer

We have a top level ` init py` in `dagster user code` which allows us to simply reference that module That file imports from submodules so we don t have a single file equivalent but starting up with a module vs a file may matter I can t remember

Adam Bloom · Answer

a sensor at `dagster user code sensors sensor a` has an import like the following ```from dagster user code jobs dbt import dbt test doc job factory```

Adam Bloom · Answer

the trick to all of this is that `dagster user code` needs to be a resolvable module in the interpreter and since that s how we launch dagster we know it is

Adam Bloom · Answer

one area of confusion is that python modules can be both installed in a venv or system site packages AND run directly from source I ve found that import errors are both more common and more confusing when running from source

Danny Steffy · Answer

we have an empty ` init py ` at the top level of our `dagster code` folder Do we need to add something in there to make it a resolvable module

Adam Bloom · Answer

I suspect you need to get to a place where you re pointing at your `dagster code` module instead of at `repo py` so you might need to modify that file to load the contents of `repo py`

Adam Bloom · Answer

but not 100 sure on this either it s been many months since I set this up and I haven t had to touch the module structure imports since then

Jake Kagan · Answer

scroll up a few and you ll see what i did

Jake Kagan · Answer

i posted at 10 07AM

Danny Steffy · Answer

I m very new to python so I m not sure I follow how to use that to use relative imports

Danny Steffy · Answer

here s the specific error we re getting when trying to import a resource to the sensor module ``` dagster core errors DagsterImportError Encountered ImportError `attempted relative import beyond top level package` while importing module repo from file opt dagster app repo py Local modules were resolved using the working directory ` opt dagster app` If another working directory should be used please explicitly specify the appropriate path using the ` d` or ` working directory` for CLI based targets or the `working directory` configuration option for `python file` based workspace targets ```

Adam Bloom · Answer

what s an example of what one of those imports looks like

Adam Bloom · Answer

if you re resolving from that directory you should be able to import `jobs `

Danny Steffy · Answer

`from resources sql resources import SQL RESOURCES`

Danny Steffy · Answer

hm so I should be able to just do `from resources sql resources import SQL RESOURCES`

Adam Bloom · Answer

depending on the location of that file you may have one too many ` `s

Adam Bloom · Answer

but the absolute import should also work

Danny Steffy · Answer

holy cow that worked

Danny Steffy · Answer

I kinda feel like a dummy facepalm Python imports make my head spin Is the reason for this that the top level `repo py` is where the working directory is so we can just import modules relative to that

Jake Kagan · Answer

image png

Adam Bloom · Answer

so relative imports work relative to the file doing the importing they make my head hurt since the same import will look different in different files depending on the module depth I think absolute imports are much cleaner since the same import looks the same everywhere

Jake Kagan · Answer

how do you make absolute imports work if you have a different dev env than prod

Adam Bloom · Answer

by actually installing the module in each so you can load dagster dagit with the module command there s tricks for this in poetry but with raw pip you have the ` e` flag to install a module as editable aka just a fancy symlink

Jake Kagan · Answer

so my directory `io managers` would be treated like a module you re saying you load dagster with a list of modules im totally confused

Adam Bloom · Answer

you can do a list of modules or do a single module that contains all of your submodules

Jake Kagan · Answer

shouldn t dagster do that if the modules are inside of it

Jake Kagan · Answer

btw how is poetry im using pipfile with pycharm