https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Danny Steffy

03/15/2023, 5:08 PM
We have our dagster code layout where our jobs, resources, and sensors are all their own modules. However, we need to access resources and jobs inside of sensors. How can I reference a resource defined in the resources module inside of the job module so our ops have access to them? When I try to just import them, I get an error saying we can't do a relative import above the top level module, do I have a misunderstanding on how python references and imports are handled?
i.e. we have a job that needs to access a sql server resource that we have dynamically generated. when I define in my sensor job op the required resource key, it says it isn't defined. when I try to reference it, I get the error regarding imports. Is there a different way I should be laying out my code to remove this hurdle?
j

Jean Gonzalez

03/15/2023, 5:17 PM
are you getting circular dependency issues? if so you can always do the imports inside the sensor method
d

Danny Steffy

03/15/2023, 5:21 PM
no, I'm trying to define the resources used for a job by just accessing the resources we've already defined in the
resources
folder. I just noticed in the
Definitions
API information that
Copy code
Jobs created with @job must already have resources bound at job creation time. They do not respect the resources argument here.
This means that we would need to import the resources in the
resources
folder, but I don't see a way to do that in our current folder structure do to not being able to do a relative import above the current module, correct? Is there some other workaround or folder layout we should be using? Defining the resources a second time inside the
sensors
module would mean duplicate code
a

Adam Bloom

03/15/2023, 5:29 PM
We have our user code structured similarly. How are you invoking dagster-daemon/dagit? You need your top level module to be dagster-code, and I suspect you're a bit deeper down the tree right now
d

Danny Steffy

03/15/2023, 5:33 PM
we have dagster-daemon, dagit, and our dagster-code container. The working directory in dagster-code is
/opt/dagster/app/
which is where we copy over our dagster-code structure to.
we have an
___init___.py
in each folder in our structure. Should we not be doing that?
a

Adam Bloom

03/15/2023, 5:35 PM
doesn't hurt, but if they're empty also doesn't help
(that changed back in python 3.3)
d

Danny Steffy

03/15/2023, 5:35 PM
they're not empty, they have the definitions that we want to export to our repo.py at the base level
how do you reference resources from your sensor folder? do you do a relative import?
a

Adam Bloom

03/15/2023, 5:38 PM
one sec, let me dig up some examples from our user code. I'll note we're using absolute imports and not relative, but either should work
d

Danny Steffy

03/15/2023, 5:38 PM
our containers are all using python:3.10-slim as the base image. maybe it's an out of date version of python?
a

Adam Bloom

03/15/2023, 5:39 PM
no, don't think that's it. you still haven't answered what your entrypoint is, and I'm more suspicious of that
as an example, we launch dagster/dagit/etc with the following argument:
Copy code
dagit -m dagster_user_code
(usually managed by the helm chart, but an example from local development)
d

Danny Steffy

03/15/2023, 5:40 PM
ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "80", "-w", "workspace.yaml"]
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py", "--working-directory", "/opt/dagster/app/"]
CMD ["dagster-daemon", "run"]
those are our entry points for dagit, dagster-code gRPC container, and the daemon in order
a

Adam Bloom

03/15/2023, 5:42 PM
oh, they're all different? interesting.
We have a top level
__init__.py
in
dagster_user_code
, which allows us to simply reference that module. That file imports from submodules, so we don't have a single file equivalent. but starting up with a module vs a file may matter, I can't remember
a sensor at
dagster_user_code.sensors.sensor_a
has an import like the following:
Copy code
from dagster_user_code.jobs.dbt import dbt_test_doc_job_factory
the trick to all of this is that
dagster_user_code
needs to be a resolvable module in the interpreter. and since that's how we launch dagster, we know it is
one area of confusion is that python modules can be both installed (in a venv or system site-packages) AND run directly from source. I've found that import errors are both more common (and more confusing!) when running from source
d

Danny Steffy

03/15/2023, 5:47 PM
we have an empty
___init__.py_
at the top level of our
dagster-code
folder. Do we need to add something in there to make it a resolvable module?
a

Adam Bloom

03/15/2023, 5:48 PM
I suspect you need to get to a place where you're pointing at your
dagster-code
module, instead of at
repo.py
, so you might need to modify that file to load the contents of
repo.py
but, not 100% sure on this either - it's been many months since I set this up, and I haven't had to touch the module structure/imports since then
j

Jake Kagan

03/15/2023, 5:50 PM
scroll up a few and you'll see what i did
i posted at 10:07AM
d

Danny Steffy

03/15/2023, 5:55 PM
I'm very new to python so I'm not sure I follow how to use that to use relative imports
here's the specific error we're getting when trying to import a resource to the sensor module:
Copy code
dagster._core.errors.DagsterImportError: Encountered ImportError: `attempted relative import beyond top-level package` while importing module repo from file /opt/dagster/app/repo.py. Local modules were resolved using the working directory `/opt/dagster/app`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for `python_file`-based workspace targets.
a

Adam Bloom

03/15/2023, 6:02 PM
what's an example of what one of those imports looks like?
if you're resolving from that directory, you should be able to import
jobs.__
d

Danny Steffy

03/15/2023, 6:02 PM
from ...resources.sql_resources import SQL_RESOURCES
hm, so I should be able to just do
from resources.sql_resources import SQL_RESOURCES
?
a

Adam Bloom

03/15/2023, 6:04 PM
depending on the location of that file, you may have one too many `.`s
but the absolute import should also work
d

Danny Steffy

03/15/2023, 6:05 PM
holy cow, that worked
I kinda feel like a dummy facepalm Python imports make my head spin. Is the reason for this that the top level
repo.py
is where the working directory is, so we can just import modules relative to that?
j

Jake Kagan

03/15/2023, 6:07 PM
image.png
a

Adam Bloom

03/15/2023, 6:07 PM
so, relative imports work relative to the file doing the importing. they make my head hurt, since the same import will look different in different files (depending on the module depth). I think absolute imports are much cleaner, since the same import looks the same everywhere
j

Jake Kagan

03/15/2023, 6:08 PM
how do you make absolute imports work if you have a different dev env than prod
a

Adam Bloom

03/15/2023, 6:12 PM
by actually installing the module in each, so you can load dagster/dagit with the module command. there's tricks for this in poetry, but with raw pip, you have the
-e
flag to install a module as editable (aka just a fancy symlink)
j

Jake Kagan

03/15/2023, 6:17 PM
so, my directory
io_managers
would be treated like a module, you're saying you load dagster with a list of modules? im totally confused
a

Adam Bloom

03/15/2023, 6:25 PM
you can do a list of modules, or do a single module that contains all of your submodules
j

Jake Kagan

03/15/2023, 6:26 PM
shouldn't dagster do that if the modules are inside of it...
btw how is poetry, im using pipfile (with pycharm)