https://dagster.io/ logo
Join Slack
Powered by
We have our dagster code layout where our jobs, re...
# ask-community
d
We have our dagster code layout where our jobs, resources, and sensors are all their own modules. However, we need to access resources and jobs inside of sensors. How can I reference a resource defined in the resources module inside of the job module so our ops have access to them? When I try to just import them, I get an error saying we can't do a relative import above the top level module, do I have a misunderstanding on how python references and imports are handled?
i.e. we have a job that needs to access a sql server resource that we have dynamically generated. when I define in my sensor job op the required resource key, it says it isn't defined. when I try to reference it, I get the error regarding imports. Is there a different way I should be laying out my code to remove this hurdle?
j
are you getting circular dependency issues? if so you can always do the imports inside the sensor method
d
no, I'm trying to define the resources used for a job by just accessing the resources we've already defined in the 
resources
folder. I just noticed in the 
Definitions
API information that
Copy code
Jobs created with @job must already have resources bound at job creation time. They do not respect the resources argument here.
This means that we would need to import the resources in the 
resources
folder, but I don't see a way to do that in our current folder structure do to not being able to do a relative import above the current module, correct? Is there some other workaround or folder layout we should be using? Defining the resources a second time inside the 
sensors
module would mean duplicate code
a
We have our user code structured similarly. How are you invoking dagster-daemon/dagit? You need your top level module to be dagster-code, and I suspect you're a bit deeper down the tree right now
d
we have dagster-daemon, dagit, and our dagster-code container. The working directory in dagster-code is 
/opt/dagster/app/
which is where we copy over our dagster-code structure to.
we have an 
___init___.py
in each folder in our structure. Should we not be doing that?
a
doesn't hurt, but if they're empty also doesn't help
(that changed back in python 3.3)
d
they're not empty, they have the definitions that we want to export to our repo.py at the base level
how do you reference resources from your sensor folder? do you do a relative import?
a
one sec, let me dig up some examples from our user code. I'll note we're using absolute imports and not relative, but either should work
d
our containers are all using python:3.10-slim as the base image. maybe it's an out of date version of python?
a
no, don't think that's it. you still haven't answered what your entrypoint is, and I'm more suspicious of that
as an example, we launch dagster/dagit/etc with the following argument:
Copy code
dagit -m dagster_user_code
(usually managed by the helm chart, but an example from local development)
d
ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "80", "-w", "workspace.yaml"]
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py", "--working-directory", "/opt/dagster/app/"]
CMD ["dagster-daemon", "run"]
those are our entry points for dagit, dagster-code gRPC container, and the daemon in order
a
oh, they're all different? interesting.
We have a top level 
__init__.py
in 
dagster_user_code
, which allows us to simply reference that module. That file imports from submodules, so we don't have a single file equivalent. but starting up with a module vs a file may matter, I can't remember
a sensor at 
dagster_user_code.sensors.sensor_a
has an import like the following:
Copy code
from dagster_user_code.jobs.dbt import dbt_test_doc_job_factory
the trick to all of this is that 
dagster_user_code
needs to be a resolvable module in the interpreter. and since that's how we launch dagster, we know it is
one area of confusion is that python modules can be both installed (in a venv or system site-packages) AND run directly from source. I've found that import errors are both more common (and more confusing!) when running from source
d
we have an empty 
___init__.py_
at the top level of our 
dagster-code
folder. Do we need to add something in there to make it a resolvable module?
a
I suspect you need to get to a place where you're pointing at your 
dagster-code
module, instead of at 
repo.py
, so you might need to modify that file to load the contents of 
repo.py
but, not 100% sure on this either - it's been many months since I set this up, and I haven't had to touch the module structure/imports since then
j
scroll up a few and you'll see what i did
i posted at 10:07AM
d
I'm very new to python so I'm not sure I follow how to use that to use relative imports
here's the specific error we're getting when trying to import a resource to the sensor module:
Copy code
dagster._core.errors.DagsterImportError: Encountered ImportError: `attempted relative import beyond top-level package` while importing module repo from file /opt/dagster/app/repo.py. Local modules were resolved using the working directory `/opt/dagster/app`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for `python_file`-based workspace targets.
a
what's an example of what one of those imports looks like?
if you're resolving from that directory, you should be able to import 
jobs.__
d
from ...resources.sql_resources import SQL_RESOURCES
hm, so I should be able to just do 
from resources.sql_resources import SQL_RESOURCES
?
a
depending on the location of that file, you may have one too many `.`s
but the absolute import should also work
d
holy cow, that worked
I kinda feel like a dummy facepalm Python imports make my head spin. Is the reason for this that the top level 
repo.py
is where the working directory is, so we can just import modules relative to that?
j
image.png
a
so, relative imports work relative to the file doing the importing. they make my head hurt, since the same import will look different in different files (depending on the module depth). I think absolute imports are much cleaner, since the same import looks the same everywhere
j
how do you make absolute imports work if you have a different dev env than prod
a
by actually installing the module in each, so you can load dagster/dagit with the module command. there's tricks for this in poetry, but with raw pip, you have the 
-e
flag to install a module as editable (aka just a fancy symlink)
j
so, my directory 
io_managers
would be treated like a module, you're saying you load dagster with a list of modules? im totally confused
a
you can do a list of modules, or do a single module that contains all of your submodules
j
shouldn't dagster do that if the modules are inside of it...
btw how is poetry, im using pipfile (with pycharm)
2 Views
Open in Slack
PreviousNext
Powered by