https://dagster.io/ logo
Join Slack
Powered by
my `pyproject.yaml` has: ```[tool.dagster] package...
# ask-community
r
my 
pyproject.yaml
has:
Copy code
[tool.dagster]
package_name = "cureatr"
When I run 
dagster dev
I don't see any jobs etc. in the UI. If I change that to 
module_name = "<http://cureatr.jobs|cureatr.jobs>"
then I see my jobs. So how does 
package_name
work, it doesn't enumerate jobs/assets etc. in modules in that package?
d
It doesn't automatically scan subfolders in the package, no - it will load the package (just like if you ran 
python -m cureatr
and include any Dagster definitions that are defined as part of that load)
r
my package 
__init__.py
loads my jobs 
from .jobs import *
so the jobs should be defined in the package
d
Hm, I would expect that to work in that case. Are you using a Definitions object?
r
no, I just have some functions decorated with 
@job
- if my pyproject.yaml uses 
module_name = "<http://cureatr.jobs|cureatr.jobs>"
I see them , but if it uses 
package_name = "cureatr"
it does not
If I just import my package in python I see my job functions defined
Copy code
>>> import cureatr
>>> dir(cureatr)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'hello_redshift', 'hello_redshift_job', 'job', 'jobs', 'ops']
d
hm so i would have expected an error like this if you point it at a module with multiple jobs and no Definitions object:
Copy code
Error loading repository location hi.py:dagster._core.errors.DagsterInvariantViolationError: No repository and more than one job found in "hi". If you load a file or module directly it must have only one job in scope. Found jobs defined in variables or decorated functions: ['job1', 'job2'].
that's what i got just now when I tried
r
oh, actually it just has one job, and that job references an asset. So I guess this approach won't work for >1 job? I'm trying to simplify local development for our datascience engineers and so if they have to add all this declarative boilerplate any time they add a function thats going to be an issue. So I guess I would need to introspect my package and dynamically build a bunch of Definitions?
ok, I guess I just need one Definition in the package, and they need to add any jobs/assets etc. to it? I guess that's ok
d
That's right, yeah
r
hmm, with 
Definitions
I still don't see my job. Do I have to add a code location when using 
package_name
instead of 
module_name
?
the UI shows
Copy code
No definitions
When you add a code location, your definitions will appear here
with 
module_name
I did not have to add a code location, it found my job
d
hm module_name and package_name are identical so I don't think its that
the expected behavior here is that as long as the Definitions object is loaded as part of the import, it should be shown in the UI
r
with module_name it shows up without Definitions module_name vs package_name
d
oh I'm sorry, I know what's happening here
pyproject.toml only accepts module_name, not package_name - I was confused
r
aha - ok, so should I just put this in a local dagster.yaml?
oh, or is it workspace.yaml or something?
d
There's no functionality difference between module_name and package_name - if you change your pyproject.yaml to say module_name it should work
I just incorrectly assumed they were interchangable
r
yep, that worked! thanks
condagster 1
65 Views
Open in Slack
PreviousNext
Powered by