https://dagster.io/ logo
#ask-community
Title
# ask-community
g

geoHeil

01/04/2023, 7:59 PM
When trying to transition to the Definitons API with GRPC servers:
Copy code
# old repository:
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "FOO/repository.py", "--attribute", "myrepo"]
# ==> works fine

# new definitions:
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-m", "package_name"]
# ==> fails with: =>

DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, asset groups, or asset definitions found in "package_name"

CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-m", "package_name.package_name"]
# ==> fails with: => No module named 'package_name/package_name'` while importing module package_name/package_name
Similarly to https://github.com/slopp/dagteam/blob/main/workspace.yaml: my workspace.yaml looks like:
Copy code
load_from:
  - grpc_server:
      host: package_name
      port: 4000
      location_name: "package_name"
inside docker. But outside docker (during local development):
Copy code
load_from:
  - python_package:
      package_name: package_name.package_name
Outside docker a
pip install -e .
has been executed - and the modules are resolvable. Inside docker, it would be convenient to not first volume-map/copy the file and then additionally pip install -e . it. It would be uesful (I think also for speedier reloading) to be able to pass a file like before: "-f", "FOO/repository.py". Is there any chance I can get this convenient behavior back when using definitions? My package (scaffolded from dagster cli as package_name/package_name/assets.py and package_name/package_name/__init__.py) contains in the init:
Copy code
from . import assets
defs = Definitions(assets=(load_assets_from_modules([assets])))
However, I do not find any good way to specify the init file in the
-f
parameter. There, dagster also fails to resolve it due to missing modules.
@Ben Gotow & @Sean Lopp can you also look at this?
This problem is also reproducible when including a pip install -e . in the dockerfile.
I just re-converted all to a repository-based approach. This is working much better. Please can you assist me to fix it for the new definitions API.
However: I think both seem to have the same problem (which I could solve by now). Irrespective of Definitions or repository api. Both can be supplied with the
-f
parameter as a file. And for both, One MUST omit relative imports in the initial file (irrespective if it is the
__init__
or
repository
file. With this both variants are now working for me
s

Sean Lopp

01/04/2023, 8:54 PM
I don't think this behavior has anything to do with the definitions API. e.g. if you created a file called
definitions.py
you could use it similar to your
repository.py
file and include in
__init__.py
something like
from .definitions import defs
I believe what you are seeing is expected differences in how python resolves relative imports in installed packages vs un-installed packages
D 1
g

geoHeil

01/05/2023, 7:16 AM
@Sean Lopp https://dagster.slack.com/archives/C01U954MEER/p1672902982185429 might be something similar. But I am still super confused due to the XOR behavior
15 Views