https://dagster.io/ logo
Title
s

Szymon Piskorz

01/25/2023, 11:29 PM
Hello. I am trying to load a dbt manifest into dagster. Its a rather large one - 110MB. I had verified that on level of object returned by json.load() it has all of its data. When calling load_assets_from_modules(json_obj) I get the following:
/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_core/workspace/context.py:588: UserWarning: Error loading repository location repo.py:ValueError: None is not a valid IndirectSelection

Stack Trace:
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_grpc/server.py", line 256, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_grpc/server.py", line 114, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 28, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster/_seven/__init__.py", line 53, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jovyan/work/dagster_demo/repo.py", line 4, in <module>
    import assets
  File "/home/jovyan/work/dagster_demo/assets/__init__.py", line 3, in <module>
    dbt_assets = load_assets_from_dbt_manifest(json.load(open("manifest.json", 'r')))
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 742, in load_assets_from_dbt_manifest
    selected_unique_ids = _select_unique_ids_from_manifest_json(manifest_json, select, exclude)
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 122, in _select_unique_ids_from_manifest_json
    parsed_spec: SelectionSpec = graph_cli.parse_union([select], True)
  File "/home/jovyan/.cache/pypoetry/virtualenvs/dagster-demo-ahNgZajK-py3.8/lib/python3.8/site-packages/dbt/graph/cli.py", line 47, in parse_union
    indirect_selection=IndirectSelection(flags.INDIRECT_SELECTION),
  File "/opt/conda/lib/python3.8/enum.py", line 339, in __call__
    return cls.__new__(cls, value)
  File "/opt/conda/lib/python3.8/enum.py", line 663, in __new__
    raise ve_exc

  warnings.warn(
Can anyone help me troubleshoot that?
:rainbow-daggy: 1
Here is the code for loading assets in
./assets/__init__.py
from dagster_dbt import load_assets_from_dbt_manifest
import json
dbt_assets = load_assets_from_dbt_manifest(json.load(open("manifest.json", 'r')))
Here is the `./repo.py`:
from dagster import load_assets_from_package_module, repository, with_resources
from dagster_dbt import dbt_cli_resource

import assets


@repository
def tutorial_dbt_dagster():
    return with_resources(
        load_assets_from_package_module([assets]),
        {
            "dbt": dbt_cli_resource.configured(
                {
                    "project_dir": "../GitRepos/data_products/common/dbt",
                    "profiles_dir": "../GitRepos/data_products/common/dbt",
                },
            ),
        },
    )