https://dagster.io/ logo
#ask-community
Title
# ask-community
h

Harpal

02/08/2023, 5:48 PM
Hi team! We use dagster’s
load_assets_from_dbt_manifest()
method to load our dbt assets into a dagster. We recently changed a seed (.csv) file, ran the
dbt seed
command, commited the chabnged made to the
manifest.json
file and pushed to master. But now our dagster user deployments pod fails to spin up with this error message:
KeyError: 'depends_on'
Does anyone know what the error message is referring to and how it could be remedied? See the comments for info 👇
ERROR MSG
Copy code
│ Traceback (most recent call last):                                                                                                                                                                                                                │
│   File "/home/ubuntu/pyenv/versions/3.9.8/bin/dagster", line 8, in <module>                                                                                                                                                                       │
│     sys.exit(main())                                                                                                                                                                                                                              │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_cli/__init__.py", line 48, in main                                                                                                                                 │
│     cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123                                                                                                                                                                                    │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1137, in __call__                                                                                                                                      │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                             │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1062, in main                                                                                                                                          │
│     rv = self.invoke(ctx)                                                                                                                                                                                                                         │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1668, in invoke                                                                                                                                        │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                       │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1668, in invoke                                                                                                                                        │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                       │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1404, in invoke                                                                                                                                        │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 763, in invoke                                                                                                                                         │
│     return __callback(*args, **kwargs)                                                                                                                                                                                                            │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_cli/api.py", line 671, in grpc_command                                                                                                                             │
│     server = DagsterGrpcServer(                                                                                                                                                                                                                   │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/server.py", line 873, in __init__                                                                                                                             │
│     self._api_servicer = DagsterApiServer(                                                                                                                                                                                                        │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/server.py", line 241, in __init__                                                                                                                             │
│     self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(                                                                                                                                                                 │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/server.py", line 104, in __init__                                                                                                                             │
│     loadable_targets = get_loadable_targets(                                                                                                                                                                                                      │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets                                                                                                                   │
│     else loadable_targets_from_python_file(python_file, working_directory)                                                                                                                                                                        │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_core/workspace/autodiscovery.py", line 27, in loadable_targets_from_python_file                                                                                    │
│     loaded_module = load_python_file(python_file, working_directory)                                                                                                                                                                              │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file                                                                                                                │
│     return import_module_from_path(module_name, python_file)                                                                                                                                                                                      │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path                                                                                                            │
│     spec.loader.exec_module(module)                                                                                                                                                                                                               │
│   File "<frozen importlib._bootstrap_external>", line 850, in exec_module                                                                                                                                                                         │
│   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed                                                                                                                                                                    │
│   File "/machine-learning/moonfire_dagster/repo.py", line 25, in <module>                                                                                                                                                                         │
│     from moonfire_dagster.north_star_metrics.materialize_daily_metrics import metrics_assets_job, metrics_assets_schedule                                                                                                                         │
│   File "/machine-learning/moonfire_dagster/north_star_metrics/materialize_daily_metrics.py", line 29, in <module>                                                                                                                                 │
│     dbt_assets = load_assets_from_dbt_manifest(                                                                                                                                                                                                   │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 646, in load_assets_from_dbt_manifest                                                                                                      │
│     dbt_assets_def = _dbt_nodes_to_assets(                                                                                                                                                                                                        │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 414, in _dbt_nodes_to_assets                                                                                                               │
│     deps = _get_deps(                                                                                                                                                                                                                             │
│   File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 220, in _get_deps                                                                                                                          │
│     for parent_unique_id in node_info["depends_on"]["nodes"]:                                                                                                                                                                                     │
│ KeyError: 'depends_on'
OFFENDING CODE
Copy code
from dbt_constants import DBT_PROFILE_DIR, DBT_PROJECT_DIR, manifest_file

with open(manifest_file, "r", encoding="utf-8") as f:
    manifest_json = json.load(f)

dbt_assets = load_assets_from_dbt_manifest(
    use_build_command=True,
    manifest_json=manifest_json,
    select="tag:not_rq",
)
o

owen

02/08/2023, 6:16 PM
hi @Harpal! do you happen to be on
dbt-core>=1.4
? there's a known compatibility issue with the newest dbt version, which a fix is going out for this week
h

Harpal

02/08/2023, 6:35 PM
Hi @owen. No we're using an older version. We install it via our Dockerfile as follows: RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/dbt-core@v1.1.1#egg=dbt-core&amp;subdirectory=core" We install dbt postgres similarly with the following RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/dbt-core@v1.1.1#egg=dbt-postgres&amp;subdirectory=plugins/postgres" What is the stable version you suggest for both of these? PS: Happy to hear a better way to use these with docker.
o

owen

02/08/2023, 7:01 PM
hm I'm not particularly experienced with docker, but I am surprised to see this particular issue show up on dbt version 1.1.1. Any reason you're referencing git instead of something like
pip install dbt==1.1.1
? Regardless, this issue should be solved by the fix that's going out tomorrow, so waiting might be the easiest solution
h

Harpal

02/08/2023, 7:06 PM
Yeah our Mac M1 local machines have different dependencies from our deployed Linux containers. So we chose to have the requirements.txt file have the version for our local machines and manually install the version we need for the container this way. No worries let me SSH into the container and double check which version is running in container itself. Perhaps it's being overwritten with version 1.4 somewhere.
97 Views