19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi team! We use dagster’s `load_assets_from_dbt_ma...
# ask-communityh
Harpal
02/08/2023, 5:48 PMHi team! We use dagster’s
load_assets_from_dbt_manifest()dbt seedmanifest.jsonKeyError: 'depends_on'Harpal
02/08/2023, 5:49 PMERROR MSG
Copy code
│ Traceback (most recent call last): │
│ File "/home/ubuntu/pyenv/versions/3.9.8/bin/dagster", line 8, in <module> │
│ sys.exit(main()) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_cli/__init__.py", line 48, in main │
│ cli(auto_envvar_prefix=ENV_PREFIX) # pylint:disable=E1123 │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1137, in __call__ │
│ return self.main(*args, **kwargs) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1062, in main │
│ rv = self.invoke(ctx) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1668, in invoke │
│ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1668, in invoke │
│ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 1404, in invoke │
│ return ctx.invoke(self.callback, **ctx.params) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/click/core.py", line 763, in invoke │
│ return __callback(*args, **kwargs) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_cli/api.py", line 671, in grpc_command │
│ server = DagsterGrpcServer( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/server.py", line 873, in __init__ │
│ self._api_servicer = DagsterApiServer( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/server.py", line 241, in __init__ │
│ self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/server.py", line 104, in __init__ │
│ loadable_targets = get_loadable_targets( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets │
│ else loadable_targets_from_python_file(python_file, working_directory) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_core/workspace/autodiscovery.py", line 27, in loadable_targets_from_python_file │
│ loaded_module = load_python_file(python_file, working_directory) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file │
│ return import_module_from_path(module_name, python_file) │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path │
│ spec.loader.exec_module(module) │
│ File "<frozen importlib._bootstrap_external>", line 850, in exec_module │
│ File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed │
│ File "/machine-learning/moonfire_dagster/repo.py", line 25, in <module> │
│ from moonfire_dagster.north_star_metrics.materialize_daily_metrics import metrics_assets_job, metrics_assets_schedule │
│ File "/machine-learning/moonfire_dagster/north_star_metrics/materialize_daily_metrics.py", line 29, in <module> │
│ dbt_assets = load_assets_from_dbt_manifest( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 646, in load_assets_from_dbt_manifest │
│ dbt_assets_def = _dbt_nodes_to_assets( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 414, in _dbt_nodes_to_assets │
│ deps = _get_deps( │
│ File "/home/ubuntu/pyenv/versions/3.9.8/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 220, in _get_deps │
│ for parent_unique_id in node_info["depends_on"]["nodes"]: │
│ KeyError: 'depends_on'Harpal
02/08/2023, 5:56 PMOFFENDING CODE
Copy code
from dbt_constants import DBT_PROFILE_DIR, DBT_PROJECT_DIR, manifest_file
with open(manifest_file, "r", encoding="utf-8") as f:
manifest_json = json.load(f)
dbt_assets = load_assets_from_dbt_manifest(
use_build_command=True,
manifest_json=manifest_json,
select="tag:not_rq",
)o
owen
02/08/2023, 6:16 PMhi @Harpal! do you happen to be on
dbt-core>=1.4h
Harpal
02/08/2023, 6:35 PMHi @owen. No we're using an older version. We install it via our Dockerfile as follows:
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/dbt-core@v1.1.1#egg=dbt-core&subdirectory=core"
We install dbt postgres similarly with the following
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/dbt-core@v1.1.1#egg=dbt-postgres&subdirectory=plugins/postgres"
What is the stable version you suggest for both of these?
PS: Happy to hear a better way to use these with docker.
o
owen
02/08/2023, 7:01 PMhm I'm not particularly experienced with docker, but I am surprised to see this particular issue show up on dbt version 1.1.1. Any reason you're referencing git instead of something like
pip install dbt==1.1.1h
Harpal
02/08/2023, 7:06 PMYeah our Mac M1 local machines have different dependencies from our deployed Linux containers. So we chose to have the requirements.txt file have the version for our local machines and manually install the version we need for the container this way.
No worries let me SSH into the container and double check which version is running in container itself. Perhaps it's being overwritten with version 1.4 somewhere.
130 Views