Hi everyone, we are using dbt with Elementary. Dagster is running the following command:

dbt build --project-dir $PROJECT_DIR --profiles-dir $PROFILES_DIR --vars {"elementary_schema": "my_schema", "elementary_enabled": true} --select my_model

Which succeeds if we run it manually against our dbt project. But fails in Dagster with the following error:

KeyError: 'operation.elementary.elementary-on-run-start-0'
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster/_utils/__init__.py", line 443, in iterate_with_context
    next_output = next(iterator)
                  ^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 309, in _dbt_op
    yield from _stream_event_iterator(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 241, in _stream_event_iterator
    yield from _events_for_structured_json_line(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 174, in _events_for_structured_json_line
    compiled_node_info = manifest_json["nodes"][unique_id]
                         ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^

It looks like Dagster struggles to run the dbt pre-hook created by Elementary. Has anyone faced the same issue ?

Hey Romain, are you using

load_assets_from_dbt_project

or

load_assets_from_dbt_manifest

?

Hey Rex, we are using

load_assets_from_dbt_project

. We are now trying to use the new dbt_assets API that seems to be a solution to our issue. But using dagster==1.3.10 and dagster-dbt==0.19.10, and the following snippet:

from dagster_dbt.asset_decorator import dbt_assets
from dagster_dbt.cli import DbtCli, DbtManifest

manifest = DbtManifest.read(path=MANIFEST_PATH)
@dbt_assets(manifest=manifest)
def my_dbt_assets(context: OpExecutionContext, dbt: DbtCli):
    yield from dbt.cli(["build"], manifest=manifest, context=context).stream()

We get the following error:

dagster._check.ParameterCheckError: Param "command" is not a str. Got ['build'] which is type <class 'list'>.
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster/_utils/__init__.py", line 445, in iterate_with_context
    next_output = next(iterator)
                  ^^^^^^^^^^^^^^
  File "/Users/romrob/work/group-data-team/smg-data-group-ibiza/dagster/nprd/gdt/gdt/assets/__init__.py", line 42, in my_dbt_assets
    yield from dbt.cli(["build"], manifest=manifest, context=context).stream()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster_dbt/cli/resources.py", line 204, in cli
    command = check.str_param(command, "command")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dagster/_check/__init__.py", line 1347, in str_param
    raise _param_type_mismatch_exception(obj, str, param_name, additional_message)

in your Definitions object, you should make sure you’re creating a DbtCli resource.

It’s working better now indeed; thank you Rex!