https://dagster.io/ logo
Powered by
#integration-dbt
Title
# integration-dbt
b

Benedikt Buchert

01/31/2023, 2:33 PM
Hi I'm getting an error when running dbt via dagster:
Copy code
dagster_dbt.errors.DagsterDbtCliOutputsNotFoundError: Expected to find file at path /opt/dagster/app/mydbt/assets/../../mydbt/target/run_results.json
 File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 265, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 382, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 94, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/compute.py", line 177, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn):
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/compute.py", line 145, in _yield_compute_results
    for event in iterate_with_context(
  File "/usr/local/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 457, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 425, in _dbt_op
    dbt_output = DbtOutput(result=dbt_resource.get_run_results_json())
  File "/usr/local/lib/python3.8/site-packages/dagster_dbt/cli/resources.py", line 321, in get_run_results_json
    return parse_run_results(project_dir, target_path)
  File "/usr/local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 171, in parse_run_results
    raise DagsterDbtCliOutputsNotFoundError(path=run_results_path)

The above exception occurred during handling of the following exception:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/dagster/app/mydbt/assets/../../mydbt/target/run_results.json'
File "/usr/local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 168, in parse_run_results
    with open(run_results_path, encoding="utf8") as file:

The above exception occurred during handling of the following exception:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 117: invalid continuation byte

File "/usr/local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 420, in _dbt_op
    dbt_output = dbt_resource.run(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/dagster_dbt/cli/resources.py", line 145, in run
    return self.cli("run", models=models, exclude=exclude, select=select, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dagster_dbt/cli/resources.py", line 90, in cli
    return execute_cli(
  File "/usr/local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 107, in execute_cli
    line = raw_line.decode("utf-8").rstrip()
I'm not sure were to search in my dbt project were it is creating 0xf0
✅ 2
daggy love 1
How do I ensure the encoding is correct.
o

owen

01/31/2023, 5:55 PM
hi @Benedikt Buchert! this is a compatibility issue between 
dagster_dbt
and 
dbt  1.4.x
this will be fixed in this week's release, but until then if you downgrade dbt to < 1.4, this error should be resolved
👍 1
b

Benedikt Buchert

02/01/2023, 6:48 AM
Thank you I will downgrade dbt and test again.
Worked like a charm. Thank you!
l

Lewis Baker

02/08/2023, 5:02 PM
@owen I’m still receiving this error. Using dbt-bigquery==1.0.0
Any idea how I could resolve it?
o

owen

02/08/2023, 5:04 PM
hi! sorry about this -- the compatibility fix slipped to this week's release. the same fix applies here (just pinning dbt to < 1.4) but we have a fix tested and ready to merge for this Thursday
l

Lewis Baker

02/08/2023, 5:46 PM
@owen This still doesn’t seem to be working for me, the error persists
image.png
image.png
Copy code
import glob
from setuptools import find_packages, setup

setup(
    name="assets_dbt_python",
    packages=find_packages(exclude=["assets_dbt_python_tests"]),
    # package data paths are relative to the package key
    package_data={
        "assets_dbt_python": ["../" + path for path in glob.glob("dbt_project/**", recursive=True)]
    },
    install_requires=[
        "dagster",
        "dagster-cloud",
        "boto3",
        "dagster-dbt",
        "pandas",
        "numpy",
        "scipy",
        "dbt-core==1.3.0",
        "dbt-bigquery==1.3.0",
        "dagster-duckdb",
        "dagster-duckdb-pandas",
        # packaging v22 has build compatibility issues with dbt as of 2022-12-07
        # fixing dbt versions due to dagster conflict
        "packaging<22.0",
    ],
    extras_require={"dev": ["dagit", "pytest"]},
)
o

owen

02/08/2023, 5:47 PM
ah that looks like a different error -- are there any logs from the dbt execution that indicate what the root cause might be?
l

Lewis Baker

02/08/2023, 5:48 PM
image.png
image.png
Looks like it’s a dagster_dbt specific error
o

owen

02/08/2023, 5:52 PM
the 
DagsterDbtCliFatalRuntimeError
just wraps the underlying dbt execution error ("`could not find command, ensure it is in the user's PATH: "gcloud"` ) -- looks like 
gcloud
is not available in your environment (might need to add that to your setup.py)
l

Lewis Baker

02/08/2023, 5:54 PM
Okay, thanks for that pointer. I’m using environment variables for the profiles.yml parameters. I have a suspicion that might be the issue
Is there any recommendation on how to correctly set-up the profile parameters? https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/part-one Doesn’t seem to recommend a specific approach
o

owen

02/08/2023, 6:04 PM
as in, you're using 
{{ env(...) }}
inside your profiles.yml? In general, I wouldn't expect that to cause any issues as long as those env variables are defined in the environment you're running in (I've used that setup before with no issues).
25 Views
Powered by