19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello Dagster Team,
I’m currently working on a PoC...
# integration-dbts
Si Wang
08/20/2023, 11:55 PMHello Dagster Team,
I’m currently working on a PoC to integrate Dagster with dbt for our team. I’ve been following the instructions to load an existing dbt model using the command:
Copy code
dagster-dbt project scaffold --project-name dagster_onep --dbt-project-dir dbt_workflowsKeyError: ‘outputs’profile.yml🤖 1
p
Pedram Navid
08/21/2023, 12:56 AMthis sounds like a problem with the
profiles.yamldbt debug Copy code
dbt_proj_name:
target: dev
outputs:
dev:
type: duckdbs
Si Wang
08/21/2023, 2:03 AMthe dbt model runs perfectly by its on, I have looked at the details of the error message, it seems that dagster could not parse the
outputsSi Wang
08/21/2023, 2:03 AM Copy code
dbt_workflows:
target: dev
outputs:
dev:
dataset: headroom_dev
job_execution_timeout_seconds: 3600Si Wang
08/21/2023, 2:04 AMthis is a partial config from profiles.yml
p
Pedram Navid
08/21/2023, 2:49 AMjust to confirm, the
profiles.ymldbt_workflowss
Si Wang
08/21/2023, 2:50 AMyes it is
p
Pedram Navid
08/21/2023, 2:51 AMCould you provide the full traceback from the exception?
s
Si Wang
08/21/2023, 2:52 AM╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/SW/.pyenv/versions/3.9.2/lib/python3.9/site-packages/dagster_dbt/cli/app.py:189 in │
│ project_scaffold_command │
│ │
│ 186 │ │
│ 187 │ dagster_project_dir = Path.cwd().joinpath(project_name) │
│ 188 │ │
│ ❱ 189 │ copy_scaffold( │
│ 190 │ │ project_name=project_name, │
│ 191 │ │ dagster_project_dir=dagster_project_dir, │
│ 192 │ │ dbt_project_dir=dbt_project_dir, │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ console = <console width=223 ColorSystem.TRUECOLOR> │ │
│ │ dagster_project_dir = PosixPath(‘/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_project_dir = PosixPath(’/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ project_name = ‘dagster_onep’ │ │
│ │ use_dbt_project_package_data_dir = False │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/SW/.pyenv/versions/3.9.2/lib/python3.9/site-packages/dagster_dbt/cli/app.py:88 in │
│ copy_scaffold │
│ │
│ 85 │ │
│ 86 │ dbt_profiles_path = dbt_project_dir.joinpath(DBT_PROFILES_YML_NAME) │
│ 87 │ dbt_profiles_yaml: Dict[str, Any] = yaml.safe_load(dbt_profiles_path.read_bytes()) │
│ ❱ 88 │ dbt_adapter_packages = [ │
│ 89 │ │ dbt_adapter_pypi_package_for_target_type(target[“type”]) │
│ 90 │ │ for profile in dbt_profiles_yaml.values() │
│ 91 │ │ for target in profile[“outputs”].values() │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ dagster_project_dir = PosixPath(‘/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_profiles_path = PosixPath(’/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_profiles_yaml = { │ │
│ │ │ ‘config’: { │ │
│ │ │ │ ‘send_anonymous_usage_stats’: False, │ │
│ │ │ │ ‘use_colors’: False, │ │
│ │ │ │ ‘partial_parse’: False, │ │
│ │ │ │ ‘printer_width’: 120, │ │
│ │ │ │ ‘warn_error’: True, │ │
│ │ │ │ ‘fail_fast’: True │ │
│ │ │ }, │ │
│ │ │ ‘dbt_workflows’: { │ │
│ │ │ │ ‘target’: ‘dev’, │ │
│ │ │ │ ‘outputs’: { │ │
│ │ │ │ │ ‘dev’: { │ │
│ │ │ │ │ │ ‘dataset’: ‘headroom_dev’, │ │
│ │ │ │ │ │ ‘job_execution_timeout_seconds’: 3600, │ │
│ │ │ │ │ │ ‘job_retries’: 0, │ │
│ │ │ │ │ │ ‘location’: ‘US’, │ │
│ │ │ │ │ │ ‘method’: ‘oauth’, │ │
│ │ │ │ │ │ ‘priority’: ‘interactive’, │ │
│ │ │ │ │ │ ‘project’: ‘gcp-wow-rwds-ai-onep-dev’, │ │
│ │ │ │ │ │ ‘threads’: 20, │ │
│ │ │ │ │ │ ‘type’: ‘bigquery’ │ │
│ │ │ │ │ }, │ │
│ │ │ │ │ ‘prod’: { │ │
│ │ │ │ │ │ ‘dataset’: ‘headroom_dev’, │ │
│ │ │ │ │ │ ‘job_execution_timeout_seconds’: 3600, │ │
│ │ │ │ │ │ ‘job_retries’: 0, │ │
│ │ │ │ │ │ ‘location’: ‘US’, │ │
│ │ │ │ │ │ ‘method’: ‘oauth’, │ │
│ │ │ │ │ │ ‘priority’: ‘interactive’, │ │
│ │ │ │ │ │ ‘project’: ‘gcp-wow-rwds-ai-onep-dev’, │ │
│ │ │ │ │ │ ‘threads’: 20, │ │
│ │ │ │ │ │ ‘type’: ‘bigquery’ │ │
│ │ │ │ │ } │ │
│ │ │ │ } │ │
│ │ │ } │ │
│ │ } │ │
│ │ dbt_project_dir = PosixPath(’/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_project_name = ‘dbt_workflows’ │ │
│ │ dbt_project_yaml = { │ │
│ │ │ ‘name’: ‘dbt_workflows’, │ │
│ │ │ ‘version’: ‘1.0.0’, │ │
│ │ │ ‘config-version’: 2, │ │
│ │ │ ‘profile’: ‘dbt_workflows’, │ │
│ │ │ ‘model-paths’: [‘models’], │ │
│ │ │ ‘analysis-paths’: [‘analyses’], │ │
│ │ │ ‘test-paths’: [‘tests’], │ │
│ │ │ ‘seed-paths’: [‘seeds’], │ │
│ │ │ ‘macro-paths’: [‘macros’], │ │
│ │ │ ‘snapshot-paths’: [‘snapshots’], │ │
│ │ │ ... +3 │ │
│ │ } │ │
│ │ dbt_project_yaml_path = PosixPath(’/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ project_name = ‘dagster_onep’ │ │
│ │ use_dbt_project_package_data_dir = False │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/SW/.pyenv/versions/3.9.2/lib/python3.9/site-packages/dagster_dbt/cli/app.py:91 in │
│ <listcomp> │
│ │
│ 88 │ dbt_adapter_packages = [ │
│ 89 │ │ dbt_adapter_pypi_package_for_target_type(target[“type”]) │
│ 90 │ │ for profile in dbt_profiles_yaml.values() │
│ ❱ 91 │ │ for target in profile[“outputs”].values() │
│ 92 │ ] │
│ 93 │ │
│ 94 │ if use_dbt_project_package_data_dir: │
│ │
│ ╭─────────────────────── locals ───────────────────────╮ │
│ │ .0 = <dict_valueiterator object at 0x1075c89f0> │ │
│ │ profile = { │ │
│ │ │ ‘send_anonymous_usage_stats’: False, │ │
│ │ │ ‘use_colors’: False, │ │
│ │ │ ‘partial_parse’: False, │ │
│ │ │ ‘printer_width’: 120, │ │
│ │ │ ‘warn_error’: True, │ │
│ │ │ ‘fail_fast’: True │ │
│ │ } │ │
│ ╰──────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: ‘outputs’
Si Wang
08/21/2023, 2:52 AMimage.png
Si Wang
08/21/2023, 2:53 AMimage.png
p
Pedram Navid
08/21/2023, 2:54 AMIf you format the errors as code, that will format it better. You can use three backticks
Copy code
```s
Si Wang
08/21/2023, 2:54 AM Copy code
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/SW/.pyenv/versions/3.9.2/lib/python3.9/site-packages/dagster_dbt/cli/app.py:189 in │
│ project_scaffold_command │
│ │
│ 186 │ │
│ 187 │ dagster_project_dir = Path.cwd().joinpath(project_name) │
│ 188 │ │
│ ❱ 189 │ copy_scaffold( │
│ 190 │ │ project_name=project_name, │
│ 191 │ │ dagster_project_dir=dagster_project_dir, │
│ 192 │ │ dbt_project_dir=dbt_project_dir, │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ console = <console width=223 ColorSystem.TRUECOLOR> │ │
│ │ dagster_project_dir = PosixPath('/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_project_dir = PosixPath('/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ project_name = 'dagster_onep' │ │
│ │ use_dbt_project_package_data_dir = False │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/SW/.pyenv/versions/3.9.2/lib/python3.9/site-packages/dagster_dbt/cli/app.py:88 in │
│ copy_scaffold │
│ │
│ 85 │ │
│ 86 │ dbt_profiles_path = dbt_project_dir.joinpath(DBT_PROFILES_YML_NAME) │
│ 87 │ dbt_profiles_yaml: Dict[str, Any] = yaml.safe_load(dbt_profiles_path.read_bytes()) │
│ ❱ 88 │ dbt_adapter_packages = [ │
│ 89 │ │ dbt_adapter_pypi_package_for_target_type(target["type"]) │
│ 90 │ │ for profile in dbt_profiles_yaml.values() │
│ 91 │ │ for target in profile["outputs"].values() │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ dagster_project_dir = PosixPath('/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_profiles_path = PosixPath('/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_profiles_yaml = { │ │
│ │ │ 'config': { │ │
│ │ │ │ 'send_anonymous_usage_stats': False, │ │
│ │ │ │ 'use_colors': False, │ │
│ │ │ │ 'partial_parse': False, │ │
│ │ │ │ 'printer_width': 120, │ │
│ │ │ │ 'warn_error': True, │ │
│ │ │ │ 'fail_fast': True │ │
│ │ │ }, │ │
│ │ │ 'dbt_workflows': { │ │
│ │ │ │ 'target': 'dev', │ │
│ │ │ │ 'outputs': { │ │
│ │ │ │ │ 'dev': { │ │
│ │ │ │ │ │ 'dataset': 'headroom_dev', │ │
│ │ │ │ │ │ 'job_execution_timeout_seconds': 3600, │ │
│ │ │ │ │ │ 'job_retries': 0, │ │
│ │ │ │ │ │ 'location': 'US', │ │
│ │ │ │ │ │ 'method': 'oauth', │ │
│ │ │ │ │ │ 'priority': 'interactive', │ │
│ │ │ │ │ │ 'project': 'gcp-wow-rwds-ai-onep-dev', │ │
│ │ │ │ │ │ 'threads': 20, │ │
│ │ │ │ │ │ 'type': 'bigquery' │ │
│ │ │ │ │ }, │ │
│ │ │ │ │ 'prod': { │ │
│ │ │ │ │ │ 'dataset': 'headroom_dev', │ │
│ │ │ │ │ │ 'job_execution_timeout_seconds': 3600, │ │
│ │ │ │ │ │ 'job_retries': 0, │ │
│ │ │ │ │ │ 'location': 'US', │ │
│ │ │ │ │ │ 'method': 'oauth', │ │
│ │ │ │ │ │ 'priority': 'interactive', │ │
│ │ │ │ │ │ 'project': 'gcp-wow-rwds-ai-onep-dev', │ │
│ │ │ │ │ │ 'threads': 20, │ │
│ │ │ │ │ │ 'type': 'bigquery' │ │
│ │ │ │ │ } │ │
│ │ │ │ } │ │
│ │ │ } │ │
│ │ } │ │
│ │ dbt_project_dir = PosixPath('/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ dbt_project_name = 'dbt_workflows' │ │
│ │ dbt_project_yaml = { │ │
│ │ │ 'name': 'dbt_workflows', │ │
│ │ │ 'version': '1.0.0', │ │
│ │ │ 'config-version': 2, │ │
│ │ │ 'profile': 'dbt_workflows', │ │
│ │ │ 'model-paths': ['models'], │ │
│ │ │ 'analysis-paths': ['analyses'], │ │
│ │ │ 'test-paths': ['tests'], │ │
│ │ │ 'seed-paths': ['seeds'], │ │
│ │ │ 'macro-paths': ['macros'], │ │
│ │ │ 'snapshot-paths': ['snapshots'], │ │
│ │ │ ... +3 │ │
│ │ } │ │
│ │ dbt_project_yaml_path = PosixPath('/Users/SW/woolies_git/thecounts-onep-headroom… │ │
│ │ project_name = 'dagster_onep' │ │
│ │ use_dbt_project_package_data_dir = False │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /Users/SW/.pyenv/versions/3.9.2/lib/python3.9/site-packages/dagster_dbt/cli/app.py:91 in │
│ <listcomp> │
│ │
│ 88 │ dbt_adapter_packages = [ │
│ 89 │ │ dbt_adapter_pypi_package_for_target_type(target["type"]) │
│ 90 │ │ for profile in dbt_profiles_yaml.values() │
│ ❱ 91 │ │ for target in profile["outputs"].values() │
│ 92 │ ] │
│ 93 │ │
│ 94 │ if use_dbt_project_package_data_dir: │
│ │
│ ╭─────────────────────── locals ───────────────────────╮ │
│ │ .0 = <dict_valueiterator object at 0x1075c89f0> │ │
│ │ profile = { │ │
│ │ │ 'send_anonymous_usage_stats': False, │ │
│ │ │ 'use_colors': False, │ │
│ │ │ 'partial_parse': False, │ │
│ │ │ 'printer_width': 120, │ │
│ │ │ 'warn_error': True, │ │
│ │ │ 'fail_fast': True │ │
│ │ } │ │
│ ╰──────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'outputs'Si Wang
08/21/2023, 2:55 AMThanks a lot for prompt response
p
Pedram Navid
08/21/2023, 3:01 AMah, can you remove the
config:profiles.ymlPedram Navid
08/21/2023, 3:02 AMlooks like a bug where we didn't account for the
configprofiles.yamlr
rex
08/21/2023, 2:25 PMThanks for the report Si: this should be resolved in this week’s release with https://github.com/dagster-io/dagster/pull/15951.
s
Si Wang
08/21/2023, 8:26 PMThanks both I have the dagster up and running!
🚀 2
3 Views