In Dagster running on ECS I have an env var

dagsterSecret={ lot of stuff, "bigquery_creds"="JSON STUFF"}

. I am trying to set up DBT in dagster that connects to BigQuery and use this auth the nested env var for authentication. Does anyone have any notes, or point me to a resource on how this typically works? I will have almost the same question but for Redshift too

You should rely on how dbt accepts this configuration: • Bigquery: https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup#authentication-methods • Redshift: https://docs.getdbt.com/docs/core/connect-data-platform/redshift-setup#authentication-methods

Thanks, @rex. I am doing what the dbt docs say and loading from

config/profiles.yml

and using env variables from ECS that are tagged

dagster

But the problem is it blows up on the usercode server because the variables are not set there, they are only set on the run instances. Trying to understand how that is supposed to work when using Dagster in ECS with DBT.

Do you have error logs from the user code server? Could you post them here?

@rex Yes, This is my config/profiles.yml which works if

dagsterSecrets

is set:

config:
  send_anonymous_usage_stats: False

bigquery:
  target: dev

  outputs:
    dev:
      type: bigquery
      method: service-account-json
      project: product-analytics-prod
      dataset: dagster_testing_euw1
      threads: 1
      connect_timeout: 999999

      keyfile_json:
        type: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['service_account'] }}"
        project_id: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['project_id'] }}"
        private_key_id: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['private_key_id'] }}"
        private_key: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['private_key'] }}"
        client_email: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['client_email'] }}"
        client_id: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['client_id'] }}"
        auth_uri: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['auth_uri'] }}"
        token_uri: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['token_uri'] }}"
        auth_provider_x509_cert_url: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['auth_provider_x509_cert_url'] }}"
        client_x509_cert_url: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['client_x509_cert_url'] }}"

This is the error:

:
                :
                :
August 10, 2023 at 19:14 (UTC-5:00)	dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2): Running with dbt=1.5.4 Encountered an error:	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	Parsing Error	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	Env var required but not provided: 'dagsterSecret'	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	manifest, cli_output = _load_manifest_for_project(	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	^^^^^^^^^^^^^^^^^^^^^^^^^^^	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 84, in _load_manifest_for_project	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	cli_output = execute_cli(
                :
                :
                :

It's because you're using

load_assets_from_dbt_project

, which compiles the manifest at runtime. So this requires your code server to have the profile with env vars in order to compile the manifest. If you switch to

load_assset_from_dbt_manifest

, or

@dbt_assets

, you won't need to to run a dbt command at runtime to generate your dbt assets. Instead, we'll use the precompiled manifest that you provide: https://docs.dagster.io/integrations/dbt/reference#loading-models-using-load_assets_from_dbt_manifest. Could you try that?

@rex, will be glad to try. Where would I put

@dbt_assets

in project_fully_feature project to do that?

FYI

project_fully_feature

already uses `load_assets_from_dbt_manifest`: https://github.com/dagster-io/dagster/blob/master/examples/project_fully_featured/project_fully_featured/assets/__init__.py. But you would put it in this file.

@rex, Yes, that works. But I guess the downside is it now depends on how dbt model is built when I create my docker image instead of dynamically getting configured based on the runtime env.