In Dagster running on ECS I have an env var `dagst...
# integration-dbt
In Dagster running on ECS I have an env var
dagsterSecret={ lot of stuff, "bigquery_creds"="JSON STUFF"}
. I am trying to set up DBT in dagster that connects to BigQuery and use this auth the nested env var for authentication. Does anyone have any notes, or point me to a resource on how this typically works? I will have almost the same question but for Redshift too
Thanks, @rex. I am doing what the dbt docs say and loading from
and using env variables from ECS that are tagged
But the problem is it blows up on the usercode server because the variables are not set there, they are only set on the run instances. Trying to understand how that is supposed to work when using Dagster in ECS with DBT.
Do you have error logs from the user code server? Could you post them here?
@rex Yes, This is my config/profiles.yml which works if
is set:
Copy code
  send_anonymous_usage_stats: False

  target: dev

      type: bigquery
      method: service-account-json
      project: product-analytics-prod
      dataset: dagster_testing_euw1
      threads: 1
      connect_timeout: 999999

        type: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['service_account'] }}"
        project_id: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['project_id'] }}"
        private_key_id: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['private_key_id'] }}"
        private_key: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['private_key'] }}"
        client_email: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['client_email'] }}"
        client_id: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['client_id'] }}"
        auth_uri: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['auth_uri'] }}"
        token_uri: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['token_uri'] }}"
        auth_provider_x509_cert_url: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['auth_provider_x509_cert_url'] }}"
        client_x509_cert_url: "{{ fromjson(fromjson(env_var('dagsterSecret'))['bq_credentials'])['client_x509_cert_url'] }}"
This is the error:
Copy code
August 10, 2023 at 19:14 (UTC-5:00)	dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2): Running with dbt=1.5.4 Encountered an error:	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	Parsing Error	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	Env var required but not provided: 'dagsterSecret'	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	manifest, cli_output = _load_manifest_for_project(	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	^^^^^^^^^^^^^^^^^^^^^^^^^^^	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	File "/usr/local/lib/python3.11/site-packages/dagster_dbt/", line 84, in _load_manifest_for_project	59641355e51047b098011e43d2e91444	usercode
August 10, 2023 at 19:14 (UTC-5:00)	cli_output = execute_cli(
That is because Dagster EcsRunLauncher only loads AWS secrets tagged as
on the run server and not on the usercode server
I am not sure why it is even getting loaded on the user-code server in the first place. I have essentially followed this project_fully_feature but using BigQuery instead of DuckDB. And it works if I hard code the env var on the usercode server. But that should not be needed.
It's because you're using
, which compiles the manifest at runtime. So this requires your code server to have the profile with env vars in order to compile the manifest. If you switch to
, or
, you won't need to to run a dbt command at runtime to generate your dbt assets. Instead, we'll use the precompiled manifest that you provide: Could you try that?
@rex, will be glad to try. Where would I put
in project_fully_feature project to do that?
already uses `load_assets_from_dbt_manifest`: But you would put it in this file.
👍 1
@rex, Yes, that works. But I guess the downside is it now depends on how dbt model is built when I create my docker image instead of dynamically getting configured based on the runtime env.
One interesting thing is the credentials on the the "usercode" server do not even have to be correct. I would really like to keep using
This solves the problem but I think there is a bigger problem of usercode invoking things but there is no way to get AWS secrets onto the usercode server:
Copy code
type: "{{                        fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['service_account'] }}"
        project_id: "{{                  fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['project_id'] }}"
        private_key_id: "{{              fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['private_key_id'] }}"
        private_key: "{{                 fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['private_key'] }}"
        client_email: "{{                fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['client_email'] }}"
        client_id: "{{                   fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['client_id'] }}"
        auth_uri: "{{                    fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['auth_uri'] }}"
        token_uri: "{{                   fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['token_uri'] }}"
        auth_provider_x509_cert_url: "{{ fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['auth_provider_x509_cert_url'] }}"
        client_x509_cert_url:  "{{       fromjson(fromjson(env_var('dagsterSecret', ''), '{}')['bq_credentials'], '{}')['client_x509_cert_url'] }}"
Updated this ticket with the info #15770