Is it possible to select a specific DBT profile target when loading DBT assets in Dagster? https://docs.getdbt.com/docs/get-started/connection-profiles#understanding-targets-in-profiles (not to be confused with the target directory that contains the compilation artifacts). As far as I can tell, you can only specify a

profiles_dir

so it will use the default target defined in

pofiles.yml

. While with the DBT CLI, we can specify a target different from the default by using the

--target

parameter. I think it would be nice to have this

target

parameter in the dagster-dbt API. Our use case is that we have two distinct databases with their own specific DBT assets. A current workaround is to manage two different

profiles_dir

which have each a different

profiles.yml

with a different default target.

This is possible when you configure dbt_cli_resource.

dbt_cli_resource.configured({'profiles_dir': pathToDir, 'project_dir': pathToproject, 'target': yoirPreferredTarget})

You can now go ahead and use this resource to load your dbt assets

Ah yes, I missed it in the documentation: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.dbt_cli_resource Thanks for your help!

Curious if this actually works. When I tried using

load_assets_from_dbt_project

with

with_resources

, the resource isn't used when loading assets You can see this in the dbt command that is run when loading Dagster code (like when dagit starts you can see the dbt command that is run to create the dbt assets). That dbt command doesn't use

--target

even if your resource is configured with a target That matches what I see in the code https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_defs.py#L63-L68 which doesn't include

target

as a flag

I didn't have time to test a use case with different targets yet, but why would the target change the DBT models loaded by Dagster? It is not supposed to select different models, only change the database/schema against which DBT queries will run. If you want to load specific DBT models, you have to list them in the

select

argument of

load_assets_from_dbt_project

.

load_assets_from_dbt_project

uses

dbt ls ...

to load models as assets (by creating a manifest, then parsing it). Like you said in your original message, if you don't specify a target, dbt uses the default target as specified in your

profiles.yml

In my case, I use a different database and credentials for different targets (very common). I don't have the local dev credentials in my production environment. So when Dagster attempts to load models, it fails because it's using the default target and those environment variables don't exist in production

I might be missing something, I'm not following why specifying the target in the dbt resource config isn't working for you. DBT model selection and target selection is two distinct things. When Dagit will load your DBT assets, it will load the DBT models you selected, this is independent of the target, so it makes sense that it doesn't mention the target at this point. When you will materialize your DBT assets, Dagster will start DBT which will run against the database you've configured in the target, either the target you've provided in the DBT resource config, or the default target configured in your DBT profile. Is the issue that your DBT assets change depending on the target? Then you'd have to split your DBT assets into multiple groups, each with its specific DBT model selection and DBT target.

When Dagit will load your DBT assets, it will load the DBT models you selected, this is independent of the target

that's what I'm saying, this isn't true. Dagster needs to generate a manifest in order to parse the dbt models into assets. Dagster creates a manifest by running

dbt ls ...

. Any dbt command uses the default target if it isn't specified

Ok, now I understand what you're saying. It seems to be a limitation on DBT side to not allow picking a target for

dbt ls

. I guess the workaround would be to either have a custom DBT

profiles.yml

for each environment, or maybe if it's possible, have the value of the

target

in

profiles.yml

also taken from an environment variable.

following up on this thread, passing

target

here does work, would the dagster team be open to making target a param? Right now we essentially have forked the

load_assets_from_dbt_project

path just to set that param. for context on why thats currently needed, the bigquery io manager requires the second to last value in the asset key to be the dataset, but we set different datasets in different envs, we use a custom

node_info_to_asset_key

to grab the "custom schema" for the asset key but the issue is the custom schema is always for the dbt profile default instead of the proper target