https://dagster.io/ logo
#dagster-dbt
Title
# dagster-dbt
g

geoHeil

12/13/2022, 2:35 PM
How can I pass (arbitrary variables i.e. lets say a date variable) to DBT run on DBT core orchestrated via dagster?
r

rex

12/13/2022, 2:36 PM
You can supply a
partitions_def
and
partition_key_to_vars_fn
to the function that generates your dbt assets
a

Adam Bloom

12/13/2022, 2:36 PM
Are you using assets or ops?
g

geoHeil

12/13/2022, 2:37 PM
assets
no not for the partitions def - but it might be a way to parse arbitrary variables here
a

Adam Bloom

12/13/2022, 2:38 PM
@rex’s answer is the way, even if the assets don’t need to be partitioned. There isn’t currently another way.
Note that you can use a StaticPartitionsDefinition with a single partition to make this work
👍🏽 1
👍 1
o

owen

12/13/2022, 8:49 PM
Another option would be to define a config-mapping over the dbt_cli_resource to fill in the values that won't change, while allowing you to configure the vars parameter after the fact:
Copy code
@configured(dbt_cli_resource, config_schema={"vars": dict})
def custom_dbt_cli_resource(config):
    return {
        "vars": config["vars"],
        "project-dir": DBT_PROJECT_DIR,
        "profiles-dir": DBT_PROFILES_DIR
    }
a

Archit Raj

02/13/2023, 6:55 AM
@owen After making this dbt_cli_resource how to make a job out of it Please help me in this
o

owen

02/13/2023, 5:14 PM
You can do something like: https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/part-two#step-2-define-a-dagster-code-location, but instead of the default
dbt_cli_resource
, you can pass in your
custom_dbt_cli_resource
(i.e.
resources = {"dbt": custom_dbt_cli_resource}
)
a

Archit Raj

02/13/2023, 5:41 PM
@owen Great , This step worked After this , i want to make it as a job and then pass the parameter using launch pad . Please assist
o

owen

02/14/2023, 12:38 AM
if you create a job to materialize your assets using
define_asset_job(...)
then attempting to launch a run for that job in the UI should prompt you to fill in configuration for the vars parameter
g

geoHeil

03/02/2023, 9:09 PM
Does this mean if I have daily and monthly partitioned tables I need to tag them and load them separately? Or how can I map the partitionsdef to potentially > 1 type of partitioning in the partition_key_to_vars_fn function? I.e. I would not want to trigger the monthly one on a daily basis.
o

owen

03/02/2023, 11:26 PM
ah yeah good question -- the short answer is that yes you'd need to invoke
load_assets_from_dbt...
multiple times, once for each subset of assets with a different partitions definition
9 Views