Hello, is there any documentation or example on how to use DynamicPartitionsDefinition and dynamic_partitioned_config? I’m using this example but don’t know how to integrate the dynamic_partitioned_config part in. (used for populating dbt run config)

hi @Peter Lim, it's confusing, but the dynamic_partitioned_config refers to a deprecated type of DynamicPartitionsDefinition (which involved returning partitions from a custom function) rather than the current type (which stores and retrieves partition names from the instance database). are you using the

load_assets_from_dbt_manifest

integration or are you running dbt from an op? if the former, I'd recommend just setting up a

partition_key_to_vars_fn

to convert the given partition key to variables that will be passed into your dbt project. If the latter, you can get the current partition key at runtime from

context.partition_key

, then use that to construct arguments to pass into your dbt invocation

Hi Owen! I’m using

load_assets_from_dbt_manifest

and your solution with

partition_key_to_vars_fn

worked. Is there a way to expose required config at the launcher level (i.e select partition and it scaffolds the config mapping) to dbt? I have this but looking for how config gets passed. I used to have

@static_partitioned_config

attached to it. Thank you!!

@configured(dbt_cli_resource, config_schema={"vars": dict,"target": str})
def custom_dbt_cli_resource(config):
    return {
        "vars": config["vars"],
        "project-dir": DBT_PROJECT_PATH,
        "profiles-dir": DBT_PROFILES_PATH,
        "target": config["target"]
    }

hi @Peter Lim -- there's not a great way to do this via the load_assets_from_dbt_manifest entrypoint, but this is the sort of usecase that the new dbt_assets decorator is useful for: https://github.com/dagster-io/dagster/discussions/14477. You can use that to define a custom config schema on the decorated function. Then, when you attempt to materialize an asset, the UI will prompt you to fill out the missing config

Hi @owen, sorry for coming back to the thread but do you have an example of running dbt using ops?

If the latter, you can get the current partition key at runtime from

context.partition_key

, then use that to construct arguments to pass into your dbt invocation

I’m trying to invoke runs using ops instead of assets using dagster 1.3, and a lot of features I want to use are tied to ops instead of

define_asset_job

Thank you!!

which features in particular, out of curiosity? any op-based dbt execution just consists of invoking the DbtCliClient, i.e.

@op
def do_dbt_stuff(context, dbt: DbtCliClient):
    dbt.run(<args go here>, vars={"some_var": context.partition_key})

If

dynamic_partitioned_config

is deprecated, should we just use a regular PartitionedConfig for supplying run config to partitioned assets that use the new DynamicPartitionsDefinition (the one based off of the instance db)

From my perspective, partitioned config was historically used to translate a partition key into some specific values that could then be used within an op / asset. Nowadays, the partition key is available directly on the context, and so can be translated directly into usable values there, rather than going through a layer of indirection (this is now the recommended pattern). Would this sort of setup work for you?

Yes, that works well for me! Thanks for the explanation, now I better understand the design choices made there

Hi @owen, sorry I was out last week - expounded our use case here https://dagster.slack.com/archives/C04CW71AGBW/p1692041199995659 In addition to your snippet above, is there any way to specify asset groups in the dbt run in order to slice them up? Thank you!