Adam Bloom
06/20/2023, 10:56 PMrex
06/21/2023, 1:03 AMIt’s a bit odd to need the partitions here.With the new APIs for declaring dbt SDA’s using
@dbt_assets
and DbtCli
, you won’t need to have partitions here. You could parameterize this as config instead.
It would look something like this:
@dbt_assets(manifest=manifest)
def my_dbt_assets(context: OpExecutionContext, dbt: DbtCli, config: DbtConfig):
my_customer_vars = {"customer": config.customer}
yield from dbt.cli(["run", "--vars", json.dumps(my_customer_vars)], manifest=manifest, context=context).stream()
rex
06/21/2023, 1:07 AMOverall ask: can we explore other patterns/best practices for re-using dbt models multiple places in an asset graph? I think multiple assets makes a ton of sense (we put them in different code locations/asset groups), so adding partitioning on top of that is a bit confusing.This seems more of like a discussion on how to optimally model this business logic in Dagster, than a fully-fledged feature request. Maybe you could start a GitHub discussion and ask folks to chime in here. Tangentially, this sounds a little spiritually similar: https://discourse.getdbt.com/t/unioning-identically-structured-data-sources/921 Correct me if I’m wrong.
Adam Bloom
06/28/2023, 10:21 PM