<@U011CET83FG> posting here, but please redirect m...
# integration-dbt
a
@sandy posting here, but please redirect me elsewhere is appropriate. Seems you all are gathering feedback for some future rework. We have a dbt use case that I'm honestly not sure how unique it is, but the current way dagster handles this is a bit awkward. Use case: reuse a set of models across different schemas (each represent a different one of our customers). dbt doesn't provide a good mechanism for doing this, short of using vars (which is what we did). Each customer requires a separate run of dbt, with different vars passed. dagster oddity: we have to not only create separate assets (via an asset factory) to accomodate this, but the only way to pass vars is to make those partitioned assets. It's a bit odd to need the partitions here. Overall ask: can we explore other patterns/best practices for re-using dbt models multiple places in an asset graph? I think multiple assets makes a ton of sense (we put them in different code locations/asset groups), so adding partitioning on top of that is a bit confusing.
r
It’s a bit odd to need the partitions here.
With the new APIs for declaring dbt SDA’s using
@dbt_assets
and
DbtCli
, you won’t need to have partitions here. You could parameterize this as config instead. It would look something like this:
Copy code
@dbt_assets(manifest=manifest)
def my_dbt_assets(context: OpExecutionContext, dbt: DbtCli, config: DbtConfig):
    my_customer_vars = {"customer": config.customer}
    yield from dbt.cli(["run", "--vars", json.dumps(my_customer_vars)], manifest=manifest, context=context).stream()
Overall ask: can we explore other patterns/best practices for re-using dbt models multiple places in an asset graph? I think multiple assets makes a ton of sense (we put them in different code locations/asset groups), so adding partitioning on top of that is a bit confusing.
This seems more of like a discussion on how to optimally model this business logic in Dagster, than a fully-fledged feature request. Maybe you could start a GitHub discussion and ask folks to chime in here. Tangentially, this sounds a little spiritually similar: https://discourse.getdbt.com/t/unioning-identically-structured-data-sources/921 Correct me if I’m wrong.
a
Actually getting a chance to dig into this now - looks like there aren't ways to supply asset groups or non argument deps to that new decorator yet. Conceptually, it makes a lot of sense to me though! I may open a PR if I find I can't live without those few things - still exploring the best way to construct this