Hi! Can we somehow assign partition_defs to assets...
# ask-community
y
Hi! Can we somehow assign partition_defs to assets loaded using load_assets_from_dbt_project? Use case - we have models that have to be updated on daily basis. We mark them with "daily" tags, then we use load_assets_from_dbt_project with select argument "tag: daily", and it would be great to create a patririoned job from these assets.
a
@Yevhen Samoilenko that currently is unavailable according to @owen
You can use the vars argument in a dbt resource to pass a date
vars: {'run_date' : '2022-02-02'}
y
Yeah, I know. But the ultimate goal is some benefits in dagster usage itself. For example to create a job on top of these partitioned assets, and track partitions in dagit, create schedule using this job, etc. Also it is not possible (at least for now) to rematerialize a subset of assets if there are a daily partitioned asset and an asset loaded from dbt (one of dbt models with tag "daily").
the other benefit is an opportunity to build partitioned jobs dynamically. For example:
daily_assets = [
i for i in all_assets if i.partitions_def == DAILY_PARTITIONS_DEF
]
daily_job = AssetGroup(
assets=daily_assets,
resource_defs=resource_defs,
).build_job(
name="daily_job",
tags={"env": env},
)
but now I cannot do this because I'm not able to assign partitions_def to assets that were loaded from dbt project
p
dbt asset partitioning is something we’re working on supporting but it won’t be a quick fix. Check our release notes in the coming months for updates. cc @owen in case there’s an GH issue tracking this.
o
s
circling back here - we're now looking into direct support for partitioned assets with dbt a few questions for you @Yevhen Samoilenko, if this is still something you're interested in: • in your project, what partitions do you use (i.e. daily), and are all models partitioned or are some partitioned and some not? • how do you change behavior of dbt to respect the partition (is it just vars?) • do you use incremental models to handle inserting just the records corresponding to the partition, or is there something else going on?
y
Hi, @sandy! This is great news! Answering your questions: - as of now, we use hourly, daily, and monthly partitions. - we use dbt tags to select particular models (e.g. 'hourly', 'daily', 'monthly') in combination with partition_start and partition_end vars - for now (to not overcomplicate things), we use only view and table materializations, but I think soon we will be interested in this functionality
s
Thanks @Yevhen Samoilenko - very helpful. We'll post updates on the GH issue above as we make progress