19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#ask-community
Title
# ask-community
y
Yevhen Samoilenko
05/01/2022, 10:42 AMHi! Can we somehow assign partition_defs to assets loaded using load_assets_from_dbt_project? Use case - we have models that have to be updated on daily basis. We mark them with "daily" tags, then we use load_assets_from_dbt_project with select argument "tag: daily", and it would be great to create a patririoned job from these assets.
a
Alec Ryan
05/01/2022, 11:42 AM@Yevhen Samoilenko that currently is unavailable according to @owen
You can use the vars argument in a dbt resource to pass a date
vars: {'run_date' : '2022-02-02'}
y
Yevhen Samoilenko
05/01/2022, 2:06 PMYeah, I know. But the ultimate goal is some benefits in dagster usage itself. For example to create a job on top of these partitioned assets, and track partitions in dagit, create schedule using this job, etc. Also it is not possible (at least for now) to rematerialize a subset of assets if there are a daily partitioned asset and an asset loaded from dbt (one of dbt models with tag "daily").
the other benefit is an opportunity to build partitioned jobs dynamically. For example:
daily_assets = [i for i in all_assets if i.partitions_def == DAILY_PARTITIONS_DEF]daily_job = AssetGroup(assets=daily_assets,resource_defs=resource_defs,).build_job(name="daily_job",tags={"env": env},)but now I cannot do this because I'm not able to assign partitions_def to assets that were loaded from dbt project
p
prha
05/02/2022, 4:34 PMdbt asset partitioning is something we’re working on supporting but it won’t be a quick fix. Check our release notes in the coming months for updates. cc @owen in case there’s an GH issue tracking this.
o
owen
05/02/2022, 4:38 PMhttps://github.com/dagster-io/dagster/issues/7683 just made one
s
sandy
06/27/2022, 8:33 PMcircling back here - we're now looking into direct support for partitioned assets with dbt
a few questions for you @Yevhen Samoilenko, if this is still something you're interested in:
• in your project, what partitions do you use (i.e. daily), and are all models partitioned or are some partitioned and some not?
• how do you change behavior of dbt to respect the partition (is it just vars?)
• do you use incremental models to handle inserting just the records corresponding to the partition, or is there something else going on?
y
Yevhen Samoilenko
07/01/2022, 9:59 AMHi, @sandy! This is great news! Answering your questions:
- as of now, we use hourly, daily, and monthly partitions.
- we use dbt tags to select particular models (e.g. 'hourly', 'daily', 'monthly') in combination with partition_start and partition_end vars
- for now (to not overcomplicate things), we use only view and table materializations, but I think soon we will be interested in this functionality
s
sandy
07/01/2022, 2:39 PMThanks @Yevhen Samoilenko - very helpful. We'll post updates on the GH issue above as we make progress
3 Views