https://dagster.io/ logo
#ask-community
Title
# ask-community
b

Binoy Shah

08/17/2023, 2:19 PM
Question about Partitions And PartitionMapping • For Assets, how can we map many Upstream Partitions to Single Downstream Partition such that Upstream Asset may materialize X number of times, but downstream only materializes Once. ◦ I have seen this happening for scenario {Upstream Asset with DailyPartition} --> {Downstream Asset with Weekly Partition}, the Downstream asset executes/materializes only once for 7 materializations of Upstream Daily Partitions I am asking because I want to adopt similar flow for Upstream Assets with Dynamic Partitions and Downstream Asset just Singular Partition
🤖 1
s

sean

08/17/2023, 2:31 PM
Hi Binoy, Do you mean “downstream only materializes once [by the Automaterialization Daemon]“?
b

Binoy Shah

08/17/2023, 2:33 PM
So that part I am not sure, if it is Automaterialization Daemon, This was an observation for DailyPartition --> WeeklyPartition Assets materialization and logs statement that I put to track how many times DailyPartition’ed Asset was called and How many times WeeklyPartition’ed Asset was called and it appeared to be 7:1 and I wanted to know how it is done, since I wanted to apply same concept to a different partitioning scheme
The code when materialized for 7 days it prints logs as
Daily Inventory With Partition is…
x 7
Weekly Inventory with Partition is…
x 1
s

sean

08/17/2023, 2:40 PM
This was an observation for DailyPartition --> WeeklyPartition Assets materialization and logs statement that I put to track how many times DailyPartition’ed Asset was called and How many times WeeklyPartition’ed Asset was called
Something has to be launching materializations, whether it be a person doing it manually, a schedule, a sensor, or the automaterialize daemon. Can you check the “Deployment” of the UI and see if “Auto-materializing” is enabled?
b

Binoy Shah

08/17/2023, 2:41 PM
I run it as
dagster dev
image.png
s

sean

08/17/2023, 3:17 PM
I’m a little unclear on what the problem is-- there are different ways to trigger materializations. You could run your weekly partitioned asset on a weekly schedule and the daily one on a daily schedule, and IIUC that would do what you’re asking for.
b

Binoy Shah

08/17/2023, 3:26 PM
I think I understood Partitions a bit differently, I was falsely under impression that Partitions are causing to converge from multi to single executions between upstream and downstream assets, but in fact it’s my misguided thought process. What is really happening is: For 2 Assets
Asset_D
and
Asset_W
[ `Asset_D`:DailyPartition and `Asset_W`:WeeklyPartitions each has its own Job definition at Daily and Weekly partition scheme and both are running at their respective frequency. Convergence is just a perception 🤷‍♂️
And there was no problem, i was just trying to see if there was an automatic way to provide some custom “convergence” from multi to single partition which is not based on dates
s

sean

08/17/2023, 3:32 PM
Ah yes, yeah you are free to schedule the materialization of differently partitioned assets however you want, when they are executed is a separate concern from the mapping between them.
7 Views