Hey! I have an hourly partitioned asset that has t...
# ask-community
s
Hey! I have an hourly partitioned asset that has two upstreams(hourly and daily). Is there a way to specify that daily upstream should be “watched/checked” a day late. So today’s hourly should run if yesterday’s daily is done. This might be kinda done with offsets but with that partitions will be wrong (today’s hourly would be yesterday’s hourly) if I understand correctly. Is there a workaround for this problem?
s
if I'm understanding your situation correctly, then I think you might already see that behavior in the default case? if you have a hourly asset that depends on a daily asset, then the daily partition for tuesday won't show up until the start of wednesday, and thus the downstream hourly partitions for tuesday won't be filled in until wednesday or is it that you want the partition for the first hour of wednesday to use data from tuesday in the upstream daily asset?
s
first hour of wednesday to use data from tuesday in the upstream daily asset? Yes, rather this. I would want to run wednesdays hourly partitions even though one of the upstreams of that is a daily partition that doesn’t have a wednesdays partition yet. Right now we “solved” the situation by setting hourly asset’s end offset -24
s
I see
TimeWindowPartitionMapping
has
start_offset
and
end_offset
that allows shifting the time dependencies on a particular edge. However, these parameters don't yet work for the case where the upstream is daily and the downstream is hourly Here's an issue that tracks this: https://github.com/dagster-io/dagster/issues/14736
s
Thank you!