https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Philippe Laflamme

03/29/2023, 1:56 AM
I have an asset, say
A
with daily partitioning with
start_date
of
start_a
. I also have a
B
asset that consumes
A
but which is offset by one day. As such, the
start_date
for
B
is
start_a + 1
since it can only run once
A
has at least one day available. This works all fine, except that I find myself having to treat these 2 assets separately in some places because their
PartitionDefinitions
are not exactly the same. Specifically, I cannot use
build_asset_job
and specify both
A
and
B
. This happens quite a bit throughout because this "offset" pattern is common in the data I'm dealing with (i.e.: I also use
end_offset
with positive and negative values). So I end up having to create many jobs that really shouldn't have to. Also, I haven't found a great way of always triggering the downstream job since asset materialization sensors only looks at the latest partition AFAIK. Is there a better way to deal with lots of variability in
PartitionDefinitions
like I have?
🤖 1
s

sandy

03/29/2023, 11:15 PM
There's currently a restriction that each run needs to target the same partition across all assets - it's not impossible, but breaking this constraint would require some deep surgery If you're using the asset reconciliation sensor, it should handle this automatically as long as you're using TimeWindowPartitionMapping with an offset to express the partition dependency If you're using a custom multi-asset sensor, it should also respect the partition mapping https://docs.dagster.io/_apidocs/partitions#dagster.TimeWindowPartitionMapping
also, you can now select assets with different partitionings and click the materialize button. dagster will launch a backfill that gets split into sub-runs that target different partitions
p

Philippe Laflamme

03/30/2023, 1:18 PM
On that last point, shouldn’t that make it possible to define some kind of “multi-run” job? One that would allow targeting related assets, but that have different partition definition (say ones that only differ by start/end offset)? It’s great that I can do this in the UI, but it would be a lot more convenient to be able to make a formal job out of this.
s

sandy

03/30/2023, 3:03 PM
🙏 1
3 Views