https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
a

Alec Koumjian

02/28/2023, 9:16 PM
Is there a particular reason that a partitioned asset can't accept a partitioned asset with a different upstream partition definition? For a non-partitioned downstream asset, the default IO Manager simply gives you a dictionary of partitions by key. Seems like that could be enabled for partitioned downstream assets too? For example, if the upstream asset is partitioned on space, or some static partition definition, a downstream monthly partitioned asset may actually need to scan the entire upstream asset to do some joins or map/reduce.
As a result of this limitation, we are having to contort the bleeding edge ability to load assets into ops using
to_source_asset
and manually yield a bunch of materializations inside an
op
/
@job
.
s

sandy

02/28/2023, 10:07 PM
I think you can accomplish this if you set
AllPartitionMapping
s

Stephen Bailey

03/01/2023, 1:41 AM
you can also use `non_argument_deps`to logically link but not actually load the asset at runtime
p

Peter Davidson

03/01/2023, 8:13 AM
You can build your own PartitionMapping as an interim fix. This isn't yet supported by dagster though. I shared some code somewhere else for the partitioin mapping i built
a

Alec Koumjian

03/01/2023, 1:17 PM
Thanks all, think we are going to try
AllPartitionMapping
for now. My team is disinclined to use a custom partition mapping given the warning that the API is subject to change on a whim. For the
non_argument_deps
options, we do actually need the read in the asset as runtime to do an in memory join.
plus2 1
3 Views