https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Terry Lines

12/21/2022, 11:59 PM
Hello!. Is there a straight-forward way to generate a non-partioned asset from a partitioned upstream asset. In my case, I have 50 csv files each representing a US state, and I wish to load them into one table. It feels sensible to define the group of files as a partioned asset but I couldn't quite work out how to then generate the downstream asset - is the set of partitioned files available as an iterable in the op?
1
👀 1
dagster bot responded by community 1
g

Gabe Schine

12/22/2022, 12:30 AM
/sub - I will be doing something similar in the future and will need the same advice!
p

Peter Davidson

12/22/2022, 7:15 AM
similar question from me:
How to select a specific partition key in upstream assets, and use it downstream (potentially generating assets with a different partition definition).
a

Andras Somi

12/22/2022, 8:21 AM
When you pass the partitioned asset to a non-partitioned downstream asset (via
AssetIn
in the
@asset
decorator) it will become a dict of partition key -> result mapping. It’s likely documented, I just happened to discover this by accident the other day.
p

Peter Davidson

12/22/2022, 8:44 AM
how do you then access a specific partiction? AssetIn.get(part_key)?
p

Peter Davidson

12/22/2022, 10:57 AM
This is an immensely helpful code snippet, thnak you for sharing (I gave it a )
not sure I see where a specific partition is being selected though. I see a dict of assets being created but don't they all have daily partitions, and the downstream asset has also the daily partition? maybe i need to study this a bit more
@Terry Lines see other thread and here for something I got working -> not an exact fit for your problem though. dagster-poc/__init__.py at 9006f49d004c2313cba01647f670fd1c867018fc · pdavidsonFIA/dagster-poc
3 Views