Hello!. Is there a straight-forward way to generat...
# ask-community
Hello!. Is there a straight-forward way to generate a non-partioned asset from a partitioned upstream asset. In my case, I have 50 csv files each representing a US state, and I wish to load them into one table. It feels sensible to define the group of files as a partioned asset but I couldn't quite work out how to then generate the downstream asset - is the set of partitioned files available as an iterable in the op?
👀 1
dagster bot responded by community 1
/sub - I will be doing something similar in the future and will need the same advice!
similar question from me:
How to select a specific partition key in upstream assets, and use it downstream (potentially generating assets with a different partition definition).
When you pass the partitioned asset to a non-partitioned downstream asset (via
in the
decorator) it will become a dict of partition key -> result mapping. It’s likely documented, I just happened to discover this by accident the other day.
how do you then access a specific partiction? AssetIn.get(part_key)?
This is an immensely helpful code snippet, thnak you for sharing (I gave it a )
not sure I see where a specific partition is being selected though. I see a dict of assets being created but don't they all have daily partitions, and the downstream asset has also the daily partition? maybe i need to study this a bit more
@Terry Lines see other thread and here for something I got working -> not an exact fit for your problem though. dagster-poc/__init__.py at 9006f49d004c2313cba01647f670fd1c867018fc · pdavidsonFIA/dagster-poc