Hello, I have an asset creating a list, I want to ...
# ask-community
Hello, I have an asset creating a list, I want to create dynamically partioned assets. How would you do this ? For example : @asset def my_dataframe(column_list): df = pd.DataFrame({'column1': column_list, 'column2': ['A', 'B', 'C']}) return df My column1 would be dynamic as I don't know column_list in advance
Can I avoid Sensor which is less event driven ?
you would add partitions directly using
In the first asset. Downstream assets would likely have to live in a separate job. cc @claire just to make sure I have the story right here
Hey Thomas, yep, you can add partitions via
. One caveat here is that unexpected behaviors may occur if you mutate the dynamic partitions def during run for one of its partitions: i.e. if you added partitions in an upstream dynamic-partitioned asset, and attempted to read all partitions in in a downstream unpartitioned asset in the same run. So I would recommend not mutating the partitions def in one of its runs.
Hey Thomas, trying to better understand your use case so we can plan for an improved API around adding dynamic partitions in assets. What's your use case for doing something like this? Are downstream assets using those partitions?
Hello, I've just just read this answer (no notification). Yes downstream assets will use this partition, e.g. a subpart of previous dataframe to make numerical calculation.