The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hello, I have an asset creating a list, I want to create dynamically partioned assets. How would you do this ?

For example :
@asset
def my_dataframe(column_list):
    df = pd.DataFrame({'column1': column_list, 'column2': ['A', 'B', 'C']})
    return df

My column1 would be dynamic as I don't know column_list in advance

Can I avoid Sensor which is less event driven ?

you would add partitions directly using `context.instance.add_dynamic_partitions` In the first asset. Downstream assets would likely have to live in a separate job. cc <@U028M11QNDD> just to make sure I have the story right here

Hey Thomas, yep, you can add partitions via `context.instance.add_dynamic_partitions(...)`.

One caveat here is that unexpected behaviors may occur if you mutate the dynamic partitions def during run for one of its partitions: i.e. if you added partitions in an upstream dynamic-partitioned asset, and attempted to read all partitions in in a downstream unpartitioned asset in the same run. So I would recommend *not* mutating the partitions def in one of its runs.

Hey Thomas, trying to better understand your use case so we can plan for an improved API around adding dynamic partitions in assets.

What's your use case for doing something like this? Are downstream assets using those partitions?

Hello, I've just just read this answer (no notification).

Yes downstream assets will use this partition, e.g. a subpart of previous dataframe to make numerical calculation.