I have a set A of partitions, generated dynamically. Each member of these partitions will have their own dynamically generated sub-partitions, B(a). Each set of these sub-partitions are unique to the parent a, so I don't think I can use multi-paritions. I understand that there is partition mapping I can do, but can I within the UI ask dagster to run the parent partition A, and all of its sub-partitions? Thanks.

As far as I know, it isn't currently possible to create a partition mapping between dynamic partitions. IMO it would be a very useful feature. There is an open issue here: https://github.com/dagster-io/dagster/issues/13139 I've had luck using the

DynamicOut(put)

objects in Dagster to do computation on sub-units of the data, while keeping the assets partitioned with the same partition definition. I also use an

asset_sensor

to trigger materializations of a downstream asset after a dynamic partition in the upstream asset is materialized, as I haven't had the best luck using

AutoMaterializePolicy

with our dynamically-partitioned assets. https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs#dynamic-graphs https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors

Thanks a lot for this, super useful. I hadn't seen DynamicOut before, looks like this could work well here.