https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Thomas Monk

07/01/2023, 11:18 AM
I have a set A of partitions, generated dynamically. Each member of these partitions will have their own dynamically generated sub-partitions, B(a). Each set of these sub-partitions are unique to the parent a, so I don't think I can use multi-paritions. I understand that there is partition mapping I can do, but can I within the UI ask dagster to run the parent partition A, and all of its sub-partitions? Thanks.
j

Justin Taylor

07/01/2023, 11:58 AM
As far as I know, it isn't currently possible to create a partition mapping between dynamic partitions. IMO it would be a very useful feature. There is an open issue here: https://github.com/dagster-io/dagster/issues/13139 I've had luck using the
DynamicOut(put)
objects in Dagster to do computation on sub-units of the data, while keeping the assets partitioned with the same partition definition. I also use an
asset_sensor
to trigger materializations of a downstream asset after a dynamic partition in the upstream asset is materialized, as I haven't had the best luck using
AutoMaterializePolicy
with our dynamically-partitioned assets. https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs#dynamic-graphs https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors
t

Thomas Monk

07/03/2023, 3:41 PM
Thanks a lot for this, super useful. I hadn't seen DynamicOut before, looks like this could work well here.
4 Views