Guys, what do you think about hierarchical/multi-d...
# dagster-feedback
m
Guys, what do you think about hierarchical/multi-dimension partitions? I.e. for hierarchical partitioning instead of single string "2022-06-06-10:00" to be able to provide {date:"2022-06-06", time:"10:00"}. It is valid case for relational databases, where you can have sub-partitions. From the other side you may have "logical" partitioning, when you collect a part of data for a physical partition. For multi-dimension partitions it should be something like {"date": "2022-06-06", "region": "Asia"}.
👍 2
b
Hierarchial multi dimension partitions are a well accepted practice and an essential feature in many big-data projects such as Spark and parquet
p
Hi Mykola… We are currently tracking this here: https://github.com/dagster-io/dagster/issues/4591. For now, a lot of users are managing this themselves by flattening the multi-dimensional space by constructing an ordered cross-product of their partitions.
❤️ 2