is there a way to partition partitions? I read abo...
# ask-community
f
is there a way to partition partitions? I read about the multipartitions, but they're not subdivisions, they are a cross product of two independent methods of tagging the data (like date and some other label)
🤖 1
g
plus1 - I don’t think there is a native way, I’d also like to see this. In the meantime I use static/dynamic partitions where the partition key is like
parent/child
split by a sensible delimiter
f
yep, was thinking of doing the same
the problem is that I might end up breaking dagit's UI with the amount of subdivisions we want to track 😬
blob sad 1
for instance, we might have hundreds of datasets in a project and each of those might have thousands of entities we want to track individually. It quickly adds up
s
Hi Fabio, To be clear, is the primary difference between what you’re asking for and multipartitions that you want the freedom to have a different second axis for each element in the first axis?
f
yes, that's exactly it @sean if we went for the cross product, we would have many combinations that aren't valid. So we would like the second axis to depend on the first axis
but most important of all is the way we visually track these partitions. It is a hard one to design
s
OK. Yeah I definitely see the appeal here, but I strongly suspect we’re not going to implement this in a first-class way anytime soon, simply because there’s al ready a ton of work going in other areas of partitions. I have to recommend that for now you follow @Guy McCombe’s advice and “simulate” this using a delimiter in the partition key.
That said, I recommend opening a feature request on GH to inject the idea into the conversation
👍 1
g
When you do, @Fabio Picchi, link it here and I’ll throw my 👍 on it 😁
f