https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
b

Benjamin Weise

04/21/2023, 1:16 AM
Hi team, just wanted to say that it's really gratifying to be looking to implement a particular workflow in dagster, and come across a feature that has just been implemented that aligns well with what I'm trying to do. In this case, the
MultiPartitionMapping
that @claire merged in this PR from a few weeks ago 🙂
t

Tim Castillo

04/21/2023, 5:10 PM
Hahah, and on behalf of the team, we appreciate the kind words daggy love
c

claire

04/21/2023, 5:13 PM
Glad you found it useful, much appreciated!
b

Benjamin Weise

04/24/2023, 1:04 AM
So, following on from this, I'm delving deeper into multipartitions and was wondering if supporting > 2 partitions was on the roadmap? My use case is that I'm currently using the 2 partitions to partition by day and equipment ID, and am looking to run a series of forecasting techniques against each partition. Ideally I would add the forecast technique as a (3rd) partition, which is currently unsupported. It is not a big deal to treat them as separate assets per forecast type, it just adds a little bit of boiler plate. But was just interested if there are thoughts on extending multi-partitions.
❤️ 1
c

claire

04/25/2023, 6:10 PM
Hey Benjamin, it's something that we've considered but isn't immediately on the roadmap. The blockers here are mostly just improving performance and updating the UI to handle more than two dimensions. I'd recommend filing a feature request for this if this is functionality you want. It would help to know how many dimensions you'd want and an estimate of the total number of partitions
b

Benjamin Weise

04/25/2023, 10:25 PM
Thanks Claire, I had imagined that was the case. For now I think splitting the 3rd dimension out into separate assets works reasonably for my use case, but I'll file a request if I feel that it's warranted.
🌈 1
j

John Smith

06/29/2023, 10:39 PM
reviving this ask on new thread in case there is a workaround. otherwise I'll raise a feature request as it's critical for my org's dagster adoption. https://dagster.slack.com/archives/C01U954MEER/p1688077355089059
s

Sean Lopp

07/03/2023, 7:39 PM
@John Smith here is an example of using multiple assets to handle 3+ dimensions: https://gist.github.com/slopp/98753ab6efcd654da1b7a49c5639aabf
j

John Smith

07/03/2023, 8:35 PM
Thanks @Sean Lopp however this can quickly get very messy in the UI when I have 20 shapes and 20 colours. In my usecase I'd ideally want to see the diagram below in the UI, not each individual color or shape. only when I "drill" into each node (e.g. colorshape) do I want to see the "partitions" so I can see the woods from the trees at a higher level
s

Sean Lopp

07/03/2023, 8:51 PM
This message contains interactive elements.