https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alexander Vandenberg-Rodes

12/30/2022, 10:00 PM
Hi all, I have the following use case for Partition Mapping: • There is a set of assets sharing a static partition definition P • There is another set of assets sharing a different static partition definition Q. The job of some of the latter assets is to roll up or union the data from any partitions in P that map to the same partition in Q according to a predefined function from P to Q. Here's an example gist: https://gist.github.com/AlexanderVR/67ade7ecfb1d72578675eca2171f92cd This works great, especially with the magical
build_asset_reconcilation_sensor
:-) How experimental is this functionality? Can I expect this approach to be stable in the next couple months? Asking because there are a few nagging issues as of Dagster 1.1.7. Details in thread...
👀 1
1
1) PartitionMapping is marked as experimental, with the docs saying "Overriding PartitionMapping outside of Dagster is not supported. The abstract methods of this class may change at any time." 2) I have to reach into
dagster._core
to get the typing information for
PartitionsSubset
3) Testing using
materialize/materialize_in_memory
at the end of the gist fails with `dagster._core.errors.DagsterInvariantViolationError: Attempting to access run_id, but it was not provided when constructing the OutputContext 4) In the dagster UI it's not possible to attempt to materialize both assets at once because they don't share the same partition definition
c

claire

01/03/2023, 6:29 PM
Hi Alexander. `PartitionMapping`s are now marked as stable (unexperimentalized), though it is not supported to create custom partition mappings. I'd say that having a mapping between to static partition definitions seems like a fairly common use case and that we can internally support a partition mapping with this functionality. If you don't mind, you could open up a PR to add your partition mapping to Dagster, or you could open up an issue so we can take this on.
a

Alexander Vandenberg-Rodes

01/03/2023, 7:29 PM
Thanks Claire, will be happy to open a PR for this case. Still needs a little thinking through, e.g. for performance with many partitions might be nice to cache to inverse mapping. Should be able to work on it in next couple days.
🌈 1
c

claire

01/03/2023, 8:08 PM
Thanks Alexander!
a

Alexander Vandenberg-Rodes

01/05/2023, 7:20 AM
Hi Claire, here is the PR: https://github.com/dagster-io/dagster/pull/11521 While the contributor guidelines were very clear, I’m wondering if the Rosetta section is still needed? I did not run into issues with arm64
9 Views