https://dagster.io/ logo
Title
f

Félix Tremblay

04/21/2023, 12:54 PM
Another question about dynamic partitions. When creating partitions, is there a way to define additional metadata (i.e., other than the name of the partition) for the partition? Or maybe, attach additional metadata to the run? Basically, each asset partition needs to be defined using 3 values. And I would like to avoid naming them f"{metadata_value_1}--{metadata_value_2}--{metadata_value_3}", if possible. Thank you!
s

Sean Lopp

04/21/2023, 2:56 PM
Right now a partition key is just that - a single string. I would recommend for this use case that inside your asset function you potentially map the key to the set of metadata values - would the be possible for your use case?
f

Félix Tremblay

04/21/2023, 4:08 PM
Hi @Sean Lopp, thanks for the answer. For my use case, a mapping could not work. Therefore, I would have to concatenate all necessary information in a single string. For my use case, I need to create dynamic time ranges, so each partition is defined by an arbitrary start and end datetime. Some partitions could have a length of less than 1 minute, while others more than a year. Could it be considered for a future update to let the user add more metadata to a partition?
s

Sean Lopp

04/21/2023, 5:12 PM
Hey Felix can you file a Github issue with this request and the context of your use case? Thanks!
f

Félix Tremblay

04/22/2023, 4:07 PM
Hey @Sean Lopp, FYI there's a simple pattern that I find even better since it's more robust and explicit. We can define a Pydantic model to represent the partition metadata, and use (de)-serialization like
my_model.json()
and
MyModel.parse_raw(...)
s

Sean Lopp

04/24/2023, 3:56 PM
Great! If you find that to be a useful pattern perhaps you could add it as a GitHub issue / discussion and the team can review. My one concern - does it lead to partition names in the UI that are treated poorly?
f

Félix Tremblay

04/25/2023, 3:00 AM
@Sean Lopp, here's the link to the GitHub discussion
:ty-spinny: 1