https://dagster.io/ logo
#ask-community
Title
# ask-community
n

Nima Rahnemoon

08/17/2023, 10:27 PM
Some assets need to be generated per dataset and indirectly per image, while others need to only be generated per image
🤖 1
s

sean

08/18/2023, 2:46 PM
Hi, could you elaborate on what you mean here by “indirectly per image”?
n

Nima Rahnemoon

08/18/2023, 7:59 PM
A dataset has many images. A dataset may also have some other attributes e.g. calibrations. Each image has a calibration through the dataset.
s

sean

08/18/2023, 8:15 PM
I am still a little confused-- how are you imagining mapping this data model into Dagster? With “dataset” as a dynamically partitioned asset and “images” as another dynamically partitioned asset?
n

Nima Rahnemoon

08/18/2023, 8:16 PM
Yes, that's exactly what I was thinking
But I'd like there to be some way of saying ImagePartition imageXXX is linked to DatasetPartition datasetYYY
DatasetSource (StaticParition: real or sim) has many Datasets (DynamicParition) has many Images (DynamicParition)
s

sean

08/18/2023, 8:26 PM
I see-- have a look at `PartitionMapping`: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partition_mapping.py#L53-L101 We have a bunch of specialized subclasses but in this case I believe you’d need to implement your own
PartitionMapping
subclass. Then you would list the set of all images across all datasets in the
DynamicPartitionsDefinition
for your images, and use your custom partition mapping to define the relationship between the dataset
DynamicPartitionsDefinition
and the image one.