Some assets need to be generated per dataset and indirectly per image, while others need to only be generated per image

Hi, could you elaborate on what you mean here by “indirectly per image”?

A dataset has many images. A dataset may also have some other attributes e.g. calibrations. Each image has a calibration through the dataset.

I am still a little confused-- how are you imagining mapping this data model into Dagster? With “dataset” as a dynamically partitioned asset and “images” as another dynamically partitioned asset?

Yes, that's exactly what I was thinking

I see-- have a look at `PartitionMapping`: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partition_mapping.py#L53-L101 We have a bunch of specialized subclasses but in this case I believe you’d need to implement your own

PartitionMapping

subclass. Then you would list the set of all images across all datasets in the

DynamicPartitionsDefinition

for your images, and use your custom partition mapping to define the relationship between the dataset

DynamicPartitionsDefinition

and the image one.