Hi there! We would like to do a POC in our company, but I'm not use our use case is covered. We need to create a pipeline that processes sets of images. This pipeline has intermediate products. So our first guess was to define this intermediate products as assets, with their dependencies being other intermediate products, so the DAG is generated. However, what we need is that each time our cameras collect a new set, a new pipeline starts for that set (and assets are generated for that set). But it seems we would need to define the sets before we know them.... Maybe we need asset partitions that are dynamic instead of static? We could simply use ops and jobs but that would limit the benefits of using dagster in my opinion... Or maybe I totally messed up with my proposal and there's an easy way of using dagster that can fulfill our needs 😅 Thanks for your time!

hi @Eduard Ethan Carrés Hidalgo! Dynamic partitions + Assets is one of our most highly-requested features, and something we're actively working on. In the meantime, the standard "hack" to get around that lack of support is to represent your partitions as StaticPartitionsDefinition, i.e.

def get_partitions_def():
    sets = get_camera_data_sets()
    return StaticPartitionsDefinition(sets)

as long as whatever process you're using to figure out what data sets actually exist isn't too expensive, this should work fine (you could also consider having a separate process write out the existing sets to a file, then read from that file in this function)

Many thanks!!!