https://dagster.io/ logo
Title
h

Holger Bruch

05/17/2023, 5:19 PM
Beginner question: I’ve got a number of datasets, all having the same format, but updated independently. They all should be processed by more or less the same processing steps (some of them with dataset specific parameters). Most documentation I’ve read so far seems to describe explicitly defined assets. DynamicOut might be what I’m looking for, but I’m not sure. Could someone point me to an example/docs where assets are defined based on e.g. a supplied list and can be sensed for required re-materialization? Background: currently, I transform these datasets via a makefile which is hard to understand and I’d like to rebuild with dagster.
:dagster-bot-responded-by-community: 1
z

Zach

05/17/2023, 5:21 PM
Sounds like partitioned assets might be a decent fit here
h

Holger Bruch

05/17/2023, 9:39 PM
I didn‘t see my files as being a set of partitions, but yes, that might work. Thanks, I‘ll share my learnings…