https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jon Erik Kemi Warghed

08/07/2023, 1:21 PM
I have a file that can be dropped in a folder, it can contain data for x days. Downstream there is an asset that is partitioned per day. How do I mark for the downstream asset that now there is data for day z,y and q? My naive approach would be to sense that there is a file, have an asset using a dynamic partitioning, output multiple daily partitions, then have downstream asset- with daily partitioning connected to this however I am getting lost in the jungle of concepts here I feel and would need some pointers.
here are some relevant discussions about constructing dynamic pipeline: • https://github.com/dagster-io/dagster/discussions/15119 — it lists the different dynamic scenarios that sometimes would involve different apis • https://github.com/dagster-io/dagster/discussions/14715 — talks about how you can get partition keys in a dynamic partition. here’s a blog post about dynamic partition: https://dagster.io/blog/dynamic-partitioning
2 Views