https://dagster.io/ logo
#ask-community
Title
# ask-community
h

Holger Bruch

08/10/2023, 8:48 AM
I’m currently exploring how to setup a dynamically partitioned dataset (~20-50 partitions), running a backfill every minute. Is there a write up for such a scenario? Things I considered already: Using in_process execution, how to detect and create new partitions, how to run a backfill on every tick. I’m still not sure, if such a scenario would be to fine grained and which options I had to avoid latency/overhead issues. Dagster’s connection pooling issues might be a blocker at the moment(?). What else might need to be considered (e.g. log purging etc). Could anyone share her/his experiences?
dagster bot responded by community 1
z

Zachary Tessler

08/10/2023, 1:41 PM
I have a setup where new dynamic partitions are detected and added by a sensor that runs frequently and starts jobs just for those new partitions. Separately, I have a schedule that runs the job for all known partitions (ie a backfill) at a different frequency
3 Views