https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
g

Guillaume Onfroy

04/04/2023, 9:03 AM
Hi. We have the following requirement: for our daily Dagster jobs, we need the assets to backfill for X number of days in the past, instead of only materializing the current day. Basically a "rolling" backfill. Our idea was to introduce a new daily job composed of one op that triggers a Dagster backfill of the main assets job (using somehow the internal python modules, not the CLI). But we're wondering if that's even possible in the context of the OSS deployment (Helm chart) to trigger a backfill from a job's pod since the Dagster daemon is not running or accessible there. Any thoughts or recommendations?
d

daniel

04/04/2023, 3:00 PM
Hi Guillaume - one thing to note is that Dagster schedules can return multiple RunRequests from the schedule function, and each of them will be launched on the schedule's cron schedule. So you could potentially implement the backfill that way by returning X RunRequests each tiem? You could also launch a backfill via the Dagster Graphql API (the same way that Dagit does) - but the option above is what i'd try first. The main #dagster-support channel may be a better bet for questions about how to model that exactly
g

Guillaume Onfroy

04/04/2023, 3:06 PM
Hi @daniel. Thanks for answering. So am I understanding this correctly: instead of defining a schedule based on a single job, we would define a different schedule based on multiple
RunRequests
? Any chance you could roughly point me to the right API to achieve that?
d

daniel

04/04/2023, 3:08 PM
Sure - here's an example of a schedule that returns multiple RunRequests: https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules#static-partitioned-jobs
g

Guillaume Onfroy

04/04/2023, 3:11 PM
I see, so the trick comes down to using "static partitions" which I would generate dynamically based on the date?
d

daniel

04/04/2023, 3:11 PM
The example there was just intended to show multiple run requests - i don't think you would use static partitions for this.
But generating run requests corresponding to the (time-based I think right?) partitions that you want to launch from the schedule
g

Guillaume Onfroy

04/04/2023, 3:12 PM
Correct. I want everyday to have the equivalent of a 7 days backfill
d

daniel

04/04/2023, 3:14 PM
ok, I think you would still use the run_request_for_partition function like in that example, but for the time-based partition keys that you want to run
g

Guillaume Onfroy

04/04/2023, 3:15 PM
That helps a lot, thanks!
condagster 1
2 Views