19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
n
Nick Dellosa
11/17/2021, 6:06 PMIs there a way to launch a job run for every partition in a partition set at once for a non-time-based dynamcally generated partition config? And is it possible to do this on a schedule?
s
sandy
11/18/2021, 2:13 AMHi @Nick Dellosa - if you want to kick this off manually, you can launch a backfill from the "Partitions" tab of a run page
It's currently only possible to trigger one run at a time from a schedule. If you define a sensor, you can trigger multiple runs on the same tick. E.g.
@dynamic_partitioned_config(...)
def my_partitioned_config(...):
...
@sensor(job=...)
def my_sensor():
if ...:
for partition_key in my_partioned_config.get_partition_keys():
yield RunRequest(tags={"partition": partition_key}, run_config=my_partitioned_config.get_run_config(partition_key))n
Nick Dellosa
11/18/2021, 4:42 PM@sandy Any plans in the future to make scheduled backfills possible? Or is it a design decision for schedules to strictly be one run per tick?
s
sandy
11/18/2021, 4:55 PM@Nick Dellosa - I created an issue for it: https://github.com/dagster-io/dagster/issues/5714 (actually thought we had one already, but couldn't find it). I do think we'd like to enable this, though not sure on timeline
💯 1
actually wait, I think this is already possible:
@dynamic_partitioned_config(...)
def my_partitioned_config(...):
...
def evaluate_tick():
if ...:
for partition_key in my_partioned_config.get_partition_keys():
yield RunRequest(tags={"partition": partition_key}, run_config=my_partitioned_config.get_run_config(partition_key))
my_schedule = ScheduleDefinition(job=..., execution_fn=evaluate_tick)n
Nick Dellosa
11/18/2021, 5:05 PM!!! awesome, gonna try this now