https://dagster.io/ logo
#ask-ai
Title
# ask-ai
n

Nasser Diallo

03/11/2024, 12:05 PM
how can I run a partitionned job on mutiples partitions but I don't want parallel excuction on partitions. only one partition at a time
s

Scout

03/11/2024, 12:05 PM
To run a partitioned job on multiple partitions without parallel execution, you can use the
backfill
feature in Dagster, which allows you to launch runs for each partition sequentially. When you create a backfill, you can specify the partitions you want to run and configure the execution to process one partition at a time. Here's how you can achieve this: 1. Define a Schedule: Define a schedule that targets the partitions you want to run. The schedule can be set up to run at a frequency that matches your partitioning scheme (e.g., daily if your partitions are daily). 2. Backfill Policy: Set the
backfill_policy
on your asset or job to
BackfillPolicy.single_run()
. This policy ensures that when you launch a backfill, it will execute the partitions sequentially in a single run, rather than in parallel. 3. Launch Backfill: Use the Dagster UI or the GraphQL API to launch a backfill for the partitions you want to run. You can select the specific partitions you want to include in the backfill. 4. Concurrency Limits: Optionally, you can also set concurrency limits using tags to ensure that only one run is executed at a time. This can be done by setting a tag concurrency limit in your
dagster.yaml
configuration file. Here is an example of how you might configure the
backfill_policy
on an asset:
Copy code
python from dagster import asset, BackfillPolicy  @asset(backfill_policy=BackfillPolicy.single_run()) def my_asset():     # Your asset code here
Please note that the
BackfillPolicy.single_run()
policy is used to ensure that the backfill runs partitions sequentially in a single run. However, if you want to enforce that only one partition is processed at a time across multiple runs, you would need to manage the concurrency at the job level, possibly by using tags and setting appropriate concurrency limits. Keep in mind that the exact implementation details may vary based on your specific use case and Dagster version. Always refer to the latest Dagster documentation for the most accurate and up-to-date guidance on configuring backfills and concurrency.
n

Nasser Diallo

03/11/2024, 12:17 PM
how can I set concurrency limits using tags to ensure that only one run is executed at a time.
how can I set concurrency limits using tags to ensure that only one run is executed at a time.
11 Views