https://dagster.io/ logo
#ask-ai
Title
# ask-ai
p

Pierre Cadman

03/12/2024, 6:27 PM
How can I prevent two different assets from running concurrently
s

Scout

03/12/2024, 6:27 PM
To prevent two different assets from running concurrently, you can use concurrency limits in Dagster, which are typically configured through tags. The documentation provides an example of how to manage concurrency using tags: 1. Assign a unique tag to each asset or job that you want to control concurrency for. 2. Set a concurrency limit for that tag in your Dagster instance configuration. Here's a step-by-step guide based on the documentation: 1. Assign Tags to Assets/Jobs: Assign a unique tag to each asset or job. For example, if you have two assets,
asset1
and
asset2
, you could assign them tags like
asset1_concurrency
and
asset2_concurrency
. 2. Configure Concurrency Limits: In your Dagster instance configuration file (usually
dagster.yaml
), you can set up
tag_concurrency_limits
to enforce concurrency limits based on the tags you've assigned. Here's an example configuration that sets a concurrency limit of 1 for each tag:
Copy code
yaml    tag_concurrency_limits:      - key: "asset1_concurrency"        limit: 1      - key: "asset2_concurrency"        limit: 1
This configuration ensures that only one run of each tagged asset can execute at a time. 3. Apply Tags When Launching Runs: When you launch runs for your assets, make sure to include the appropriate tags. This can be done programmatically or through the Dagster UI by editing the tags before launching a run. By following these steps, you can ensure that
asset1
and
asset2
do not run concurrently. Each asset will wait for the other to complete if the concurrency limit has been reached. Please note that this approach is based on the use of tags and instance configuration. If you have more complex concurrency requirements or need further assistance, please refer to the documentation or consider reaching out to the community for more specific use cases.
8 Views