https://dagster.io/ logo
#ask-ai
Title
# ask-ai
p

Pierre Cadman

03/13/2024, 10:18 AM
How can I ensure different dbt_assets do not run concurrently
s

Scout

03/13/2024, 10:18 AM
To ensure that different
dbt_assets
do not run concurrently, you can use concurrency limits that are applied through tags in Dagster. This approach allows you to control the execution of assets so that they do not overlap, which can be particularly useful if your assets are interacting with shared resources that cannot handle concurrent access. Here's a general approach based on the documentation: 1. Assign a unique tag to each
dbt_asset
or group of
dbt_assets
that you want to run exclusively. 2. Configure concurrency limits in your Dagster instance settings to enforce that only one run with a given tag can execute at a time. For example, you might have a
dagster.yaml
configuration that looks something like this:
Copy code
yaml run_coordinator:   module: dagster._core.run_coordinator   class: QueuedRunCoordinator   config:     tag_concurrency_limits:       - key: "dbt_asset_group"         limit: 1
In this configuration, you would tag your
dbt_assets
with a
dbt_asset_group
tag, and the concurrency limit would ensure that only one asset with that tag can run at a time. Please note that the exact configuration and implementation may vary depending on your specific use case and the version of Dagster you are using. Always refer to the latest documentation for the most accurate and up-to-date guidance on configuring concurrency limits and tags.
2 Views