Hi team, we want to set tag concurrency limit to set max quota for each user code deployments that are running in our dagster deployment to make sure one of them don't take all the allowed max limit. We also want to apply limits to certain use cases under each of the user code. Does the team recommend any specific structure for this use case? We are currently trying something similar to below

queuedRunCoordinator:
  enabled: true
  config:
    max_concurrent_runs: 25
    tag_concurrency_limits:
      - key: "user-code-1"
        limit: 15
      - key: "user-code-1"
        value: "usecase-1"
        limit: 10
      - key: "user-code-1"
        value: "usecase-2"
        limit: 5

      - key: "user-code-2"
        limit: 15
      - key: "user-code-2"
        value: "usecase-1"
        limit: 10

      - key: "databricks"
        limit: 10

Hi @Arun Kumar, this structure seems reasonable to me. There may be some tricks for setting tags on every pipeline and sensor/schedule within a repo- cc @chris

Thanks @johann for the response. Quick follow ups: 1. When a particular job satisfies multiple conditions, does the coordinator satisfy all the limits. In the above example, we have a global limit on

databricks

tag, so if we get a

databricks

job from

user-code-1

, will it still be limited to 10 runs irrespective of the limit on

user-code-1

? 2. If I understand currently, if any user code job missed to set the appropriate user-code tag, it will fall under the global runs and can possibly exhaust the global limit (25 runs in the above example). Is there any way to avoid this situation?

1. When a job falls under all limits, it will only launch once it won’t exceed any of them. So to your question, the databricks job from user-code-1 will launch if both of those limits can be respected. If for example the databricks limit was met but the user-code-1 limit was not, then other jobs from user-code-1 that don’t have a databricks tag may proceed

Thanks @johann I was looking for some way to make sure that every user deployment will always have a valid tag. Currently we do not have any control and anyone with a new user code can just exhaust all the global limits affecting other user code deployments.