I have a few resources that perform API calls. Each must respect rate limits that are specific to its API endpoint. An op uses one of these resources; which specific resource depends on its inputs. Ideally, each resource should not issue its API call if it exceeds the rate limit, and it should not raise an error as it is normal for the number of invocations during a run to be higher than the rate limit allows. So it seems like a global inter-process queueing mechanism is needed. Best approach for this in Dagster?

In Dagster, managing rate limits for resources that perform API calls can be approached in a few ways. However, the documentation does not provide a built-in global inter-process queueing mechanism specifically designed for rate limiting API calls across multiple processes. Instead, you can consider the following approaches based on the documentation and discussions: 1. External Rate Limiting Libraries: You can use external libraries that provide rate limiting features and support distributed systems. For example, PyrateLimiter is mentioned in a discussion where it supports the use of a Redis server for synchronization across distributed systems. This could be integrated into your resources to ensure that API calls respect the rate limits. 2. Concurrency Limits: While not directly related to rate limiting, Dagster allows you to configure concurrency limits using tags. This can help prevent too many concurrent executions that might hit the rate limits. You can specify

tag_concurrency_limits

in your job's config to limit the number of ops/assets with a specific tag that can run at a given time. However, this approach limits concurrency rather than rate limiting API calls per time interval. 3. Custom Middleware or Services: You might need to implement custom middleware or services that manage the rate limiting logic. This could involve a service that queues up requests and ensures that they are dispatched at a rate that complies with the API's rate limits. Your ops would interact with this service to perform API calls. 4. Dagster Sensors: Although not a direct solution for rate limiting, you could potentially use sensors to monitor the state of your resources and trigger ops based on the availability to make API calls without exceeding rate limits. 5. External Asset: If the operation takes a long time due to external service rate limits and you don't want to keep a Dagster run going, you could model it as an External Asset. You would submit your request to the external service in an op and use another server to listen for a callback, then report a materialization for the asset when the callback is received. Remember that these approaches may require additional infrastructure, such as a Redis server for distributed rate limiting or a custom service for queuing and dispatching API calls. It's important to choose an approach that aligns with your system's architecture and the specific requirements of the APIs you are interacting with.