Hi, I'm working on an op that processes a dataframe and make several API calls for each row. My initial implementation was to use DynamicOutput to manage the downstream ops. However, the APIs have rate limits and managing a few hundred thousand of parallel ops at one time seems quite expensive. Any other ideas on how I can approach this?
Yes thanks @claire I found that after reading some other threads. On a side note, I might also redesign the way I chunk the data, so that each parallel op handles more than one row of data. I can then use pyrate to rate limit the subsequent calls. Since I would be using dagster-serverless, I think there would already be a natural limit based on cpu cores too