https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alexander Whillas

07/15/2022, 2:35 AM
Is it possible for dagster to manage job queuing?
I have to hit an API that can only handle 8 concurrent requests at once. I need to make sure of this across all jobs that might run. So jobs can't run in parallel globally. @yuhan pointed out you can set
max_concurrent
to be 1 but that is per execution? I need to make sure across all concurrent executions. Perhaps I need to put an API Infront of the API which manages a queue of requests and has a really long timeout? Or the first half of the job ends at a queue for the API, and then a sensor picks up the second half. Is there any kind of queue management in Dagster?
maybe Run Coordinators?
OK, so I can split the job into 2 parts, the first that hits the API and has a `max_concurrenct=1`and then the rest of the processing of those results in another job... so is it possible to get one job to launch another after it finishes?