The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

A couple queuing questions:
• Is there a LIFO option for the queue?
• Is there a way for the sensor to know how many things are in the queue? 
    ◦ I’m running a sensor based on files in a directory but only one run can be done concurrently. I batch files into batches of up to 100 but as files come in there might be two queue runs with 50 and I’d like to run them in a single run or prevent a run to get queued until it has hit 100 or if there are no runs in the queue.

you can tag jobs with `dagster/priority=N` to control order of job scheduling

you can definitely use a `graphql` query to l<https://docs.dagster.io/concepts/dagit/graphql#get-a-list-of-dagster-runs|ist jobs that have been submitted> and their current status. not sure if theres an easier way to do it than graphql.

i could see some "wait for all other jobs" logic being baked into either the sensor, or a first step in the jobs themselves. not sure if there's downsides to having a sensor hanging out and polling for other jobs to complete before requesting more runs

Thanks! I’ll give the graphql query a try and if it’s not too slow that should work.