Hello, what would you say are reasonable limits to think about task counts with dynamic orchestration? Hundreds, thousands or tens of thousands tasks? 🙂

Hmm thats a good question. I don’t think there is anything about dynamic orchestration specifically, so it’s more a general question of how many solids/ops in a single pipeline/job. I think how you are executing it may have an effect. We have not yet optimized extensively the “load all the events for a specific run” flows that happen in

dagit

when you are looking at a run so you may OOM dagit if its running with limited resources and you get up past the tens of thousands range. Even if it loads

dagit

is going to probably be an underwhelming product experience if it is trying to present the status of tens of thousands of things in a single place. I think it’s likely a useful lens to consider the experience of a task going wrong and someone having to go look at the run to figure out what happened. I would speculate that having 100 separate runs each with 100 tasks would be more manageable for human interactions than 1 run with 10000 tasks. That said, it really depends on the context. Would love for you to report back with what you try and how it goes.

In our context it really isn’t logical to split those runs as they are very logically separated already. That would make it harder 🙂 I’ve tried implementing that on prefect, but when using dask as an executor it had problems with memory usage and leaks. I’m hoping that dagsters k8s executor will be better in this case. I think I can pay overhead of starting new pod for each op.