Many of my pipelines through the years had to limit concurrency due to limited resources such as ram, bandwidth, singletons and so on. The feature that I am currently missing the most is to limit operator concurrency WITHOUT going via dockers. Is that anywhere in the roadmap?

you can simply set tags and concurrency limits for tags

Does this work for operator concurrency or for job concurrency?

Job

Could you please direct me to the right place in documentation?

Many thanks 🙏 Just to make sure that I understand, this requires each op to run in it’s own kubernetes job, right?

indeed: to run each op in its own Kubernetes job, Dagster also allows you to use Celery to limit the number of ops that can concurrently connect to a resource across all running Dagster jobs. In

In that case, I’ll be happy for the requested feature of limiting concurrency in general in Operator level, without dockers :) Is that anywhere in the roadmap?

I cannot comment on this - I am not working for dagster but @sandy should be able to tell you

You can use celery for per-op concurrency today, which doesn't require docker: https://docs.dagster.io/deployment/guides/celery

Nice! many thanks!