Correct me if I am wrong, we cannot have celery an...
# deployment-kubernetes
v
Correct me if I am wrong, we cannot have celery and non celery runLaunchers coexist , right ?
d
that's currently right. This is something we'd like to address in the future though (much like the config stories for run launchers and executors will become more consistent, the way you specify which ones are possible will also become more consistent)
v
perfect. Another question, is celery always step isolated , as in can I force celery to execute the pipeline in s a single process ( very akin to non celery RunK8s without steps )
I think I cannot, but would be nice to hear from the source...
d
We don't have that currently - the use case would be global resource limits at the pipeline level rather than at the step level? We do have a built-in feature that covers a lot of similar ground that doesn't require celery: https://docs.dagster.io/deployment/run-coordinator#limiting-run-concurrency
v
I do set that up, yep ...
Copy code
runCoordinator = {
        enabled = true
        type    = "QueuedRunCoordinator"
        config = {
          queuedRunCoordinator = {
            maxConcurrentRuns : 20
            tagConcurrencyLimits : []
            dequeueIntervalSeconds : 2
          }
        }
      }
But then too, it will not be a t the pipeline level right ? Even if I do a tag based concurrency...
I think it is a nice to have.... celery still is important but an ability to mix up launchers where we may not require step isolation...
d
What exactly do you mean by at the pipeline level? That config says that only 20 pipeline runs can be happening at once
and you could also use tags to say that a certain pipeline can only have up to N runs happening at once
v
As in the whole pipeline ( even with n steps ) execute in a single process with celery EDIT as in no step isolation ...
d
yeah, we don't have a way to do pipeline-level queues in celery currently - the native run queue feature is intended to cover a lot of the same use cases that somebody would want to use celery for pipeline runs though, so I'd be curious what additional features celery has that would cause you to prefer it
v
I think we have had a discussion on that. Pretty much guard rails around our critical resources.
d
Right - I believe I understand the use case for wanting to apply global limits on steps, as well as global limits on pipelines. My understanding is that there are two potential ways to enforce a global limit on pipelines in Dagster - one using Celery (which does not exist yet) and one using the QueuedRunCoordinator (which does exist). With both, you can set up queues and apply global pipeline run limits to those queues. I'm curious what celery gives you to accomplish that goal that the Dagster-native/QueuedRunCoordinator solution does not. Happy to jump on a call quick too to get on the same page.
v
Yep, does tomorrow suit you ?
I can get on like 10 AM EST ?
d
for sure, I'll send something out
v
thanks