Does anyone here used Keda with a redis broker for a dagster-celery-k8s deploy? I'm having a bit of trouble getting Keda to read the redis queue length (it seems to always read llen queue is zero). (Alternative approaches to autoscaling also welcome.)
n
Noah K
03/17/2021, 8:25 PM
keda yes, redis no
Noah K
03/17/2021, 8:25 PM
I would reeeeeeally recommend you not use Redis as your Celery broker unless you do not care about things getting dropped 😕
m
Matyas Tamas
03/17/2021, 8:27 PM
rabbitmq then?
Matyas Tamas
03/17/2021, 8:27 PM
keda seemed to have more direct support for that - thanks for the tip!
n
Noah K
03/17/2021, 10:15 PM
That's what I use (though differently than core)
Noah K
03/17/2021, 10:15 PM
I've been upgrading Keda's rabbit support specifically for our Dagster testing though 🙂