https://dagster.io/ logo
c

Chris Roth

05/19/2020, 1:01 AM
i'm having an issue with solids getting stuck waiting in queue - my whole dagster cluster is stuck atm, anyone seen this happen before?
the last thing i see in the log is "Submitting celery task for step "authenticate.compute" to queue "dagster"."
currently trying to roll back commits to debug, but i wish there was a better way to debug this
n

nate

05/19/2020, 1:08 AM
are you scheduling tasks onto a non-default queue? this can happen if no celery workers are servicing a queue
not sure if you have flower set up but it can be helpful to debug stuff like this
c

Chris Roth

05/19/2020, 1:08 AM
nope, haven't changed the queue at all
hm i have flower but not sure what to look for in flower
n

nate

05/19/2020, 1:09 AM
do you see the task for authenticate.compute show up there?
c

Chris Roth

05/19/2020, 1:14 AM
nope
but i haven't changed anything with my celery environment, that's the weird part
just gonna keep walking about one commit at a time 😕
figured it out - adding to $PYTHONPATH prevents dagster celery worker from connecting to redis