19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi Guys! I am trying to test Celery execution in m...
# announcementsi
Istvan Darvas
11/26/2020, 2:55 PMHi Guys! I am trying to test Celery execution in my local environment based on your example (using rabbitmq). I try to start the pipeline from dagit playground with execution celery, starting, then stucking in some waiting....local celery and dagit were started from the same directory...and no exception, it is not logging anything...I put some debug stuff into the thread if there is someone who can check and give me some advice. thanks in advance
Istvan Darvas
11/26/2020, 2:56 PMI try to go with this example:
Istvan Darvas
11/26/2020, 2:56 PMIstvan Darvas
11/26/2020, 2:57 PMfrom dagit I started like this
Istvan Darvas
11/26/2020, 2:57 PMlocal celery got something
Istvan Darvas
11/26/2020, 2:57 PMIstvan Darvas
11/26/2020, 2:58 PMdagit log says submitting
Istvan Darvas
11/26/2020, 3:00 PMcontinuously running / waiting unfortunatly I dont understand
Istvan Darvas
11/26/2020, 3:01 PMno exception, no error just running...What is the problem here? 😉 - what do I miss?
Istvan Darvas
11/26/2020, 3:02 PMI patched my code like it was in your example
c
cat
11/26/2020, 7:11 PMHmm could you try running ‘dagster-celery worker start -A dagster_celery.app -Q dagster’ to explicitly point the worker to the dagster queue? (Based on this https://github.com/celery/celery/issues/3759)
i
Istvan Darvas
11/26/2020, 7:13 PM@cat thanks, tomorrow, I am going to check this...
c
cat
11/26/2020, 7:13 PMFor reference, this is the celery worker command for k8s: https://github.com/dagster-io/dagster/blob/1d91ee8bccdbe3169745cf607ab89b677ffba37a/helm/dagster/templates/deployment-celery.yaml#L44
i
Istvan Darvas
11/26/2020, 7:14 PMthank you very much
c
cat
11/26/2020, 7:14 PMWhich also uses ‘-q dagster’
cat
11/26/2020, 7:14 PMYup lmk if that doesn’t work and I’ll keep digging
i
Istvan Darvas
11/26/2020, 7:15 PMthanks, I will give you a feedback...
Istvan Darvas
11/27/2020, 9:49 AM@cat Hi! So I did everything as you suggested but the behavior is just the same. I started celery (locally) like this: "dagster-celery worker start -A dagster_celery.app -q dagster --loglevel=INFO" and it is continuously running. maybe is it waiting for something?
Istvan Darvas
11/27/2020, 9:50 AMIstvan Darvas
11/27/2020, 10:09 AM@cat I have just read through the celery issue you mention, maybe this is my problem
Istvan Darvas
11/27/2020, 10:10 AMI am going to try to deploy celery on kubernetes, and test there...
👍 1
Istvan Darvas
11/27/2020, 10:10 AMthanks
c
cat
12/03/2020, 3:28 AMHey, just wanted to follow up and see if deploying celery on k8s went smoothly or if you ran into issues?
12 Views