https://dagster.io/ logo
Powered by
s

szalai1

09/25/2020, 3:55 PM
Hey, 
ModuleNotFoundError: No module named 'celery.backends.amqp'
I'm getting this error on all of our pipelines. Full error in thread. (let me know if this should go to 
Discussions
)
Copy code
ModuleNotFoundError: No module named 'celery.backends.amqp'
  File "/usr/local/lib/python3.8/site-packages/dagster_celery/core_execution_loop.py", line 115, in core_celery_execution_loop
    step_results[step.key] = step_execution_fn(
  File "/usr/local/lib/python3.8/site-packages/dagster_celery_k8s/executor.py", line 217, in _submit_task_k8s_job
    return task_signature.apply_async(
  File "/usr/local/lib/python3.8/site-packages/celery/canvas.py", line 222, in apply_async
    return _apply(args, kwargs, **options)
  File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line 561, in apply_async
    return app.send_task(
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 716, in send_task
    self.backend.on_task_call(P, task_id)
  File "/usr/local/lib/python3.8/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 1169, in backend
    return self._get_backend()
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 884, in _get_backend
    backend, url = backends.by_url(
  File "/usr/local/lib/python3.8/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/usr/local/lib/python3.8/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/lib/python3.8/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
a

alex

09/25/2020, 3:59 PM
what version of 
celery
do you have?
looks like they just released 
5.0.0
yesterday
s

szalai1

09/25/2020, 4:01 PM
dagster/k8s-example:0.9.7
and we run 
0.9.11
on dagster (I've upgraded it from 0.9.7, because of this error)
j

Josh Karlin

09/25/2020, 4:04 PM
Yeah I had this yesterday evening. Need to pin celery at <5.0.0 as celery 5 is not backwards compatible
s

szalai1

09/25/2020, 4:06 PM
shouldn't 
dagster/k8s-example:0.9.7
serve as a pin?
a

alex

09/25/2020, 4:07 PM
I’m not sure how to interpret that you are using 
dagster/k8s-example:0.9.7
but also
and we run 0.9.11 on dagster
is it that you have 
dagster/k8s-example:0.9.7
set for the user code image, but you deployed using the 
0.9.11
helm chart ?
j

Josh Karlin

09/25/2020, 4:09 PM
If you have updated your environment to get dagster 0.9.11 then I think it will update celery to 5.0.0. From what I could see, dagster specifies something like celery >4.3 with no upper limit. The dagster/k8s-example image by itself should still work
s

szalai1

09/25/2020, 4:10 PM
the dagster instance uses 0.9.11 (installed with pip) the celery workers use the 
dagster/k8s-example:0.9.7
image from your docker repo
I will update celery workers too
ok, so I changed the worker image to 
image: 'dagster/k8s-celery-worker:0.9.11'
and getting the same error, now on the worker instance.
Copy code
User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,
[2020-09-25 16:20:01,438: CRITICAL/MainProcess] Unrecoverable error: ModuleNotFoundError("No module named 'celery.backends.amqp'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 41, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 112, in start
    self.on_start()
  File "/usr/local/lib/python3.7/site-packages/celery/apps/worker.py", line 136, in on_start
    self.emit_banner()
  File "/usr/local/lib/python3.7/site-packages/celery/apps/worker.py", line 159, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/usr/local/lib/python3.7/site-packages/celery/apps/worker.py", line 221, in startup_info
    results=self.app.backend.as_uri(),
  File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 1169, in backend
    return self._get_backend()
  File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 886, in _get_backend
    self.loader)
  File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'celery.backends.amqp'
a

alex

09/25/2020, 4:22 PM
5.0.0
Breaking Change Remove AMQP result backend
digging now - do you set the 
backend
for celery explicitly by chance?
no looks like that just what we default to
j

Josh Karlin

09/25/2020, 4:26 PM
It seems the 
dagster/k8s-celery-worker:0.9.11
image has celery 5.0.0! Unless imports have changed in dagster 0.9.11 then this image probably won't work
a

alex

09/25/2020, 4:28 PM
only fails if the 
backend
is explicitly set to 
ampq
, which we do in our 
k8s
helm chart but not in the 
dagster_celery
module it self
👍 1
s

szalai1

09/25/2020, 4:29 PM
removed it from worker config
now the worker fails without error message 🤦
a

alex

09/25/2020, 4:30 PM
thats not great
alright so to get back to working while I figure out what we need to do - I think you want to go back to using 
0.9.10
and then on the instance make sure to pin 
celery
to 
<5.0.0
unless you can figure out why what that failure is from when its goes back to defaulting to 
rpc://
for the backend, but that seems tough if you are getting no error message
s

szalai1

09/25/2020, 4:40 PM
found the problem
Copy code
Try 'celery status --help' for help.

Error: no such option: -A
a

alex

09/25/2020, 4:40 PM
😐
s

szalai1

09/25/2020, 4:40 PM
but going back to 0.9.10
a

alex

09/25/2020, 4:42 PM
hmm, I installed 
5.0.0
locally and 
-A
is still there
what did you when you
removed it from worker config
s

szalai1

09/25/2020, 4:48 PM
deleted the line from the config file, this one: 
backend: amqp
👍 1
a

alex

09/25/2020, 4:49 PM
ohhh i see 
celery status
doesn’t take 
-A
anymore
woof
ok
s

szalai1

09/25/2020, 4:51 PM
a

alex

09/25/2020, 4:52 PM
ah i meant 
0.9.9
(
0.9.10
was a botched release due to some tooling issues)
ok we are pretty sure we can fix everything we need by fixing the helm chart - working on final verification then will do the publish
thanks a ton for the report and responding with follow up information
c

cat

09/25/2020, 5:38 PM
@szalai1 to remove the -A, you can remove the livenessProbe field under the celery section of the helm chart
a

alex

09/25/2020, 6:07 PM
helm chart published - so hopefully next time you try to upgrade everything will work smoothly
s

szalai1

09/26/2020, 8:34 AM
thank you very much for the help. few things I noticed: • I tried to go back to 0.9.7 if I don’t install dagster-celery directly it will be installed and its version: 0.9.11 • if I install dagster_celery directyl celery will be installed but version 5.0.0 I could resolve this by installing 
celery==4.4.7
explicitly.
Powered by