Hello Again I ve worked on my k8s integration of dagster Aft dagster #announcements

Hello Again ! I've worked on my k8s integration of...

Gaetan DELBART

03/17/2020, 6:50 AM

Hello Again ! I've worked on my k8s integration of dagster. After launching a

helm install dagster helm/dagster

, my celery workers fail to start The result of a

kubectl logs pod-name

Copy code

Traceback (most recent call last):
  File "/usr/local/bin/dagster-celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_celery/cli.py", line 159, in worker_start_command
    pythonpath = get_config_dir(config_yaml)
  File "/usr/local/lib/python3.7/site-packages/dagster_celery/cli.py", line 69, in get_config_dir
    if 'broker' in validated_config:
TypeError: argument of type 'NoneType' is not iterable

if I run a kubectl exec on the pod, and look at the content of the celery-config.yaml, it looks like this

Copy code

execution:
  celery:
      broker:  "<pyamqp://test>:test@dagster-rabbitmq:5672//"
      backend: 
      config_source:

So, it looks like the helm template is working as expected. Dos anyone has any advice about this ?

Gaetan DELBART

03/17/2020, 7:55 AM

After digging around in the source code, I find this comment in https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery/dagster_celery/executor.py

Copy code

execution:
          celery:
            config:
              broker: '<pyamqp://guest>@localhost//'  # Optional[str]: The URL of the Celery broker
              backend: 'rpc://' # Optional[str]: The URL of the Celery results backend
              include: ['my_module'] # Optional[List[str]]: Modules every worker should import
              config_source: # Dict[str, Any]: Any additional parameters to pass to the
                  #...       # Celery workers. This dict will be passed as the `config_source`
                  #...       # argument of celery.Celery().

But, even if I change the helm template, I can't get the worker to start. (Same error as the one above)

Gaetan DELBART

03/17/2020, 10:26 AM

Ok found it, I've found what went wrong ! The config can't have empty lines

Copy code

execution:
  celery:
      broker:  "<pyamqp://test>:test@dagster-rabbitmq:5672//"

Gaetan DELBART

03/17/2020, 10:27 AM

So, I've changed https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-k8s/helm/dagster/templates/configmap-celery.yaml to prevent empty lines if the config is not enable !

nate

03/17/2020, 3:28 PM

ah thanks for flagging this @Gaetan DELBART, and for debugging the issue! I filed https://github.com/dagster-io/dagster/issues/2289 and will get this fixed shortly

❤️ 1

nate

03/18/2020, 8:09 PM

@Gaetan DELBART this is now fixed on master and will be in the next release—thanks again for the report!

2 Views

Open in Slack

Previous Next