Working on setting Dagster up with a Dask cluster running on

Question

Working on setting Dagster up with a Dask cluster running on top of an SGE cluster When I follow the docs for Dask deployment I get ```AttributeError Client object has no attribute periodic callbacks ```

alex · Answer

whats the full stack trace i don t see us calling that code directly

Justin Swaney · Answer

```2020 03 18 23 25 13 dagster ERROR dask pipeline 89634b3d 7383 4f94 8ecd 777b0b8672e2 PIPELINE FAILURE Execution of pipeline dask pipeline failed Traceback most recent call last File shared anaconda3 envs fulcrumseek bin dagster line 11 in lt module gt sys exit main File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster cli init py line 38 in main cli obj= pylint disable=E1123 File shared anaconda3 envs fulcrumseek lib python3 7 site packages click core py line 829 in call return self main args kwargs File shared anaconda3 envs fulcrumseek lib python3 7 site packages click core py line 782 in main rv = self invoke ctx File shared anaconda3 envs fulcrumseek lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File shared anaconda3 envs fulcrumseek lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File shared anaconda3 envs fulcrumseek lib python3 7 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File shared anaconda3 envs fulcrumseek lib python3 7 site packages click core py line 610 in invoke return callback args kwargs File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster cli pipeline py line 317 in pipeline execute command execute execute command env kwargs mode File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster cli pipeline py line 322 in execute execute command return do execute command pipeline env mode File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster cli pipeline py line 345 in do execute command raise on error=False File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster core execution api py line 200 in execute pipeline pipeline execution iterator pipeline context execution plan pipeline run File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster core execution api py line 77 in pipeline execution iterator pipeline context execution plan=execution plan pipeline run=pipeline run File shared anaconda3 envs fulcrumseek lib python3 7 site packages dagster dask engine py line 83 in execute with dask distributed Client dask config build dict pipeline name as client File shared anaconda3 envs fulcrumseek lib python3 7 site packages distributed client py line 641 in init Unexpected keyword arguments format str sorted kwargs ValueError Unexpected keyword arguments threads per worker Exception ignored in lt function Client del at 0x7fd7e3d21b90 gt Traceback most recent call last File shared anaconda3 envs fulcrumseek lib python3 7 site packages distributed client py line 1131 in del File shared anaconda3 envs fulcrumseek lib python3 7 site packages distributed client py line 1346 in close AttributeError Client object has no attribute periodic callbacks ```

alex · Answer

gt Unexpected keyword arguments threads per worker

alex · Answer

seems to be the real issue

alex · Answer

alex · Answer

cc < nate>

nate · Answer

thanks for sleuthing this < alex> I can get a fix out soon

nate · Answer

filed <https github com dagster io dagster issues 2295|https github com dagster io dagster issues 2295>

Justin Swaney · Answer

It looks like they made this change <https github com dask distributed commit 856bf29c0c45cc4411e47575fb8f637dc0cf9b77 diff 96a27223dc91b5c9ea3d03684d79ad3f|here> and when I revert to `distributed=v2 5 2` it gets a little further and actually writes intermediates to S3