https://dagster.io/ logo
k

King Chung Huang

07/05/2020, 3:27 AM
With the
dask_executor
, how do I connect to an existing Dask cluster? The documentation (https://docs.dagster.io/_apidocs/libraries/dagster_dask#dagster_dask.dask_executor) seems to suggest that I can configure
address
on the
local
type. But, doing so results in a
TypeError: __init__() got an unexpected keyword argument 'address'
. Here's how I currently have the config.
Copy code
execution:
  dask:
    config:
      cluster:
        local:
          address: <tcp://xxxxxxxxxxxxx:8786>
d

David

07/05/2020, 12:25 PM
When using
local
,
dask_executer
is executing a
LocalCluster
so you need to use LocalCluster parameters.
k

King Chung Huang

07/05/2020, 4:29 PM
So, that means it's not possible to connect to an existing cluster, then?
d

David

07/05/2020, 5:22 PM
Before that
dask_executer
runs only with
local
mode, so it was initialized directly with
Client
and has the parameter address. Now,
dask_executer
can run with serval clusters (e.g. yarn, kubs...). so the
local
cluster is no longer executing directly with
Client
, only with
LocalCluster
.
k

King Chung Huang

07/05/2020, 5:45 PM
Hmm, looking at the
dagster_dask
code, I see that cluster objects like
LocalCluster
are being created and then passed to
Client
. The problem with that though is there's no way to connect to an existing scheduler.
5 Views