Hi guys, I need to execute dask code inside a solid. This dask code connects to external cluster. Is it possible? @daniel @prha

Hi David. I’m not as familiar with our dask executor. Have you already looked at the docs page on the executor: https://docs.dagster.io/deploying/dask

this should be possible -- are you encountering issues?

@prha Dask executor allows to distribute solids between different dask workers. However, I would like to execute dask code inside a solid

We do lots of Dask execution within solids (which is different than running a pipeline through the Dask executor, as you noted). I’ve recently been publishing our code to merge back into the Dagster project. Once https://github.com/dagster-io/dagster/pull/2811 is merged, you can use it to declare a

dask

resource to support the computation of Dask graphs within and across solids.

@prha @King Chung Huang thank you for your help, I make it work. However, I don´t understand dagster-dask-executor

@David Lacalle Castillo, unfortunately that’s not possible with the existing Dask executor. I did a different implementation of Client & Cluster setup for the Dask resource, which does support connecting to an existing cluster. I’m thinking of following up later on to bring the implementations in the executor and resource together so that they’re equally capable.