Hi guys, I need to execute dask code inside a soli...
# announcements
d
Hi guys, I need to execute dask code inside a solid. This dask code connects to external cluster. Is it possible? @daniel @prha
p
Hi David. I’m not as familiar with our dask executor. Have you already looked at the docs page on the executor: https://docs.dagster.io/deploying/dask
m
this should be possible -- are you encountering issues?
d
@prha Dask executor allows to distribute solids between different dask workers. However, I would like to execute dask code inside a solid
k
We do lots of Dask execution within solids (which is different than running a pipeline through the Dask executor, as you noted). I’ve recently been publishing our code to merge back into the Dagster project. Once https://github.com/dagster-io/dagster/pull/2811 is merged, you can use it to declare a
dask
resource to support the computation of Dask graphs within and across solids.
There’s also related enhancements to the DataFrame type in https://github.com/dagster-io/dagster/pull/2821 and https://github.com/dagster-io/dagster/pull/2888. And, there’ll be more goodies coming after these get through!
👍 2
d
@prha @King Chung Huang thank you for your help, I make it work. However, I don´t understand dagster-dask-executor
Is it possible to point it to a remote cluster? I setup this cluster with dask-scheduler and dask-worker
I would like dagster-dask-executor to the scheduler
k
@David Lacalle Castillo, unfortunately that’s not possible with the existing Dask executor. I did a different implementation of Client & Cluster setup for the Dask resource, which does support connecting to an existing cluster. I’m thinking of following up later on to bring the implementations in the executor and resource together so that they’re equally capable.