Hi everyone; first up thanks for all the great wor...
# announcements
Hi everyone; first up thanks for all the great work on dagster, really enjoying playing around with it so far. I've created a pipeline with multiple solids that can all be ran in parallel (potentially) but which all require a GPU. I can see a way to talk about resources in a distributed setting in dask: https://docs.dagster.io/deploying/dask is there a way to talk about resources for local execution? specifically for this pipeline to run each solid that needs access to the GPU needs to wait for the other to finish. I can clearly get the behaviour I want by ... putting explicit dependencies between each solid in the dag, i.e. turning
(A -> D, B -> D, C -> D)
(A -> B, B -> C, C -> D)
but that seems weird.
soooo some internal discussions leads me to believe i was thinking about this wrong and I should maybe just use something with a bit more resource savvy to run my pipelines locally. playing with dask now 🙂
Hi Sina, you could also take a look at dagster-celery. That’s currently how we support those types of resource requirements
👍 1
is there a nice example of using celery to run local dagster pipeline?
Sorry I missed this message! Here’s our doc page, https://docs.dagster.io/_apidocs/libraries/dagster_celery and https://docs.dagster.io/deploying/celery (note that if you are only running locally, the parts of the docs about persistent storage won’t apply
@nate do we have a celery example?
hey @Sina Samangooei we unfortunately haven’t documented the resource management in dagster-celery yet, but it was the reason we introduced this library. To use this, you can define two Celery queues: •
 <- N workers, default queue, most work happens here •
 <- X dagster-celery workers (no more than X solids executing in parallel ), GPU workloads go here then you can tag your GPU solids like this and only X will ever run concurrently (even across multiple pipelines):
Copy code
@solid(tags={'dagster-celery/queue': 'gpu-queue'})
 also supports setting priorities for execution:
Copy code
@solid(tags={'dagster-celery/priority': 3})
wow nice this is exactly what i need 😄 i'll dig through code/docs and see if i can make it go
Where do you define the variable X, if using K8s based celery, i.e. celeryK8sRunLauncher in the values.yaml file or somewhere?