Hi everyone; first up thanks for all the great work on dagster, really enjoying playing around with it so far. I've created a pipeline with multiple solids that can all be ran in parallel (potentially) but which all require a GPU. I can see a way to talk about resources in a distributed setting in dask: https://docs.dagster.io/deploying/dask is there a way to talk about resources for local execution? specifically for this pipeline to run each solid that needs access to the GPU needs to wait for the other to finish. I can clearly get the behaviour I want by ... putting explicit dependencies between each solid in the dag, i.e. turning

(A -> D, B -> D, C -> D)

into

(A -> B, B -> C, C -> D)

but that seems weird.

Hi Sina, you could also take a look at dagster-celery. That’s currently how we support those types of resource requirements

is there a nice example of using celery to run local dagster pipeline?

Sorry I missed this message! Here’s our doc page, https://docs.dagster.io/_apidocs/libraries/dagster_celery and https://docs.dagster.io/deploying/celery (note that if you are only running locally, the parts of the docs about persistent storage won’t apply

hey @Sina Samangooei we unfortunately haven’t documented the resource management in dagster-celery yet, but it was the reason we introduced this library. To use this, you can define two Celery queues: •

dagster

 <- N workers, default queue, most work happens here •

gpu-queue

 <- X dagster-celery workers (no more than X solids executing in parallel ), GPU workloads go here then you can tag your GPU solids like this and only X will ever run concurrently (even across multiple pipelines):

@solid(tags={'dagster-celery/queue': 'gpu-queue'})

FWIW 

dagster-celery

 also supports setting priorities for execution:

@solid(tags={'dagster-celery/priority': 3})

wow nice this is exactly what i need 😄 i'll dig through code/docs and see if i can make it go

Where do you define the variable X, if using K8s based celery, i.e. celeryK8sRunLauncher in the values.yaml file or somewhere?