Hello! I am quite new to Dagster and am experimenting with it lately. Perhaps I am missing something, I am trying to find out if Dagster support resource management natively to limit parallelism during executions (i.e. similar to https://luigi.readthedocs.io/en/stable/api/luigi.task.html?highlight=resources#luigi.task.Task.resources), but haven't found anything particularly interesting on the documentation site (other than https://docs.dagster.io/docs/deploying/dask#managing-compute-resources-with-dask which forces me to use dask, which I am avoiding for now) Would any kind soul point me to the right direction/share some dagster recipe for achieving resources management please, thanks a bunch 🙏

hi Ryan, can you say a little more about the problem you're trying to solve? it's possible to limit or enable parallelism at several different levels in Dagster, but typically this is a characteristic of the executor you choose to run your pipelines. for instance, if you're running on celery, you can use set up a number of celery queues with independent parallelism limits and route dagster tasks appropriately; or on dask, you can use dask's facilities, etc.

hi! an example would be something like a situation where i am given X amount of concurrent query slots in Redshift, and i would like to configure dagster to not concurrently executes >X solids that uses Redshift, because i know they would just mostly hang until a query slot is free

yep, so currently folks are using the celery-based executor in prod to solve this exact issue

thanks for the confirmation that it is indeed supported 👍