Jack Whelan

06/11/2021, 10:03 AM
Hello, I was hoping someone could offer some advice on how to implement this pipeline in dagster such that the API request sections are run concurrently as much as possible (or a simple compromise). The pipeline will operate for multiple files in a directory and the order of the files is not important so if possible the pipeline could run for multiple files concurrently. I could not find much about asynchronous solids in the docs and I'm quite new to all of this. Any advice would be appreciated. Thanks!


06/11/2021, 1:44 PM
Hi Jack - you can use the multiprocess executor on your pipeline to run solids in parallel:

Jack Whelan

06/12/2021, 3:50 AM
Thanks for the response Daniel! Is dagster compatible with using asyncio.ensure_future, and asyncio.gather in a solid?