Hello, I was hoping someone could offer some advice on how to implement this pipeline in dagster such that the API request sections are run concurrently as much as possible (or a simple compromise). The pipeline will operate for multiple files in a directory and the order of the files is not important so if possible the pipeline could run for multiple files concurrently. I could not find much about asynchronous solids in the docs and I'm quite new to all of this. Any advice would be appreciated. Thanks!

Hi Jack - you can use the multiprocess executor on your pipeline to run solids in parallel: https://docs.dagster.io/_apidocs/execution#dagster.multiprocess_executor

Thanks for the response Daniel! Is dagster compatible with using asyncio.ensure_future, and asyncio.gather in a solid?