Hi, currently we are using ECS Run Launcher and Multiprocess Step Executor. We've created a simple resource for executing SFN. It works, but we need to use a beefy ECS instance to run multiple steps in parallel, as each step operates as a separate process, solely for executing SFN and polling for state, that is a huge waste of resources. I am curious if Pipes can help me with this... Plus is it possible to specify concurrency if I use Pipes?

curious if Pipes can help me with this

I don’t think so, pipes just does the pattern of “invoke external process and poll for it” and eases moving metadata and logs back from external process. heres the issue for in process concurrency https://github.com/dagster-io/dagster/issues/4041