Hi, currently we are using ECS Run Launcher and Multiprocess Step Executor. We've created a simple resource for executing SFN. It works, but we need to use a beefy ECS instance to run multiple steps in parallel, as each step operates as a separate process, solely for executing SFN and polling for state, that is a huge waste of resources. I am curious if Pipes can help me with this... Plus is it possible to specify concurrency if I use Pipes?

Based on the documentation, Dagster Pipes is designed to work with existing data platform infrastructure and can be used to execute code in external environments, such as AWS Step Functions (SFN), while minimizing the additional dependencies in those environments. Pipes allow for parameter passing, streaming logging, and structured metadata back into Dagster's observability and data quality tools. Regarding your use case with ECS and SFN, Pipes could potentially help you execute SFN tasks without the need for a beefy ECS instance dedicated to running multiple steps in parallel. With Pipes, you could offload the execution of tasks to SFN and only use ECS to orchestrate these tasks, which could be more resource-efficient. As for specifying concurrency when using Pipes, the documentation does not explicitly mention concurrency controls within the context of Pipes. However, Dagster does provide mechanisms to control concurrency in general, such as setting

max_concurrent

for multiprocess execution and using

tag_concurrency_limits

to limit the number of steps with certain tags that can execute at once. To integrate Pipes with SFN, you would likely need to implement a custom Pipes client that can start SFN executions through the AWS SDK. This client would handle the interaction with SFN, including starting executions and polling for their state. The documentation provides an example of integrating with Databricks, which could serve as a conceptual starting point for integrating with SFN. Keep in mind that Pipes is an experimental feature, and you should refer to the documentation for the latest details on its capabilities and how to use it effectively in your specific scenario. If you decide to proceed with Pipes, you may need to explore and potentially contribute to its development to fit your needs fully.