I would like to be able to use EFS when using AWS ECS with Dagster Cloud.
For example, when using dagster-dbt, we need run_results.json to rerun a query that failed in a dbt run. run_results.json is stored in the project as a fixed file name as the final run result, so it must be stored in the persistent file system The run_results.json is stored in the project with a fixed file name as the final run result, so the run_results.json will be lost if it is not stored in the persistent file system.
(and then you would presumably set up the EFS volumes themselves separately outside of dagster)
04/05/2023, 8:52 PM
Thank you Daniel, I think that adding that option would work well.
04/06/2023, 1:24 PM
There's a PR to add this for open source here: https://github.com/dagster-io/dagster/pull/13409 - and then adding it to Cloud is straightfoward. I think it's likely that we can add this in the release next Wednesday but will let you know if that changes.