Hi. I would like to be able to use EFS when using ...
# dagster-plus
a
Hi. I would like to be able to use EFS when using AWS ECS with Dagster Cloud. For example, when using dagster-dbt, we need run_results.json to rerun a query that failed in a dbt run. run_results.json is stored in the project as a fixed file name as the final run result, so it must be stored in the persistent file system The run_results.json is stored in the project with a fixed file name as the final run result, so the run_results.json will be lost if it is not stored in the persistent file system.
d
This sounds very reasonable and like something we could add support for pretty quickly - would a "volumes" and "mountPoints" config option matching the format here that you can configure do the trick? https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html
(and then you would presumably set up the EFS volumes themselves separately outside of dagster)
a
Thank you Daniel, I think that adding that option would work well.
d
There's a PR to add this for open source here: https://github.com/dagster-io/dagster/pull/13409 - and then adding it to Cloud is straightfoward. I think it's likely that we can add this in the release next Wednesday but will let you know if that changes.
daggy love 1