https://dagster.io/ logo
Title
a

Aki Iwa

04/05/2023, 5:37 AM
Hi. I would like to be able to use EFS when using AWS ECS with Dagster Cloud. For example, when using dagster-dbt, we need run_results.json to rerun a query that failed in a dbt run. run_results.json is stored in the project as a fixed file name as the final run result, so it must be stored in the persistent file system The run_results.json is stored in the project with a fixed file name as the final run result, so the run_results.json will be lost if it is not stored in the persistent file system.
d

daniel

04/05/2023, 2:34 PM
This sounds very reasonable and like something we could add support for pretty quickly - would a "volumes" and "mountPoints" config option matching the format here that you can configure do the trick? https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html
(and then you would presumably set up the EFS volumes themselves separately outside of dagster)
a

Aki Iwa

04/05/2023, 8:52 PM
Thank you Daniel, I think that adding that option would work well.
d

daniel

04/06/2023, 1:24 PM
There's a PR to add this for open source here: https://github.com/dagster-io/dagster/pull/13409 - and then adding it to Cloud is straightfoward. I think it's likely that we can add this in the release next Wednesday but will let you know if that changes.
:daggy-love: 1