I launched a run which includes a subset of assets (which succeeded), then launched another run with downstream assets. Now I get an error something like:

FileNotFoundError: [Errno 2] No such file or directory: '/opt/dagster/dagster_home/storage/project_settings

. I think this is due to the fact that every run is inside its own Docker container. How to prevent this? Or is this a valid use case at all?

Based on the documentation, it seems that you are encountering a

FileNotFoundError

because the file or directory expected by the downstream assets is not present in the storage location. This can happen in containerized environments like Docker where each run may be isolated and not have access to the files created by previous runs unless the storage is configured to be persistent across containers. To prevent this issue, you would typically need to configure your storage to be persistent and accessible to all runs. For example, you could use an external storage system like a cloud blob storage (e.g., AWS S3, GCP Cloud Storage) that is accessible from all your Docker containers. This way, when one run writes to the storage, subsequent runs can read from the same location regardless of the container they are executed in. The documentation mentions that you can switch from using the default filesystem IO manager to something that persists across containers like blob storage. This would allow you to persist run storage when re-executing runs in Kubernetes, which is similar to your use case with Docker containers. To configure persistent storage, you would need to set up an IO manager that points to your persistent storage backend. Here is a general approach based on the documentation: 1. Choose a persistent storage solution that is accessible from all your Docker containers. 2. Configure an IO manager in your Dagster system to use this persistent storage. 3. Update your asset storage definitions to use the configured IO manager. Please note that the exact steps to configure persistent storage and IO managers would depend on your specific storage backend and Dagster setup. You can refer to the Dagster documentation for more details on configuring IO managers and persistent storage solutions. If you continue to face issues, it might be helpful to ensure that your

DAGSTER_HOME

environment variable is set correctly and that the storage configuration in your

dagster.yaml

is pointing to the correct persistent storage location. If the problem persists, you may want to provide more details about your setup, including how your Docker containers are configured and how your Dagster assets are set up to manage storage.