Giacomo Colombo
03/05/2024, 2:29 PMDAGSTER_HOME
environment variable just today, but this moring! Wait, this means that specifying the DAGSTER_HOME
the storage folder with all result of materialization will be always available even offline without being removed from a temporary file?
So it means that every time I will restart a Dagster service I will get all my asset materialized in previous session already materialized ?Scout
03/05/2024, 2:29 PMDAGSTER_HOME
environment variable is an important step in ensuring that your Dagster instance has a persistent location to store metadata, including the results of asset materializations. When DAGSTER_HOME
is set, Dagster uses the specified directory to store various instance-related data, such as run history, event logs, and output from materialized assets.
If you do not set DAGSTER_HOME
, Dagster will default to using a temporary directory for storage, which means that all data will be lost when the Dagster process exits. By setting DAGSTER_HOME
to a specific directory on your filesystem, you ensure that the data is retained across Dagster service restarts, and you can access the results of previous materializations even when offline.
It's important to note that the actual storage of materialized asset outputs is handled by the configured IO manager, which is responsible for writing outputs to and reading inputs from storage. If you are using the default filesystem IO manager and you have set DAGSTER_HOME
, the outputs will be stored in a subdirectory within DAGSTER_HOME
unless otherwise configured.
In summary, setting DAGSTER_HOME
allows you to have a consistent and persistent location for your Dagster instance's metadata and materialized asset data, which will be available across service restarts and offline access. DagsterInstance() on Dagster Cloud Serve... button [docs] - dagster.yaml file per project button Dagster Docs button Using environment variables and secrets ... button Thank you for your feedback!Giacomo Colombo
03/05/2024, 5:19 PM