https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Rubén Briones

05/05/2023, 9:43 AM
Hello, my
DAGSTER_HOME/history/runs
folder size is around 8GB, and
DAGSTER_HOME/storage
is about 1GB. It is safe to remove some of this folders? Or how can I free up some space? I would like to remove old logs, but without removing the sucess/fail state of old partitions. Thank you so much in advance!
t

Tim Castillo

05/05/2023, 11:49 AM
Hi! Our recommended way for reducing the size of these databases is to occasionally delete old and unneeded records, like runs from a significant period of time ago. Here's a Github discussion that shows you how to clean up old runs https://github.com/dagster-io/dagster/discussions/12047
r

Rubén Briones

05/05/2023, 1:30 PM
Hi Tim, thanks for the fast answer. But in our case 100% of the runs are partitions, so if we delete the old runs then we would have to be careful for not re-running old partitions that will be shown as "missing" after the cleanup. Is it not possible to only delete the logs, but keeping the SUCCEED status of the runs/partitions?
t

Tim Castillo

05/05/2023, 1:33 PM
Hmm, there might be a better way, but after deleting, you can tell Dagster that those partition materializations already happened https://github.com/dagster-io/dagster/discussions/12561 Would that fit the bill?
r

Rubén Briones

05/05/2023, 1:38 PM
Looks good, we will try it. Thanks Tim!
Only one more question, is it possible that the
history
folder is so heavy because we are using SQLite as database (the default), instead of Postgres?
98 Views