👋 Question regarding retention for

event_logs

and

compute_logs

. We’ve encountered a case where every ~12m our Dagster database CPUs are nearly pegged for ~10m and have quickly tracked it down to the following query:

SELECT event_logs.id, event_logs.event 
FROM event_logs ORDER BY event_logs.timestamp DESC, event_logs.id DESC 
 LIMIT ?

I see a fix for this just got merged yesterday(!) (there’s no index on

timestamp

, so makes sense that the current query is so, so slow): https://github.com/dagster-io/dagster/pull/6620 While investigating, though, I noticed that our

event_logs

table has accumulated many millions of rows, seemingly retained forever. I see this related issue as well: https://github.com/dagster-io/dagster/issues/4497 We’ve also noticed similar seemingly-infinite retention of our

compute_logs

(we use S3). Finally, my question. Until such cleanup is a first-class feature of Dagster, what is safe for us to prune out-of-band? Can we just periodically delete rows older than a given

timestamp

from

event_logs

and use S3 lifecycle rules for

compute_logs

? Or will attempting to do so violate internal consistency assumptions made by Dagster and result in a bad time? Thanks for any guidance here.

The answer to this kind of depends on your jobs and what views you’re relying on in dagit. It’s generally pretty safe to delete compute logs if you don’t have a need to read the stdout/stderr. In dagit, this will result in just a

no compute logs available

message. The event log table powers two main views: the individual Run view, and the asset details view. The first one is self-explanatory: we need to fetch the events to display on the run page, including all of the op timing. The second one are all the cross-run materialization events going back in time.

Thanks, @prha. I think that answers my question, but just to confirm, both

event_logs

and

compute_logs

are strictly for “human” consumption via the Dagit UI? As in, Dagster isn’t using them to make any scheduling decisions? That’s the particular case that I’m worried about (more for

event_logs

), where we DELETE some old rows and then scheduling goes haywire because Dagster needs the full event history from the beginning of time to decide what to do.

For compute logs, I think we’re robust to failures in fetching them from S3.

Ohh, that’s good to be aware of, I’ll definitely have to check for that case before actually deleting anything. Thanks, @prha.

All that being said, aside from the sheer storage, we are striving to make dagster / dagit performant so that you will not have to delete metadata in order to keep it functionally operational

Great to hear! I think https://github.com/dagster-io/dagster/pull/6620 will help with the immediate issue that surfaced this for us (that 100%-CPU-for-10-minutes

event_logs

query). That said, we’re still seeing many GiB/month of growth in

event_logs

, and presumably that will only ramp up further as we migrate more jobs (we’re currently only running a tiny fraction within Dagster), so having an an official, out-of-the-box way to manage retention would be a great feature to have.

We have this issue that talks about pruning old runs / event logs, that you could upvote and track: https://github.com/dagster-io/dagster/issues/4100

Upvoted.