I was interested if it was possible to store event logs in S3 or Google Cloud Storage ? For example instead of storing a SQLite database locally, just store in some bucket?
04/03/2023, 4:14 PM
Hi Alfie - currently, the Dagster web server needs to be able to query the event log to be able to serve many of the web pages, so it needs to be backed by a SQL database
04/03/2023, 4:26 PM
Hi Sandy, that makes sense. Thanks for the help 🙂
04/05/2023, 2:41 PM
I’m currently trying out using Timescale DB which can offload old data to S3 but it keeps it queryable through Postgres.
There are a few rough spots that I’ve run into so far, but it seems like a promising option to deal with the volume of event logs.