https://dagster.io/ logo
#announcements
Title
# announcements
m

mrdavidlaing

12/23/2020, 6:37 PM
Q: Why are Asset Materialisation details only viewable in Dagit (stored?) when using a Postgres backend? What is it about SQLite that makes it unsuitable?
👀 2
p

prha

12/23/2020, 11:44 PM
Our default Sqlite implementation shards all event logs (including asset materializations) by run_id. This makes it impossible to generate views that indexes by asset keys across runs. We do have an implementation that does not shard, that does support these asset views. It can be configured in your
dagster.yaml
using the following:
Copy code
event_log_storage:
  module: dagster.core.storage.event_log.sqlite.consolidated_sqlite_event_log
  class: ConsolidatedSqliteEventLogStorage
You may run into some scaling issues after running a ton of runs, because the single event_log sqlite file might get extremely large.
m

mrdavidlaing

12/24/2020, 10:16 AM
Good to know; thanks!
FWIW; it looks like we also need a
base_dir
config; ala:
Copy code
event_log_storage:
  module: dagster.core.storage.event_log.sqlite.consolidated_sqlite_event_log
  class: ConsolidatedSqliteEventLogStorage
  config:
    base_dir: "/Users/dlaing/workspace/tanzu-dwh-dagster/shared/tanzu-dwh-dagster-home/logs"
👍 1