The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Q:  Why are Asset Materialisation details only viewable in Dagit (stored?) when using a Postgres backend?  What is it about SQLite that makes it unsuitable?

Our default Sqlite implementation shards all event logs (including asset materializations) by run_id.  This makes it impossible to generate views that indexes by asset keys across runs.  We do have an implementation that does not shard, that does support these asset views.  It can be configured in your `dagster.yaml` using the following:
```event_log_storage:
  module: dagster.core.storage.event_log.sqlite.consolidated_sqlite_event_log
  class: ConsolidatedSqliteEventLogStorage```

You may run into some scaling issues after running a ton of runs, because the single event_log sqlite file might get extremely large.

FWIW; it looks like we also need a `base_dir` config; ala:

```event_log_storage:
  module: dagster.core.storage.event_log.sqlite.consolidated_sqlite_event_log
  class: ConsolidatedSqliteEventLogStorage
  config:
    base_dir: "/Users/dlaing/workspace/tanzu-dwh-dagster/shared/tanzu-dwh-dagster-home/logs"```