Hello everyone! When trying to separate the

event_logs

from the

run_storage

and

schedule_storage

(Sqlite for event logs and PostgreSQL for the others) we are not able to see any logs happening inside a job on the Dagit UI. Note that when only using a PostgreSQL database this worked fine. How can we still be able to see all the logs inside a job on the Dagit UI while keeping the separate databases? Please find below (part of) the Dagster.yaml Before:

storage:
  postgres:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432

After:

event_log_storage:
  module: dagster._core.storage.event_log
  class: SqliteEventLogStorage
  config:
    base_dir: /opt/dagster/dagster_home/event_logs

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432

This is running in ECS? Are you doing anything to persist your storage separately from the ECS task? Otherwise, you’re probably losing the sqlite file as soon as the task finishes running.

Hi Jordan, sorry for the late reply, but yes this looks like something caused by persistence of data! Thanks :)