Hi all! I'm trying to deploy Dagster to a GCP Compute Engine VM (Ubuntu 22.04 LTS). I want to make both dagit and dagster-daemon run in background, so I push them into background like this:

export DAGSTER_HOME=/path/to/dagster_home
dagit -h 0.0.0.0 -p 3000 & disown
dagster-daemon run & disown

This works well at first glance - both processes are running and executing jobs correctly. However I observe three connected unexpected behaviors: 1. Logs from both dagit and dagster-daemon are written to current terminal (stderr?). See attached screenshot for an example when starting dagit. 2. When I close SSH connection to VM, dagit still works but all jobs fail with the error

OSError: [Errno 5] Input/output error

. Presumably, Dagit cannot get stdout and stderr because the terminal is closed. 3. According to this page, running command above should automatically make Dagit write logs to

$DAGSTER_HOME/logs

. However, when I check this folder after running a couple of jobs - the folder is empty. Instead, I only see some run logs per job in

$DAGSTER_HOME/storage/<random_UUID>/compute/logs

. Is this behavior expected? Given above, I expect: 1. dagit to write all logs to

$DAGSTER_HOME/logs

and don't write anything to stdout/stderr. 2. Do not fail any jobs after disowning processes via

& disown

and closing the terminal. I found a workaround by running both processes like this:

dagit -h 0.0.0.0 -p 3000 > /dev/null 2>&1 & disown
dagster-daemon run > /dev/null 2>&1 & disown

However this way I lose access to process logs (except job logs). And if I redirect output to a file, now I have to manually rotate the output log file. Any suggestions?

have you set this in your

dagster.yaml

?

compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: /path/to/directory

https://docs.dagster.io/deployment/dagster-instance#compute-log-storage

Hi @Kautsar Aqsa, thanks for checking. We have our compute logs set to be saved to GCS bucket (and it works), but Dagster still saves some to local storage and outputs to stderr/stdout. I assumed that this is intended, but now that you mentioned it it seems like the logs should be outputted to GCS only?

compute_logs:
  module: dagster_gcp.gcs.compute_log_manager
  class: GCSComputeLogManager
  config:
    bucket:
      env: DAGSTER_GCS_BUCKET