https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
b

Binoy Shah

03/30/2023, 4:23 PM
Is there a way to configure logging as JSON logging globally, we have Kubernetes deployment and NewRelic log ingestion at cluster level..I do not want to touch the code, is there a way to switch the logger formats and loggers to JSON formatted logging only
b

ben

03/30/2023, 9:03 PM
Hi Binoy, this may not match exactly what you’re hoping for, but you can specify a logger in a singular place (e.g. once per code location) by attaching it to the
Definitions
object: https://docs.dagster.io/concepts/logging/loggers#specifying-default-code-location-loggers You’d have to modify the code, but only in one place
b

Binoy Shah

03/30/2023, 9:09 PM
Hi Ben, thanks for the link, I have seen it. While I love the framework, logging feels a bit patchy and too many knobs kind of affair.
b

ben

03/30/2023, 9:10 PM
What would your ideal logging setup look like? This is something we’ve been talking about improving internally so any feedback would be very useful for us
b

Binoy Shah

03/30/2023, 9:11 PM
It would be rather great to see multiple global loggers configured with
dagster.yml
or some sort of
logger.yaml
that controls various default loggers
Copy code
dagit_logger:
  logger_class: dagster.logging.ConsoleLogger
  logger_format: xxx
  log_level: INFO
  enable_color: true
daemon_logger:
  logger_class: dagster.logging.JSONLogger
  combine_stacktrace: true
  log_level: ERROR
job_logger:
  logger_class: dagster.logging.FancyTracingLogger 
  log_level: DEBUG
  log_format: xxx
This file if it’s relied upon, then it can be controlled by kubernetes Helm values too and each of them can internally be associated with some kind of optional persistent backend Same works within docker, or overridden by a volume mount and locally too. Dagster project generator dumps out a starter
logger.yaml
file with commented loggers and dagster/dagit start with default values
So if this is sample of builtin logger https://docs.dagster.io/concepts/logging/loggers#configuring-built-in-loggers Questions 1. how do I override to use json formatting ? 2. How do find what other loggers and config values are allowed/used ? 3. documentation shows a single entry under heading of “Configuring built in loggers” Where are additional configuration fields Tip Here • It has a statement “add the following snippet to your config YAML”. why does’nt it say what is that config yaml.. is it
dagster.yaml
or
workspace.yaml
, it takes a mental mapping to figure out which
.yaml
file this goes under
a

Andrea Giardini

03/31/2023, 7:14 AM
☝️ Good point made here (and something that I am facing as well). I think there are certain settings (like Json logging) that make sense to have instance-wide rather than per-repo/per-definition.
10 Views