https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Mark Fickett

06/01/2022, 8:44 PM
Using Dagster Cloud with k8s executor, where do I put things that would normally go in
$DAGSTER_HOME/dagster.yaml
? Looking for
managed_python_loggers
and
dagster_handler_config
in particular. I packaged my
dagster.yaml
into my Docker image and set
DAGSTER_HOME
but the custom log handling is not being applied.
r

rex

06/01/2022, 8:46 PM
👀 1
m

Mark Fickett

06/02/2022, 3:27 PM
Thanks, I'm trying that out. If I'm setting the log configuration for the agent, then that will apply to all jobs, right? With Dagster Cloud we currently have one "prod" deployment, and only some jobs' container images will have the custom log handlers that I want to use. So maybe I should create another deployment + agent just for the jobs that have the custom handler? I tried doing this:
Copy code
helm show values dagster-cloud/dagster-cloud-agent > values.yaml
Then I edited
values.yaml
and added in my logging:
Copy code
python_logs:
  pythonLogLevel: INFO
  managedPythonLoggers:
    # Capture logs from all Python loggers into Dagster's UI.
    - root
  # Specify additional handlers for the console logs (which include all Python logs too).
  dagsterHandlerConfig:
    handlers:
      combinedJson:
        class: processing.common.log.JsonFileHandler
        level: INFO
      otelSpan:
        class: processing.common.log.OtelSpanEventHandler
        level: INFO
And finally tried to update the agent with those values:
Copy code
helm upgrade    --install user-cloud dagster-cloud/dagster-cloud-agent    --namespace dagster-cloud  -f values.yaml
However, when I run (a) a job that doesn't have my custom handlers it still works fine, and (b) the job that does have those handlers in its image, they don't seem to take effect.
d

daniel

06/02/2022, 3:37 PM
Hey Mark - just flagging that in both dagster open source and cloud this particular line could lead to slowdown if there are a lot of logs, we've seen a couple of users run into trouble after setting it:
Copy code
managedPythonLoggers:
    # Capture logs from all Python loggers into Dagster's UI.
    - root
But yeah, instance-level log configuration will apply to all jobs launched from that agent currently. We'll look into why that isn't working the way you would expect
m

Mark Fickett

06/02/2022, 3:40 PM
Yep, it does slow down the Dagit UI, but it's more useful to capture the logs than not. Also I'm trying to get all the logs from custom loggers as well as context.log into my custom handlers (JSON file for fluentd + OpenTelemetry), and pulling all the logs into Dagster seemed like the simplest way. I tried
helm show values
again and what I set didn't show up, so I think the issue may be with my helm calls (totally new territory for me).
Ah, I think
helm show values dagster-cloud/dagster-cloud-agent > values.yaml
is actually just pulling the defaults, so settings wouldn't echo back. I see that my deployment value from
values.yaml
is taking effect as expected. (I've also set up a 2nd deployment so I can keep my agents separate, very easy to do!)
r

rex

06/02/2022, 4:04 PM
if you want to see the values of your current Helm release, you should do
helm get values user-cloud -n dagster-cloud
where
user-cloud
is the name of your release, and
dagster-cloud
is your namespace
ty thankyou 1
m

Mark Fickett

06/02/2022, 4:13 PM
Found a typo converting from
lower_under
case that is in
dagster.yaml
to the
camelCase
that's in the Helm chart. Now It's picking up my log customization. Thanks for all the help!
8 Views