Logging question: I have a few file loggers configured via

dictConfig

in an application I'm porting to Dagster. They make use of a logging filter as well as file handlers. Is there a way to make my

logging.dictConfig

call get along with Dagster logging setup? I can use the

python_loggers

configuration in

dagster.yaml

to capture their output into Dagit, but that only works if I remove the

dictConfig

call. I see that I could add handlers and formatters via

dagster_handler_config

as described in the docs, but it's a little painful to port my current logging config dict setup to

dagster.yaml

, and I would also lose the filters.

cc @owen in case we have any additional tips

unfortunately, I think your workaround is the best option at the moment

OK, thanks for checking in. The case where this is trickiest is attaching handlers/filters to the root logger. I think I can set up handlers in

dagster.yaml

which will apply to the root logger, but then I'm guessing they'll catch the

dagster

logger output too? I guess a short-term help would be some more detail about how logs are set up -- when Dagster's logging config is applied, how the

dagster

logger is configured (with propagation, any other key details).

hi @Mark Fickett! The basic flow is that when the dagster run context is created (so this would be when the job is launched in a single process, or in each step's process if each step is being run in a separate process), the DagsterLogManager will be instantiated. You can see the creation code here: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/dagster/dagster/core/log_manager.py?L272:7

I'd like my logging handlers to receive the plain

LogRecord.msg

, but it seems like they're getting a pre-formatted version. For example, the

msg

is "_battery_data_job_local - 17f7577a-d149-4c4d-82fb-8fba5e810198 - data_pipe_graph_sasquatch_reactor.raw_data_graph_sasquatch_reactor._normalize_arbin8_task_sasquatch_reactor[ananke] -_ Normalizing..." where I would like it to just be "Normalizing..." (all the italicized stuff is added by Dagster). It looks like this is coming from

DagsterLogHandler._convert_record

editing the

msg

property in place: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/dagster/dagster/core/log_manager.py?L232 . And it happens to be handling the particular

LogRecord

before my handler gets to it. If I edit the source to skip that line then I get what I want. Any way to work around it? Or potentially something that could be changed?

hi @Mark Fickett! I believe the original message is preserved in

LogRecord.dagster_meta["orig_message"]

(definitely not the most convenient thing, I'll admit)

OK, I'm using a helper to pull the message out of there and put it back in LogRecord.msg for my handlers. I filed https://github.com/dagster-io/dagster/issues/7642 to request a more ideal solution, but this works well enough for now. Thanks for your help understanding what was going on!