https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
m

Mark Fickett

05/10/2022, 7:45 PM
Could the 'step skipped' logs be demoted to DEBUG level, similar to the started/succeeded messages? In console output with the level set to INFO, I'm getting lots of them. Like
dagster - INFO - Skipping step data_pipe_graph_...
. Similar, when there's an error, I get a flood of error logs for the downstream ops to the one that hit the error. Could those move to DEBUG? Like
dagster - ERROR - Dependencies for step list_test_ids_by_pipe_for_metadata failed: ['_validate_config']. Not executing.
y

yuhan

05/10/2022, 11:49 PM
I don’t think we have a super handy way to config this, but you can customize a logger to tune the log level per your need. Here’s an example: https://docs.dagster.io/concepts/logging/loggers#customizing-loggers
👀 1
m

Mark Fickett

05/11/2022, 12:50 AM
I did set the console log level following this example to
INFO
, by doing this:
Copy code
loggers:
 console:
  config:
   log_level: INFO
I guess I could create a custom logger, set its level to
INFO
, and add a filter to remove these specific messages. But what I was wondering was whether the Dagster team would consider changing the level of the "Skipping step" and "Dependencies for step ... failed" messages to DEBUG.
y

yuhan

05/11/2022, 1:34 AM
Ah I meant that you could potentially custom a logger that looks up certain message and “downgrade” the log level manually (as a workaround).
m

Mark Fickett

05/11/2022, 1:37 AM
Gocha, makes sense.
y

yuhan

05/11/2022, 1:39 AM
I could imagine making “Skipping step” a DEBUG could cause some confusion for some users, as DEBUG isn’t shown by default and when skipping is a surprise it could be misleading. As for the error message “Dependencies for step …“, I don’t have a strong opinion and am leaning towards DEBUG - technically it’s “not executing” which isn’t an error.
m

Mark Fickett

05/11/2022, 1:48 AM
Yeah, definitely don't want to make silent decisions that surprise people. My reason to argue for DEBUG would be that there's really clear UI treatment for skipped steps, and the console message seems like a backup message stream for 'wait, what exactly is happening' similar to start/succeeded/etc DEBUG messages. If I filter to INFO, these two messages (skipped and dependencies failed) actually seem to be the only two Dagster processing messages left, so they seem more like outliers than part of a clear description of execution. But I'm definitely favoring my own use case here. (:
Unfortunately defining my own @logger is not working well, because Dagster's log handler is re-logging into my handler instead of me getting the original
LogRecord
. So things like filename, (logger) name, lineno don't work -- they just point back to log_handler.py. https://github.com/dagster-io/dagster/issues/7890