Alexey Zavalskiy09/01/2022, 5:42 PM
In attachment example of .stderr file from job.
python_logs: python_log_level: DEBUG managed_python_loggers: - scrapy.core.engine - scrapy.utils.log
jamie09/01/2022, 6:30 PM
2022-09-01 17:28:34 [scrapy.utils.log] INFO: Scrapy 2.6.2 started (bot: scrapybot)
is the issue that the log you attached isn't showing up in dagit?
2022-09-01 17:28:34 [scrapy.core.engine] INFO: Spider opened
Alexey Zavalskiy09/02/2022, 6:11 AM
jamie09/13/2022, 2:57 PM
owen09/13/2022, 5:02 PM
in your dagster.yaml, does that end up capturing logs?
Alexey Zavalskiy09/15/2022, 9:51 AM
lead to capturing logs
owen09/15/2022, 8:20 PM
, which should be the actual python logger objects that Dagster is expecting to capture messages from. With your original config, I'd expect that to contain two loggers,
<Logger scrapy.core.engine (INFO)>, <Logger scrapy.utils.log (INFO)>
, which is something to watch out for (not sure if spaces get stripped out in the yaml config or not)
logging.getLogger("a.b.c") != logging.getLogger("a.b.c ")
Alexey Zavalskiy09/16/2022, 4:02 PM
from my op and get
, but still don’t have these logs in dagit if
[<Logger dagster.builtin (DEBUG)>, <Logger scrapy.core.engine (DEBUG)>, <Logger scrapy.utils.log (DEBUG)>]
logger isn’t set. Checked for some typos or spaces, didn’t help