Hi everyone! We use Dagster (v1.2.3) to run parser...
# ask-community
v
Hi everyone! We use Dagster (v1.2.3) to run parsers in pipelines. I need to display Scrapy logs in Dagit. For this I use Capturing Python Logs. In
dagster.yaml
I added:
Copy code
python_logs:
  managed_python_loggers:
    - scrapy
    - urllib3
    - my_logger
But it doesn't work. More precisely, it is the logs from Scrapy that do not work, the rest of the loggers are captured (for example,
my_logger, urllib3 etc.
) What could be the reason?
c
without knowing much about scrapy, wondering if they do some sort of custom stuff with their logger that might cause incompatibilities with what we do.
v
Ok, thanks! I'll how Scrapy works.