Mark Fickett

03/02/2022, 9:48 PM
How can I capture logs from a subprocess into Dagster? I'm migrating an existing pipeline into Dagster, and it's using concurrent.futures which spawn subprocesses. Their logs aren't captured in Dagster, even though I followed to set
python_logs:  managed_python_loggers: - root
in my
. (I confirmed that a logger in my main process does get picked up within Dagit, and the logs from the subprocess do still show in up in my terminal where I started
.) I'm hoping to do an incremental migration where I leave some chunks using concurrent.futures for a bit, but if the answer is that I just need to get everything into Dagster, that's probably OK too.