hello dagsters, how can i do the “python logging c...
# ask-community
a
hello dagsters, how can i do the “python logging capturing” described HERE with an in-process execution? i try executing a job using
.execute_in_process()
, but i get no captured python logs. when i start the job using
dagster job execute
it works.
🤖 1
j
Hi @Axel Bock do you have some code snippets you can share? any output you're getting that shows the log capture working with
dagster job execute
and not working with
execute_in_process
would also be helpful!
o
hi @Axel Bock, I believe what's happening is that by default, when you use .execute_in_process(), an in-memory dagster instance will be created to store the logs/events produced by that run. This dagster instance will not have any of the configurations set on your main dagster instance, and will disappear after the run completes. If you want the logs captured and stored in your default dags\ter instance (the one that's configured with that dagster.yaml file), you can do this with
.execute_in_process(instance=DagsterInstance.get())
. If you want the logs captured, but stored to a different, temporary dagster instance, you can do this with
.execute_in_process(instance=DagsterInstance.local_temp(overrides={"python_logs": <whatever config>})