https://dagster.io/ logo
Powered by
#dagster-support
Title
# dagster-support
w

Will Gunadi

03/22/2022, 2:59 PM
When I issue context.log_info("some string"), I can see in on Dagit, but where is it stored in the database (I use PostgreSQL backend)?
a

alex

03/22/2022, 3:01 PM
in the 
event_logs
table
w

Will Gunadi

03/22/2022, 3:01 PM
That's the first place I look, but I can't seem to find it. What event type would it be in?
a

alex

03/22/2022, 3:09 PM
null
, itll be in the 
event
col json payload
w

Will Gunadi

03/22/2022, 4:03 PM
The actual value that I'm searching for is stored in a file whose path is in that "event" field??? Why?
a

alex

03/22/2022, 4:15 PM
I don’t follow. 
context.log
should result in a row in the 
event_logs
db with a json serialized 
EventLogEntry
object in the 
event
column containing the contents of the logged string in the 
user_message
field.
w

Will Gunadi

03/23/2022, 9:56 PM
@alex this is the content of the "user_message" field in my case:
Copy code
"user_message": "Writing file at: /home/de1/dagster_f45/storage/b7066d95-83a8-496e-af7f-5d312cb17838/insights_refresh_time/last_insights_refresh"
And sure enough, if I follow that path and see the content of that file, the actual message I am looking for is in there.
Did I miss a setting that would put that message in the db instead of a file like that?
a

alex

03/23/2022, 9:59 PM
what does your 
@op
look like? It appears that this is the 
context.log.debug
from the 
fs_io_manager
resource which would correspond to returning a value from your op. If you explicitly logged the value yourself in the op i would expect it to to be in the db directly
w

Will Gunadi

03/23/2022, 10:09 PM
The line that produced that event is this:
Copy code
<http://context.log.info|context.log.info>(some_str_var)
a

alex

03/23/2022, 10:13 PM
do you then return 
some_str_var
? I think you are finding the log entry output by https://github.com/dagster-io/dagster/blame/master/python_modules/dagster/dagster/core/storage/fs_io_manager.py#L117
w

Will Gunadi

03/23/2022, 10:14 PM
Yes, I do return that same variable. So where can I find the event associated with the context.log.info, not the return?
I found it!
It's the previous status=null entry in the table.
So the question now is, how do I distinguish it from the row produced by the return value?
I notice that the event produced by the return value has level:10, and the one by the context.log.info has level:20. What does level indicate here? @alex
a

alex

03/23/2022, 10:20 PM
log.info vs log.debug
w

Will Gunadi

03/23/2022, 10:21 PM
Thank you, so now I can filter with level=20 to find all my log.info statements
a

alex

03/23/2022, 10:21 PM
what exactly is the overall goal here? would it be better to just explicitly write these values to your own table?
w

Will Gunadi

03/23/2022, 10:22 PM
Why would I need to create a separate table if I can query it right within the context of the value?
My goal is to store some data about the ETL run that I may want to query one day as a metric.
I know that I can do that using AssetMaterialization, but I'm exploring if I can also query log.info because I use those too.
And thanks to you, now I know how to query them. So Thank You! 🙂
👍 1
@alex Just want to make sure you get my Thanks! I appreciate you helping me.
👍 1
a

alex

03/24/2022, 2:10 PM
Why would I need to create a separate table if I can query it right within the context of the value?
it’s theoretically possible some future version may change how those logs are stored
2 Views
Powered by