19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
hey! I’m running on k8s and configured S3ComputeLo...
# deployment-kubernetesk
Kyle Wigley
03/14/2023, 5:17 PMhey! I’m running on k8s and configured S3ComputeLogManager for storing compute logs. I’m able to see log files in s3, but I’m unable to view them in dagit ui. Anyone experience this before or have any tips for debugging? thanks!
p
prha
03/14/2023, 5:21 PMHey Kyle… What are you seeing in dagit? Is there an S3 path visible at the bottom of the compute log pane? Is there a download link in the top right corner?
k
Kyle Wigley
03/14/2023, 5:22 PMjust a “No log file available” message for stdout/stderr for all ops
Kyle Wigley
03/14/2023, 5:24 PMlet me try flipping on debug logs for dagit to see if I see anything
p
prha
03/14/2023, 5:34 PMYeah, I’m also curious if websockets are on/off for you.
prha
03/14/2023, 5:35 PMThe last thing I’d check is to see if there are empty local files stored in your configured local path on your dagit pod.
prha
03/14/2023, 5:38 PMThere’s been lingering reports of issues where the S3 download fails but leaves an empty local file on the dagit pod, and then tricks dagit into thinking there are no logs, but haven’t gotten to the bottom of it. Can you also share what version of dagster you’re running?
k
Kyle Wigley
03/14/2023, 5:40 PMgot it, happy to help get to the bottom.. running
1.1.18Kyle Wigley
03/14/2023, 5:40 PMwebsockets are on (well, they aren’t disabled in UI)
p
prha
03/14/2023, 5:48 PMWhat about local files on the dagit pod? Usually it’s something like
$DAGSTER_HOME/logs/<run_id>/compute_logs/<hash>.errk
Kyle Wigley
03/14/2023, 5:53 PM$DAGSTER_HOME/logsp
prha
03/14/2023, 5:54 PMhmm… not a k8s expert, but do you have perms to write to the FS?
prha
03/14/2023, 5:55 PMunder the hood, the dagit pod is downloading the s3 content to the local FS and then reading them off disk
prha
03/14/2023, 5:56 PM(also might double check your
dagster.yamlk
Kyle Wigley
03/14/2023, 6:01 PMgot it! I’ll report back after digging a bit
Kyle Wigley
03/14/2023, 6:02 PMthank you btw D
Kyle Wigley
03/16/2023, 3:42 PMI am unable to explain why this is working for me today lol
🎉 1
p
prha
03/16/2023, 4:22 PMconfusing, but glad it’s working.
7 Views