https://dagster.io/ logo
Title
k

Kyle Wigley

03/14/2023, 5:17 PM
hey! I’m running on k8s and configured S3ComputeLogManager for storing compute logs. I’m able to see log files in s3, but I’m unable to view them in dagit ui. Anyone experience this before or have any tips for debugging? thanks!
p

prha

03/14/2023, 5:21 PM
Hey Kyle… What are you seeing in dagit? Is there an S3 path visible at the bottom of the compute log pane? Is there a download link in the top right corner?
k

Kyle Wigley

03/14/2023, 5:22 PM
just a “No log file available” message for stdout/stderr for all ops
let me try flipping on debug logs for dagit to see if I see anything
p

prha

03/14/2023, 5:34 PM
Yeah, I’m also curious if websockets are on/off for you.
The last thing I’d check is to see if there are empty local files stored in your configured local path on your dagit pod.
There’s been lingering reports of issues where the S3 download fails but leaves an empty local file on the dagit pod, and then tricks dagit into thinking there are no logs, but haven’t gotten to the bottom of it. Can you also share what version of dagster you’re running?
k

Kyle Wigley

03/14/2023, 5:40 PM
got it, happy to help get to the bottom.. running
1.1.18
at the moment
websockets are on (well, they aren’t disabled in UI)
p

prha

03/14/2023, 5:48 PM
What about local files on the dagit pod? Usually it’s something like
$DAGSTER_HOME/logs/<run_id>/compute_logs/<hash>.err
k

Kyle Wigley

03/14/2023, 5:53 PM
$DAGSTER_HOME/logs
is empty 🤔
p

prha

03/14/2023, 5:54 PM
hmm… not a k8s expert, but do you have perms to write to the FS?
under the hood, the dagit pod is downloading the s3 content to the local FS and then reading them off disk
(also might double check your
dagster.yaml
and see if you’ve configured a different local path)
k

Kyle Wigley

03/14/2023, 6:01 PM
got it! I’ll report back after digging a bit
thank you btw :dagster:
I am unable to explain why this is working for me today lol
🎉 1
p

prha

03/16/2023, 4:22 PM
confusing, but glad it’s working.