One of my dagster jobs failed and the logs in the job pod gi

Question

One of my dagster jobs failed and the logs in the job pod give me a traceback error I d like to see the same traceback in the Dagit UI but I can t seem to find it I have Debug filter enabled I only see that my job failed and a message that logs are being retrieved Is it possible to see the pod logs in the dagit ui

cat · Answer

hey Michiel wave when you say my dagster job fails did you mean that inspecting the K8s Job shows the K8s Job has status failed or that dagster emits a step event saying the dagster step failed

cat · Answer

In the former case we will fetch all the logs and surface the raw logs in the UI sometimes this is not possible for example if the job is deleted

cat · Answer

in the latter we just fetch the logs in order for the executor to keep traversing the execution graph We do not surface the raw logs in the UI but do surface the parsed dagster events in the UI

Michiel Ghyselinck · Answer

Hi Cat this is the traceback that I get in the logs of the K8s job pod ```Traceback most recent call last File usr local bin dagster line 8 in lt module gt sys exit main File usr local lib python3 7 site packages dagster cli init py line 40 in main cli obj= pylint disable=E1123 File usr local lib python3 7 site packages click core py line 829 in call return self main args kwargs File usr local lib python3 7 site packages click core py line 782 in main rv = self invoke ctx File usr local lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 7 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File usr local lib python3 7 site packages click core py line 610 in invoke return callback args kwargs File usr local lib python3 7 site packages dagster cli api py line 427 in execute step with structured logs command execution plan pipeline run instance run config=args run config retries=retries File usr local lib python3 7 site packages dagster core execution api py line 732 in iter execution plan=self execution plan pipeline context=self pipeline context File usr local lib python3 7 site packages dagster core execution plan execute plan py line 63 in inner plan execution iterator assert missing inputs optional uncovered inputs execution plan step key File usr local lib python3 7 site packages dagster core execution plan execute plan py line 154 in assert missing inputs optional output name=nonoptionals 0 output name dagster core errors DagsterStepOutputNotFoundError When executing test kafka compute discovered required outputs missing from previous step StepOutputHandle step key= initialising compute output name= brokers ```

Michiel Ghyselinck · Answer

I think the cause of this error is that I haven t setup intermediate storage so it is logical that it cannot find the output of another job because that is another file system But I thought I would have been able to see this this traceback in the UI but that is not the case Does this have to do with the fact that I m using K8S Celery execution but with filesystem storage enabled

cat · Answer

yes would recommend using s3 or similar for re execution cases ```storage s3 config s3 bucket bucket s3 prefix prefix ```

cat · Answer

when you click on the view raw step output on the step start event for that step do you see the error there

cat · Answer

filed <https github com dagster io dagster issues 3133> for tracking