Hi All We are trying to use failure hook applied on job leve

Question

Hi All We are trying to use failure hook applied on job level to get failure messages for the ops to go to our slack channel We have added this failure hook to a job that has multiple ops in it One of the ops failed with this following message but it didn t invoke the failure hook Can you please help us understand why the hook wasn t even called Note For other ops failure we saw hook invokations in the dagster logging console after the op was executed But for this particular op there were no log events showing hook was called or skipped and the job went on to execute the next op Eg `Skipped the execution of hook slack message on failure It did not meet its triggering condition during the execution of xxx `

claire · Answer

Hi Gayathiri would you mind sharing a code snippet containing your job so I can reproduce

claire · Answer

As well as any steps taken to reach this error

Gayathiri · Answer

Logs for the job from Dagit are ```read pg predicti d high effort STEP START Started execution of step read pg prediction std high effort 17 02 32 489 read pg predicti d high effort LOADED INPUT Loaded input tablename using input manager io manager from output result of step get pg tables 17 02 32 583 read pg predicti d high effort STEP INPUT Got input tablename of type String Type check passed 17 02 32 606 read pg std claim actions STEP FAILURE kubernetes client exceptions ApiException 404 Reason Not Found HTTP response headers HTTPHeaderDict Audit Id 901431a7 45f3 409b 98f7 f6c0283000ba Cache Control no cache private Content Type application json X Kubernetes Pf Flowschema Uid f2f5659b e562 426b 81ce d474cea20292 X Kubernetes Pf Prioritylevel Uid 3b438278 a6eb 4a1a 918a 842f3525d6bb Date Tue 15 Nov 2022 22 02 33 GMT Content Length 284 HTTP response body kind Status apiVersion v1 metadata status Failure message jobs batch dagster step 24a9f39cdeadc24dda07fa68c8aabcf9 1 not found reason NotFound details name dagster step 24a9f39cdeadc24dda07fa68c8aabcf9 1 group batch kind jobs code 404 Stack Trace File usr local lib python3 9 site packages dagster core executor step delegating step delegating executor py line 239 in execute health check result = self step handler check step health File usr local lib python3 9 site packages dagster k8s executor py line 244 in check step health job = self batch api read namespaced job File usr local lib python3 9 site packages kubernetes client api batch v1 api py line 2657 in read namespaced job return self read namespaced job with http info name namespace kwargs noqa E501 File usr local lib python3 9 site packages kubernetes client api batch v1 api py line 2744 in read namespaced job with http info return self api client call api File usr local lib python3 9 site packages kubernetes client api client py line 348 in call api return self call api resource path method File usr local lib python3 9 site packages kubernetes client api client py line 180 in call api response data = self request File usr local lib python3 9 site packages kubernetes client api client py line 373 in request return self rest client GET url File usr local lib python3 9 site packages kubernetes client rest py line 241 in GET return self request GET url File usr local lib python3 9 site packages kubernetes client rest py line 235 in request raise ApiException http resp=r 17 02 33 478 View full message read pg predicti d high effort STEP OUTPUT Yielded output df of type DataFrame Type check passed 17 02 33 675 read pg predicti d high effort HANDLED OUTPUT Handled output df using IO manager io manager 17 02 33 909 read pg predicti d high effort STEP OUTPUT Yielded output schema of type DataFrame Type check passed 17 02 33 926 read pg predicti d high effort HANDLED OUTPUT Handled output schema using IO manager io manager 17 02 34 157 read pg predicti d high effort STEP SUCCESS Finished execution of step read pg prediction std high effort in 1 54s 17 02 34 174 read pg predicti d high effort HOOK SKIPPED Skipped the execution of hook slack message on failure It did not meet its triggering condition during the execution of read pg 17 02 34 192 read pg predicti d high effort HOOK COMPLETED Finished the execution of hook slack message on success triggered for read pg 17 02 34 209 RUN FAILURE Execution of run for app to cdf failed An exception was thrown during execution dagster check CheckError Invariant failed Description Attempted to mark step read pg std claim actions as complete that was not known to be in flight Stack Trace File usr local lib python3 9 site packages dagster core execution api py line 822 in pipeline execution iterator for event in pipeline context executor execute pipeline context execution plan File usr local lib python3 9 site packages dagster core executor step delegating step delegating executor py line 220 in execute active execution handle event dagster event File usr local lib python3 9 site packages dagster core execution plan active py line 400 in handle event self mark failed step key File usr local lib python3 9 site packages dagster core execution plan active py line 342 in mark failed self mark complete step key File usr local lib python3 9 site packages dagster core execution plan active py line 387 in mark complete check invariant File usr local lib python3 9 site packages dagster check init py line 1455 in invariant raise CheckError f Invariant failed Description desc 17 02 34 530 View full message ENGINE EVENT Process for run exited pid 1 17 02 34 58```

Gayathiri · Answer

We see this behavior only for jobs that have retry policy set

Gayathiri · Answer

```Job code dagster op required resource keys= global values retry policy=dagster RetryPolicy def load bq context tablename str df pd DataFrame col df pd DataFrame gt pd DataFrame Write postgres data to bigquery Massages the data based on the pg column info so that it will fit into bq f row count df shape 0 col df global values = context resources global valuesåç df = transform df df col df df to gbq å destination table=f global values bq dataset app tablename project id=global values gcp project id chunksize=512 if exists= replace table schema= get bq schema col df return df dagster op required resource keys= global values out= df dagster Out schema dagster Out retry policy=dagster RetryPolicy def read pg context tablename str gt Tuple pd DataFrame pd DataFrame Read data from postgres Returns df the rows in the table col df metainformation about the columns in the table global values = context resources global values engine = common get postgres engine global values df = pd read sql f SELECT FROM public tablename engine col df = pd read sql col query format tablename=tablename engine return df col df dagster job name= app name a executor def=autoconfig get executor config=k8s executor config mapping resource defs= global values global values resource autoconfig get resource defs slack slack resource configured token get slack token hooks= slack message on failure def app to cdf def read and load tablename str df col df = read pg tablename df = load bq tablename df col df return df```

Gayathiri · Answer

We expect the hook to fail when there is a step failure event and run failure event But the failure hook is skipped and success hook is triggered

johann · Answer

What version of dagster are you running

johann · Answer

We had a recent fix for the kubernetes 404 on step retries in 1 0 17 0 16 17 The behavior of the hooks is unexpected I don t see `slack message on success` in your code snippet could you share where you add it

Gayathiri · Answer

Hi We are running dagster 1 0 16

Gayathiri · Answer

The slack message on success was added to the same job that I shared earlier with the failure hook I removed it later so that we could debug the behavior of the failure hook ``` dagster job name= app name a executor def=autoconfig get executor config=k8s executor config mapping resource defs= global values global values resource autoconfig get resource defs slack slack resource configured token get slack token hooks= slack message on failure slack message on success def app to cdf def read and load tablename str df col df = read pg tablename df = load bq tablename df col df return df```

Gayathiri · Answer

< johann> Could you please give us the issue number for this fix

johann · Answer

1 0 17 for the kubernetes fix

Gayathiri · Answer

Thanks I was looking for documentation or release notes so that we could check if our 404 are due to the same issue as the documentation

johann · Answer

Gotcha let me find it But it looks like the same scenario

Gayathiri · Answer

Thank you so much Also any inputs on the hook behavior with the retries that we reported would be much appreciated

johann · Answer

claire · Answer

< Gayathiri> were you able to see if the issue still exists on 1 0 17

Gayathiri · Answer

Hi We upgraded our dev to 1 0 17 and we dont see the 404s anymore Will confirm back if this upgrade fixed the inconsistent failure hook behavior that we were experiencing Thank you

Gayathiri · Answer

We dont see the inconsistent hook behaviors after upgrading to 1 0 17 Thank you

David Tong · Answer

< johann> would you know if 1 0 17 would also fix the following similar error k8s 401 instead of 404 ```kubernetes client exceptions ApiException 401 Reason Unauthorized HTTP response headers HTTPHeaderDict Audit Id 3e5fc9cb e799 4adf 8665 6f78f0e8b361 Cache Control no cache private Content Type application json Date Tue 29 Nov 2022 23 30 38 GMT Content Length 129 HTTP response body kind Status apiVersion v1 metadata status Failure message Unauthorized reason Unauthorized code 401 File opt conda envs akuna lib python3 8 site packages dagster core executor step delegating step delegating executor py line 243 in execute health check result = self step handler check step health File opt conda envs akuna lib python3 8 site packages dagster k8s executor py line 248 in check step health job = self batch api read namespaced job File opt conda envs akuna lib python3 8 site packages kubernetes client api batch v1 api py line 1257 in read namespaced job return self read namespaced job with http info name namespace kwargs noqa E501 File opt conda envs akuna lib python3 8 site packages kubernetes client api batch v1 api py line 1352 in read namespaced job with http info return self api client call api File opt conda envs akuna lib python3 8 site packages kubernetes client api client py line 348 in call api return self call api resource path method File opt conda envs akuna lib python3 8 site packages kubernetes client api client py line 180 in call api response data = self request File opt conda envs akuna lib python3 8 site packages kubernetes client api client py line 373 in request return self rest client GET url File opt conda envs akuna lib python3 8 site packages kubernetes client rest py line 239 in GET return self request GET url File opt conda envs akuna lib python3 8 site packages kubernetes client rest py line 233 in request raise ApiException http resp=r ``` this error was raised on only one pod of several in a k8s retry job where the other analogous pods were able to complete without issue