https://dagster.io/ logo
Join Slack
Powered by
Hello, I am running into a problem with failure ho...
# announcements
r
Hello, I am running into a problem with failure hooks and I don't know how to solve it. Basically, when I run a test code with success_hook and failure_hook, the success one works fine, it posts into Slack when it runs, but the failure_hook doesn't, the code raises the error but it does nothing. What could it be?
This is the code:
Copy code
from dagster import solid, execute_pipeline, ModeDefinition, pipeline, success_hook, failure_hook
from dagster_slack import slack_resource


@success_hook(required_resource_keys={'slack'})
def msg_success_slack(context):
    message = "Solid {} finished successfully".format(context.solid.name)
    context.resources.slack.chat.post_message(channel='#geral', text=message)

@failure_hook(required_resource_keys={'slack'})
def msg_failure_slack(context):
    print("blarg")
    message = "Solid {} failed".format(context.solid.name)
    context.resources.slack.chat.post_message(channel="#geral", text=message)

@solid
def a(_):
    return 1337

@solid
def b(_):
    raise ValueError("foo")

#@msg_failure_slack
@pipeline(mode_defs=[ModeDefinition(resource_defs={'slack': slack_resource})])
def notify():
    a.with_hooks({msg_failure_slack,msg_success_slack})()
    b.with_hooks({msg_failure_slack,msg_success_slack})()
And this is the error that I get: 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - ENGINE_EVENT - Starting initialization of resources [slack]. 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - ENGINE_EVENT - Finished initialization of resources [slack]. 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - PIPELINE_START - Started execution of pipeline "notify". 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - ENGINE_EVENT - Executing steps in process (pid: 7890) 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - a.compute - STEP_START - Started execution of step "a.compute". 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - a.compute - STEP_OUTPUT - Yielded output "result" of type "Any". (Type check passed). 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - a.compute - STEP_SUCCESS - Finished execution of step "a.compute" in 7.61ms. 2020-09-29 115754 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - a.compute - HOOK_SKIPPED - Skipped the execution of hook "msg_failure_slack". It did not meet its triggering condition during the execution of solid "a". 2020-09-29 115755 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - a.compute - HOOK_COMPLETED - Finished the execution of hook "msg_success_slack" triggered for solid "a". 2020-09-29 115755 - dagster - DEBUG - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - b.compute - STEP_START - Started execution of step "b.compute". 2020-09-29 115755 - dagster - ERROR - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - b.compute - STEP_FAILURE - Execution of step "b.compute" failed. ValueError: foo File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/errors.py", line 181, in user_code_error_boundary yield File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 424, in _user_event_sequence_for_step_compute_fn for event in gen: File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 100, in _execute_core_compute for step_output in _yield_compute_results(compute_context, inputs, compute_fn): File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 71, in _yield_compute_results for event in user_event_sequence: File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/definitions/decorators/solid.py", line 227, in compute result = fn(context, **kwargs) File "/opt/dagster/app/zarpo/common/hooks.py", line 23, in b raise ValueError("foo") 2020-09-29 115755 - dagster - ERROR - notify - a70989dc-1db6-4231-a470-b044d7edb348 - 7890 - PIPELINE_FAILURE - Execution of pipeline "notify" failed. Traceback (most recent call last): File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 213, in _dagster_event_sequence_for_step for step_event in check.generator(step_events): File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 289, in core_dagster_event_sequence_for_step for user_event in check.generator( File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 55, in _step_output_error_checked_user_event_sequence for user_event in user_event_sequence: File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 425, in _user_event_sequence_for_step_compute_fn yield event File "/usr/lib/python3.8/contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/errors.py", line 191, in user_code_error_boundary raise_from( File "/opt/dagster/venv/lib/python3.8/site-packages/future/utils/__init__.py", line 403, in raise_from exec(execstr, myglobals, mylocals) File "<string>", line 1, in <module> dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred during the execution of step: step key: "b.compute" solid invocation: "b" solid definition: "b" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/dagster/app/zarpo/common/hooks.py", line 32, in <module> execute_pipeline( File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/api.py", line 311, in execute_pipeline return _logged_execute_pipeline( File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/telemetry.py", line 89, in wrap result = f(*args, **kwargs) File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/api.py", line 362, in _logged_execute_pipeline return execute_run(pipeline, pipeline_run, instance, raise_on_error=raise_on_error) File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/api.py", line 161, in execute_run event_list = list(_execute_run_iterable) File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/api.py", line 722, in iter for event in self.iterator( File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/api.py", line 675, in _pipeline_execution_iterator for event in _core_execution_iterator( File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/api.py", line 642, in _core_execution_iterator for event in pipeline_context.executor.execute(pipeline_context, execution_plan): File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/executor/in_process.py", line 36, in execute for event in inner_plan_execution_iterator(pipeline_context, execution_plan): File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 76, in inner_plan_execution_iterator for step_event in check.generator( File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 275, in _dagster_event_sequence_for_step raise dagster_user_error.user_exception File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/errors.py", line 181, in user_code_error_boundary yield File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 424, in _user_event_sequence_for_step_compute_fn for event in gen: File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 100, in _execute_core_compute for step_output in _yield_compute_results(compute_context, inputs, compute_fn): File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 71, in _yield_compute_results for event in user_event_sequence: File "/opt/dagster/venv/lib/python3.8/site-packages/dagster/core/definitions/decorators/solid.py", line 227, in compute result = fn(context, **kwargs) File "/opt/dagster/app/zarpo/common/hooks.py", line 23, in b raise ValueError("foo") ValueError: foo Process finished with exit code 1
I'm running the code with:
Copy code
execute_pipeline(
    notify, {'resources': {'slack': {'config': {'token':}}}}
)
y
Hi @Rodrigo Pasquale this seems to be a limitation of the failure hook. The hook will be triggered when a step finishes. In this case, because the step fails and raises an error which fails the entire pipeline, the failure_hook won’t be executed properly — because the entire pipeline has failed.
The workaround is to add an param in execure_pipeline to not fail the entire pipeline
Copy code
execute_pipeline(
    notify, {'resources': {'slack': {'config': {'token':}}}}, raise_on_error=False
)
a
Hi @yuhan, I'm working with Rodrigo on this. Is this behaviour limited to runs started by calling 
execute_pipeline
?
Also, thanks for your help! This one had us scratching our heads hahaha
r
Thanks a lot @yuhan!
y
@Auster Cid this behavior limits to runs where the pipeline gets stopped and fails at any step failure. When a pipeline fails, the hook triggering machinery won’t run either. If you run it through Dagit, it should wrap the step failure error and continue the run until all other steps finish. So the hook can run properly in that case.
👍 1
raise_on_error is a short-term workaround. It doesn’t seem to be intuitive or natural to ask users to add it every time. So we will figure out a better way to deal with this behavior soon
a
For context, we default 
raise_on_error
in the python API 
execute_pipeline
under the assumption that most call sites are for test and having the exception raised directly is the desired default behavior.
17 Views
Open in Slack
PreviousNext
Powered by