https://dagster.io/ logo
#announcements
Title
# announcements
s

schrockn

08/06/2020, 9:43 PM
Hooks We have added hooks, currently marked as experimental, that enables you to execute cross-cutting success and failure handling policies on pipelines. This will enable you to, for example, send a slack message to a specific channel when any solid fails in a particular pipeline. You can also apply these on a per-solid basis. We imagine infrastructure teams providing common hooks for teams to apply common alerting policies to their pipelines. Example:
Copy code
@failure_hook(required_resource_keys={'slack'})
def slack_on_failure(context):
    message = 'Solid {} failed'.format(context.solid.name)
    context.resources.slack.send_message(message)

@slack_on_failure
@pipeline
def a_pipeline():
    b(a()) # will fire if either a or b fail

@pipeline
def selective_notif().
    b.with_hooks({slack_on_failure})(a()) # only fires when b fails
🔥 6
❤️ 2
blob 100 1
@Travis Cline hooks you were asking about are in
❤️ 2
dagsir 2
y

yuhan

08/06/2020, 9:59 PM
a

AXue

09/10/2020, 12:16 AM
This looks great! I’m very new to the dagster universe. Are hooks designed just for lightweight notification purposes? I’m looking for a way to catch failures of any solid within a pipeline, so that I could force execute a “catching” solid. Is there any tool that supports similar use cases? Thanks a lot! If I use hooks to do so, I would need to know the input values of the failed solid in the hook block, which I found very challenging.
y

yuhan

09/10/2020, 12:21 AM
hi @AXue! thanks for the feedback. hooks are designed for generic purposes — anything you would like to do at a per solid level. but its current state has limited exposure to the system information.
a

AXue

09/10/2020, 12:26 AM
Thanks a lot for the response! In this case, what would you recommend for catching errors in the pipeline, so that the run fails, but I can do something after it fails (gracefully)?
y

yuhan

09/10/2020, 12:37 AM
curious about what the catching solid would be doing? — does it have to be a solid inside the current pipeline? or is it going to be another pipeline (we are working a cross-dag capability now)
a

AXue

09/10/2020, 12:43 AM
The catching solid basically logs that this run has failed. Similarly, when the run starts (first solid of the pipeline), it logs that this run/job has started, and when the run succeeds (last solid of the pipeline), it logs that this run/job has completed.
y

yuhan

09/10/2020, 1:03 AM
i see! sounds like it’s not necessarily a solid, i.e. it doesn’t have to be part of the pipeline? for catching errors, we currently don’t expose step events through the success_hook and failure_hook. this is something we plan to enable. the event will have information like failure stack trace and event type to determine run start/finish/fail
👍 2