Hello! Just getting started with dagster. If a pip...
# announcements
b
Hello! Just getting started with dagster. If a pipeline fails midway, is there support for rolling back the changes? For example, if we create a hive external table in step 1 and the query fails on step 2 due to some data error in that table, do we have to build routines to drop the table from step 1? And if so, what would be the best way to set that up in dagster?
a
Hey Brendan! There is no clever way to do this with dagster at this point, though better solutions to problems like this is something we definitely plan to add. In the short term one option is to use the python api to execute the pipeline (
execute_pipeline
) and then use the result object to decide what action to take on failure. The result object gives you access to all the information emitted during pipeline execution.
b
Thanks @alex! I’ll give that a shot
a
The other thing to leverage is the type system, you could create a custom type around newly created external hive tables and then on failure just walk through all the output events that returned an instance of that type and call rollback on them
s
I guess the pipelines will need cleanup of some external resources on success, too, so more “finally” than “catch” maybe?
a
cleanup of some external resources
you can use context manager style yield once functions for resources in dagster so that you have a way to do clean up
s
great, thank you!