Hi dagsteristas I m using dagstermill to execute a notebook

Question

Hi dagsteristas I m using dagstermill to execute a notebook and would like to implement some retry logic I m raising a `RetryRequested` but I think papermill is catching the exception and wrapping it leading to a general failure of the pipeline Based on the examples and docs I think just raising a `RetryRequested` is the correct approach but I have probably missed something Stacktrace ```dagster core errors DagsterExecutionStepExecutionError Error occurred while executing solid qa master old File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan execute plan py line 193 in dagster event sequence for step for step event in check generator step events File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan execute step py line 308 in core dagster event sequence for step for user event in check generator File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan execute step py line 63 in step output error checked user event sequence for user event in user event sequence File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan compute py line 138 in execute core compute for step output in yield compute results step context inputs compute fn File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan compute py line 111 in yield compute results for event in iterate with context File opt conda envs dagster pipeline lib python3 8 site packages dagster utils init py line 384 in iterate with context return File opt conda envs dagster pipeline lib python3 8 contextlib py line 131 in exit self gen throw type value traceback File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan utils py line 62 in solid execution error boundary raise error cls The above exception was caused by the following exception papermill exceptions PapermillExecutionError Exception encountered at In 13 RetryRequested Traceback most recent call last lt ipython input 13 cd6a6478a462 gt in lt module gt 20 pass 21 else gt 22 raise RetryRequested max retries=60 seconds to wait=10 60 23 24 if last metric date without LI = expected last metric date RetryRequested File opt conda envs dagster pipeline lib python3 8 site packages dagster core execution plan utils py line 42 in solid execution error boundary yield File opt conda envs dagster pipeline lib python3 8 site packages dagster utils init py line 382 in iterate with context next output = next iterator File opt conda envs dagster pipeline lib python3 8 site packages dagstermill solids py line 192 in t fn papermill execute notebook File opt conda envs dagster pipeline lib python3 8 site packages papermill execute py line 122 in execute notebook raise for execution errors nb output path File opt conda envs dagster pipeline lib python3 8 site packages papermill execute py line 234 in raise for execution errors raise error```

alex · Answer

what version are you on the error should communicate that you are supposed to use `dagstermill yield event` <https dagster phacility com D7406> for context

Maitiú Ó Ciaráin · Answer

`0 11 14` but that makes sense I was following the example in `python modules libraries dagstermill dagstermill examples notebooks raise retry ipynb` FYI

Maitiú Ó Ciaráin · Answer

Thanks

Maitiú Ó Ciaráin · Answer

To follow up on this using `yield event` does indeed lead to a retry being requested but the solid generated by dagstermill <https github com dagster io dagster blob master python modules libraries dagstermill dagstermill solids py L300|define dagstermill solid> doesn t allow for retries I think this could be enabled by adding a `retry policy Optional RetryPolicy ` kwarg to `define dagstermill solid ` and then pasisng that to the resulting `SolidDefinition`

alex · Answer

Hm you shouldn t need a retry policy if you are using yield event The policy just takes care of throwing the request for you Can you file an issue with some more details of exactly what you are doing and seeing

Maitiú Ó Ciaráin · Answer

I ran some tests on this over the weekend and something else was going on Thanks