We are trying to figure out what the best way of viewing some of our notebook outputs would be. Currently after the op is run, the notebook is saved to azure (we use papermill not dagstermill since these are spark jobs). It would be neat if we could somehow output the notebook as an event and have it render in dagit. Do you have any thoughts as to how I could start implementing something like this?
I see that there is already some way to view the notebook output when using dagstermill. However since we are running this job outside of dagster on spark cluster we use papermill directly. Is there a way to get the same UI treatment?