https://dagster.io/ logo
#random
Title
p

Peter Bower

09/01/2021, 11:17 PM
Hey there, does Dagster support sequencing Zeppelin notebooks in a similar fashion to its Jupyter support?
m

max

09/01/2021, 11:21 PM
not at present -- can you say a little more about your use case?
p

Peter Bower

09/01/2021, 11:23 PM
Essentially I've got an on-prem Spark cluster from the cloudera suite where Zeppelin notebooks are used for modelling and some big data transformations. I'm putting Dagster into the environment on Docker, and it would be handy to pipe the existing notebooks rather than converting them to scripts (at least in the first instances until refactoring), similar to how one can do so with Jupyter notebooks in the current dagster suite. Thanks
1
m

max

09/02/2021, 3:32 PM
our jupyter integration is built on top of https://papermill.readthedocs.io/en/latest/, which makes it straightforward to do all the templating and serialization that we use to communicate with the notebooks
depending how deep an integration you'd like with zeppelin, it could be more or less straightforward to set something up
happy to chat about paths forward here