This channel is a place where you can get the latest updates on using Dagster with Papermill and Noteable, ask questions, and discuss with other members of the community. Looking to get started using Dagster with Papermill or Noteable? • ✨ Check out our getting started with Noteable guide • 📚 Take a look at our API docs for `dagstermill` • 🦢 Take a look at Noteable’s API docs for `origami`

Hi everyone! I am wondering if anyone has used a notebook asset as an input to another SDA? A lot of our (no Dagster yet) pipeline is currently interspread across Jupyter notebooks that export a dataset to some file. Partly the reason for the notebooks is the "literate programming" angle since it enables the team to follow along how the analyses are conducted step-by-step. In spite of that, is using a notebook SDA to create another SDA discouraged?

Hi everyone! we presented the Dagster + Noteable integration last week at PyData! The presentation included a workshop that covers getting started in Dagster, adding a Jupyter notebook as an asset, and adding a Noteable notebook as an asset. Whether you are using Jupyter notebooks and

dagstermill

or Noteable notebooks and

papermill-origami

, the workshop should provide a great resource for getting started. I’m attaching the slides here, and you can download and go through the workshop yourself (the sample code is here). I’m currently working on transferring the workshop into a new dagstermill guide, so if you have any feedback or features you’d like to see covered in the guide, please don’t hesitate to let me know. PyData should post the full talk in the next week or so, so I’ll be posting that here once it gets uploaded. And finally, we’re going to be doing the full version of the talk again in December as a webinar! keep your eyes peeled for more info on that!

Looks like retry policies dont work with dagstermill assets

does

dagstermill

output the filled papermill notebook on failure anywhere?

New

dagstermill

features in this week’s release (1.1.5 core 0.17.5 libraries)! • notebooks run as assets with

define_dagstermill_asset

can now call

dagstermill.yield_event

in the notebook • new

save_notebook_on_failure

parameter to

define_dagstermill_asset

and

define_dagstermill_op

allows you to tell dagster to save a notebook if it fails. This will help you debug errors within the notebook. By default, the parameter is set to False, meaning the notebook will not be saved. If you set

save_notebook_on_failure=True

and the notebook fails, you will see a message in the Dagster logs telling you where the notebook was saved.

😛lanet-daggy: 😛lanet-daggy: 😛lanet-daggy: new

dagstermill

guide!! https://docs.dagster.io/integrations/dagstermill 😛lanet-daggy: 😛lanet-daggy: 😛lanet-daggy: Tutorial for getting started integrating Jupyter notebooks with the rest of your Dagster assets https://docs.dagster.io/integrations/dagstermill/using-notebooks-with-dagster Reference page for additional features https://docs.dagster.io/integrations/dagstermill/reference as always, let me know if you have any feedback or would like to see additional content in these guides!

Hello! Along with the folks at Noteable, we’ll be running a workshop on getting started with Dagster + Noteable this Wednesday. It will be a webinar version of the talk we co-presented at PyData NYC last month. We’ll walk through setting up your Dagster project with a Jupyter notebook along with a Noteable notebook, so regardless of how you write/host your notebooks there will be things to learn! sign up here https://dagster.io/events/noteable-dagster-dec-2022

Hello!

:tumbleweed:

archived the channel