wondering if there any plans to integrate Dagit with a Notebook front-end like e.g. JupyterLab? at Twitter, most of our user’s laptops don’t have access to interact with the services they need to experiment with and iterate on DS/ML pipelines - so we tend to do most Pipeline/DAG iteration in batch Compute jobs or more interactively in online Jupyter Notebooks (accessed w/ JupyterLab + IDE remoting). it looks like Dagit is already a webapp, so getting it running as a plugin on the notebook server (a jupyter “serverextension”) + an iframe-style JupyterLab extension could be trivial. this should easily extrapolate to other customers with common modes of Notebooks-as-a-DS/ML-Development-Environment (e.g. Google Cloud’s AI Platform) for drop-in Dagster dev support. here’s an example of Dask’s JupyterLab integration for inspiration:

Hey Kris thanks for the question.

yeah, we see it as both: 1) an executable node in the graph of shape

execute_notebook(notebook_in) -> notebook_out

2) a development mode for defining adhoc (and maybe soon, productionized) Pipelines (and sub-components) for iterative development the high level idea there is that if your Pipeline is encoded in a notebook (that is also parameterizable and schedulable, via an external orchestrator), that loading that notebook up to iterate on it allows for fluid movement from Experimentation <-> Production use cases. our Notebook environments also have full monorepo integration, so it’s also just as easy to load libraries up for iterative development all the way up to sending a Phabricator review and landing that. prior art for ref/context: TFX’s Iterative Notebook RFC: https://github.com/tensorflow/community/blob/master/rfcs/20190815-tfx-notebook.md Kubeflow’s Fairing project: https://www.kubeflow.org/docs/fairing/fairing-overview/ Kubeflow’s Kale project: https://medium.com/kubeflow/automating-jupyter-notebook-deployments-to-kubeflow-pipelines-with-kale-a4ede38bea1f

interesting. so there would potentially be multiple notebooks open. one for defining and running pipelines.

yep, that could happen for sure.