The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hey guys, just curious if there is an example of a notebook run via dagstermill grabbing its input parameters from the dag context (or some other dagster api), in memory, it looks like all the examples on github have the input coming from files or other such things. If there is a better channel to ask this on please let me know.

hm, have you looked at `test_notebook_dag` in dagstermill.examples

i may have missed that bit of code, thanks for showing it, just slightly confused because for example the notebook `add_two_numbers.ipynb` define the variables `a` and `b`  inside the file, so it wasn't clear that it was actually using the input variables from the dag.

yes, we could use a better example for sure

but basically dagstermill supports the case where you might want to do some interactive execution with variables specified

those then get substituted with the upstream values when a notebook is executed as part of a pipeline

<@UDJ0NL1LY> just so i understand that means you replace the very first assignment of the variable with the input from upstream in the dag, or woudl the local assignment immediately overwrite the input from upstream?

the cell tagged `parameters` is replaced at pipeline execution time, and any inputs are assigned there

so your development assigments should go in that cell

<@UDJ0NL1LY> or anyone else, have you used dagstermill with inputs and outputs to and from the dagster context *and* been able to interactively modify the notebook content after the fact and rerun the DAG from that notebook? I tried doing something like that a while back and ran into issues. If that's possible then i'd like to give it another shot

what do you mean by rerun the DAG from the notebook

like rerun a subset of the pipeline starting with the notebook solid

you will need to modify the source notebook, not the output notebook

ah ok, i'm hoping to be able to modify the output notebook. Right now a use case I have is executing a default notebook, then going in to the executed notebook and doing custom modifications on an analysis-by-analysis basis. I don't want to have to change the notebook template to add custom modifications

and then you would rerun the same pipeline, or a different pipeline?

it might be possible to have a configurable version of the dagstermill solid where you could point it dynamically at different notebooks using config

and so you could edit config to point it at the edited output notebook, e.g.

the same pipeline, but either starting from the notebook i changed, or rerunning the whole thing but hopefully making use of cached intermediate results

interesting idea with the configurable dagstermill solid...i'd have to think about that. might be more complexity than is worth it for me