Hey guys just curious if there is an example of a notebook r

Question

Hey guys just curious if there is an example of a notebook run via dagstermill grabbing its input parameters from the dag context or some other dagster api in memory it looks like all the examples on github have the input coming from files or other such things If there is a better channel to ask this on please let me know

Leor · Answer

< max> any input on this

max · Answer

hm have you looked at `test notebook dag` in dagstermill examples

Ryan Lewis · Answer

i may have missed that bit of code thanks for showing it just slightly confused because for example the notebook `add two numbers ipynb` define the variables `a` and `b` inside the file so it wasn t clear that it was actually using the input variables from the dag

max · Answer

yes we could use a better example for sure

max · Answer

but basically dagstermill supports the case where you might want to do some interactive execution with variables specified

max · Answer

those then get substituted with the upstream values when a notebook is executed as part of a pipeline

Ryan Lewis · Answer

cool awesome

Ryan Lewis · Answer

< max> just so i understand that means you replace the very first assignment of the variable with the input from upstream in the dag or woudl the local assignment immediately overwrite the input from upstream

max · Answer

the cell tagged `parameters` is replaced at pipeline execution time and any inputs are assigned there

max · Answer

so your development assigments should go in that cell

Ben Lindsay · Answer

< max> or anyone else have you used dagstermill with inputs and outputs to and from the dagster context and been able to interactively modify the notebook content after the fact and rerun the DAG from that notebook I tried doing something like that a while back and ran into issues If that s possible then i d like to give it another shot

max · Answer

what do you mean by rerun the DAG from the notebook

max · Answer

like rerun a subset of the pipeline starting with the notebook solid

Ben Lindsay · Answer

yep that s what i mean

max · Answer

i think that should work

max · Answer

you will need to modify the source notebook not the output notebook

Ben Lindsay · Answer

ah ok i m hoping to be able to modify the output notebook Right now a use case I have is executing a default notebook then going in to the executed notebook and doing custom modifications on an analysis by analysis basis I don t want to have to change the notebook template to add custom modifications

max · Answer

huh interesting

max · Answer

let me think about that

max · Answer

and then you would rerun the same pipeline or a different pipeline

max · Answer

it might be possible to have a configurable version of the dagstermill solid where you could point it dynamically at different notebooks using config

max · Answer

and so you could edit config to point it at the edited output notebook e g

Ben Lindsay · Answer

the same pipeline but either starting from the notebook i changed or rerunning the whole thing but hopefully making use of cached intermediate results

Ben Lindsay · Answer

interesting idea with the configurable dagstermill solid i d have to think about that might be more complexity than is worth it for me

max · Answer

yeah