Hey guys, just curious if there is an example of a...
# announcements
Hey guys, just curious if there is an example of a notebook run via dagstermill grabbing its input parameters from the dag context (or some other dagster api), in memory, it looks like all the examples on github have the input coming from files or other such things. If there is a better channel to ask this on please let me know.
@max any input on this?
hm, have you looked at
in dagstermill.examples
i may have missed that bit of code, thanks for showing it, just slightly confused because for example the notebook
define the variables
inside the file, so it wasn't clear that it was actually using the input variables from the dag.
yes, we could use a better example for sure
but basically dagstermill supports the case where you might want to do some interactive execution with variables specified
those then get substituted with the upstream values when a notebook is executed as part of a pipeline
cool, awesome.
@max just so i understand that means you replace the very first assignment of the variable with the input from upstream in the dag, or woudl the local assignment immediately overwrite the input from upstream?
the cell tagged
is replaced at pipeline execution time, and any inputs are assigned there
so your development assigments should go in that cell
@max or anyone else, have you used dagstermill with inputs and outputs to and from the dagster context and been able to interactively modify the notebook content after the fact and rerun the DAG from that notebook? I tried doing something like that a while back and ran into issues. If that's possible then i'd like to give it another shot
what do you mean by rerun the DAG from the notebook
like rerun a subset of the pipeline starting with the notebook solid
yep, that's what i mean
i think that should work
you will need to modify the source notebook, not the output notebook
ah ok, i'm hoping to be able to modify the output notebook. Right now a use case I have is executing a default notebook, then going in to the executed notebook and doing custom modifications on an analysis-by-analysis basis. I don't want to have to change the notebook template to add custom modifications
huh, interesting
let me think about that
and then you would rerun the same pipeline, or a different pipeline?
it might be possible to have a configurable version of the dagstermill solid where you could point it dynamically at different notebooks using config
and so you could edit config to point it at the edited output notebook, e.g.
the same pipeline, but either starting from the notebook i changed, or rerunning the whole thing but hopefully making use of cached intermediate results
interesting idea with the configurable dagstermill solid...i'd have to think about that. might be more complexity than is worth it for me