I m trying to add dagstermill to my pipeline and having a fe

Question

I m trying to add dagstermill to my pipeline and having a few problems The first 2 I was able to workaround but the last is going to a bug fix in dagstermill

Will Brown · Answer

The `Can t execute a dagstermill solid from a pipeline that wasn t instantiated using an ExecutionTargetHandle` error I m doing development of pipeline one task at a time and re executing using execute pipeline so i can iterate in debugger quickly I was able to work around this by building from execute handle This could be documented somewhere ```handle = ExecutionTargetHandle for pipeline python file rightsizedag py rightsize pipeline pipeline = handle build pipeline definition result = execute pipeline pipeline instance=instance environment dict=env run config=rc ```

Will Brown · Answer

The second issue was an exception thrown from papermill here in papermill execute py ``` Fetch the kernel name if it s not supplied kernel name = kernel name or nb metadata kernelspec name``` I was able to work around this by injecting a kernel name in the ipynb I created the notebook with dagstermill cli and it didnt have this It would be nice if kernel name could be specified as config to the solid edit the exception was a key error that kernelspec didnt exist

alex · Answer

ya 1 should definitely be documented better we are also exploring ways to improve <https dagster phacility com D2357>

Will Brown · Answer

The third issue I m stuck on is my config contains characters The parameters injected in to the notebook `context = dm dagstermill reconstitute pipeline context dm seven json loads ` This string doesn t have escaped so the whole cell fails with syntax error I tried manually editing the cell to escape all the and the notebook can run the cell and loads all my inputs

alex · Answer

cc < max>

alex · Answer

thanks for the reports we ll get this fixed

max · Answer

max · Answer

thanks for the kernelspec report too

max · Answer

max · Answer

< Will Brown> I m having trouble reproducing your string escaping issue

max · Answer

did this occur when running a pipeline using the python API or using config from YAML and dagit

Will Brown · Answer

```solids query vm resources config custom projections vm size = tostring properties hardwareProfile vmSize ``` I have this in my config ^ Those ` ` get serialized right in to the cell as is inside the ` dm seven json loads ` so the ` ` terminates the string and breaks the syntax of the cell

max · Answer

thanks much

Will Brown · Answer

I can t seem to find a clean way to access the notebook in a subsequent solid I want to follow up the notebook solid with a solid that uses nbconvert to create a report The path is yielded in a materialization only It would be great if there was an option to have the solid yield and output with the output notebook path or content

max · Answer

yep

max · Answer

i ll take a look at that tonight

Will Brown · Answer

gt 0 7 6 gt Fixed an issue when executing dagstermill solids with config that contained quote characters gt The Jupyter kernel to use may now be specified when creating dagster notebooks with the kernel flag slightly smiling face so fast

max · Answer

I think we have something to offer for the solid notebook output as well slightly smiling face

max · Answer

<https dagster phacility com D2431> will merge to master today basically this lets you specify an additional argument to `define dagstermill solid` that configures it to yield the notebook as an output

Will Brown · Answer

Oh dang i already implemented the functionality myself I went a simpler route deferring to the automatic materialization of outputs I have the same flag and add the extra output definition exactly like you did However i output the actual NotebookNode it s returned by papermill execute you dont have to actually read the notebook off disk I added NotebookNodeDagsterType In my downstream solid i can just take the incoming NotebookNode object and hand it to nbconvert

Will Brown · Answer

I can share the patch if you re interested

Will Brown · Answer

If nothing else there is a little optimization you can make

Will Brown · Answer

instead of using scrapbook read notebook you can just store the result of your papermill execute call it returns a NotebookNode and create a scrapbook models Notebook with that result as the arg It accepts a str path or a NotebookNode already in mem

max · Answer

yes would love to see the patch

Will Brown · Answer

This is against 0 7 6

Will Brown · Answer

heh i forgot i used the wrapped notebooknode exposed by the scrapbook notebook object but its identical to what execute returns the read from disk by scrapbook could be eliminated