Will Brown04/02/2020, 4:28 PM
error. I'm doing development of pipeline one task at a time and re-executing using execute_pipeline so i can iterate in debugger quickly. I was able to work around this by building from execute handle. This could be documented somewhere.
Can't execute a dagstermill solid from a pipeline that wasn't instantiated using an ExecutionTargetHandle
handle = ExecutionTargetHandle.for_pipeline_python_file('rightsizedag.py', 'rightsize_pipeline') pipeline = handle.build_pipeline_definition() result = execute_pipeline(pipeline, instance=instance, environment_dict=env, run_config=rc)
I was able to work around this by injecting a kernel name in the ipynb. I created the notebook with dagstermill cli and it didnt have this. It would be nice if kernel name could be specified as config to the solid. edit: the exception was a key error that 'kernelspec' didnt exist.
# Fetch the kernel name if it's not supplied kernel_name = kernel_name or nb.metadata.kernelspec.name
alex04/02/2020, 4:33 PM
Will Brown04/02/2020, 4:35 PM
This string doesn't have ' escaped so the whole cell fails with syntax error. I tried manually editing the cell to escape all the ' and the notebook can run the cell and loads all my inputs.
context = __dm_dagstermill._reconstitute_pipeline_context(**__dm_seven.json.loads('...
alex04/02/2020, 4:38 PM
Will Brown04/02/2020, 9:41 PM
I have this in my config ^. Those
solids: query_vm_resources: config: custom_projections: "vm_size = tostring(properties['hardwareProfile']['vmSize'])"
get serialized right in to the cell as is. inside the
terminates the string and breaks the syntax of the cell.
max04/02/2020, 9:43 PM
Will Brown04/03/2020, 12:14 AM
max04/03/2020, 12:52 AM
Will Brown04/03/2020, 11:59 AM
* Fixed an issue when executing dagstermill solids with config that contained quote characters.
* The Jupyter kernel to use may now be specified when creating dagster notebooks with the --kernel flag.🙂 so fast!
max04/03/2020, 7:42 PM
that configures it to yield the notebook as an output
Will Brown04/03/2020, 7:53 PM
max04/03/2020, 8:17 PM