https://dagster.io/ logo
r

Richard Brady

07/03/2020, 11:23 AM
Hey I am working with dagstermill and am trying to add a dagstermill solid to a pipeline but receive the error
"Can\'t execute a dagstermill solid from a pipeline that is not reconstructable."
I have read the API doc on the reconstructable class but I am not sure of its use case. Could anyone provide an example?
a

alex

07/06/2020, 3:05 PM
I am guessing you are executing the pipeline via python API doing
execute_pipeline
, and its when you pass in the pipeline there that it needs to be reconstructable, which you should be able to use the
reconstructable
function for
r

Richard Brady

07/10/2020, 2:10 PM
Hey Alex, thanks for the reply. Currently we use execute_pipeline_iterator and process each event from our pipeline individually. At the moment we cannot use execute_pipeline. Would it be possible to use the reconstructable function using execute_pipeline_iterator? We have a method that returns a PipelineDefinition, I have tried to pass this method to the reconstructable function and receive an error stating that reconstructable requires a function with a pipeline decorator but the api docs for reconstructable state that it can be a function that returns a PipelineDefinition or a function with the pipeline decorator. If you need anymore information let me know, thanks for the help.
a

alex

07/10/2020, 2:56 PM
Ya you should be able to do a function that returns one, can you share the error you are seeing
r

Richard Brady

07/10/2020, 9:57 PM
dagster.core.errors.DagsterInvariantViolationError: Reconstructable target should be a function or definition produced by a decorated function, got <class 'dagster.core.definitions.pipeline.PipelineDefinition'>.
The method I pass is called _make_pipeline(args) so I call reconstructable(make_pipeline(args)) it is quite possible the problem is on my end. Again thanks for the help._
a

alex

07/10/2020, 10:05 PM
ya you have to pass the function it self and not the results, and it has to be a function that takes no args and is defined at module scope
reconstructable(make_pipeline_fixed)
does that make sense? The idea is that we need to know how to recreate the pipeline in the other process
r

Richard Brady

07/10/2020, 10:16 PM
That's brilliant I think I can make that work. I'll let you know how it works out. Really appreciate the help.
👍 1
Hey Alex, I have been able to get a simple reconstructable pipeline working by setting the PipelineDefinition parameters inside our make_pipeline function. Would it be possible to set those parameters outside the make_pipeline function? EDIT: I figured it out, had a silly mistake in my code.