Hey I am working with dagstermill and am trying to...
# announcements
Hey I am working with dagstermill and am trying to add a dagstermill solid to a pipeline but receive the error
"Can\'t execute a dagstermill solid from a pipeline that is not reconstructable."
I have read the API doc on the reconstructable class but I am not sure of its use case. Could anyone provide an example?
I am guessing you are executing the pipeline via python API doing
, and its when you pass in the pipeline there that it needs to be reconstructable, which you should be able to use the
function for
Hey Alex, thanks for the reply. Currently we use execute_pipeline_iterator and process each event from our pipeline individually. At the moment we cannot use execute_pipeline. Would it be possible to use the reconstructable function using execute_pipeline_iterator? We have a method that returns a PipelineDefinition, I have tried to pass this method to the reconstructable function and receive an error stating that reconstructable requires a function with a pipeline decorator but the api docs for reconstructable state that it can be a function that returns a PipelineDefinition or a function with the pipeline decorator. If you need anymore information let me know, thanks for the help.
Ya you should be able to do a function that returns one, can you share the error you are seeing
dagster.core.errors.DagsterInvariantViolationError: Reconstructable target should be a function or definition produced by a decorated function, got <class 'dagster.core.definitions.pipeline.PipelineDefinition'>.
The method I pass is called _make_pipeline(args) so I call reconstructable(make_pipeline(args)) it is quite possible the problem is on my end. Again thanks for the help._
ya you have to pass the function it self and not the results, and it has to be a function that takes no args and is defined at module scope
does that make sense? The idea is that we need to know how to recreate the pipeline in the other process
That's brilliant I think I can make that work. I'll let you know how it works out. Really appreciate the help.
👍 1
Hey Alex, I have been able to get a simple reconstructable pipeline working by setting the PipelineDefinition parameters inside our make_pipeline function. Would it be possible to set those parameters outside the make_pipeline function? EDIT: I figured it out, had a silly mistake in my code.