Hey all! This should be a real quick question. I'm...
# announcements
a
Hey all! This should be a real quick question. I'm writing a "lookup" script where I am given some
SolidExecutionContext
object called
context
for some solid in composite pipeline
COMP
, made up of pipelines
A
,
B
, and
C
. If I call
context.pipeline_name
the property will return
COMP
. Let's say the context is from a solid that exists in
A
, is there an EASY way to trace the solid back to the originating pipeline? In this case, each solid in
COMP
is unique. Alternatively, if we know the pipeline's name in advance, is there a way to get all of the solids in that pipeline? That might be easier than the garbage I came up with.
a
pipeline_name
is on the
SolidExecutionContext
a
Yes, that's true; however, if I am running a pipeline composed of multiple pipelines, the pipeline_name will be returned as
COMP
. I wanted
pipeline_name
to return
A
a
Alternatively, if we know the pipeline’s name in advance, is there a way to get all of the solids in that pipeline?
if you have the a
PipelineDefinition
there is
all_solid_defs
oh i read composite pipeline as composite solid
how are you executing composite pipelines?
a
With regards to how we're executing composed pipelines, we have a sensor that's triggering it. Also that
solid_handle
doesn't seem to work for me either, I'm gonna take lunch and try out that
PipelineDefinition
solution that you mentioned. Thank you Alex!
s
Out of curiosity, what do you use this lookup for?
a
I can absolutely answer that. We have a sensor that picks up documents from a location on the file system then moves them to some "ready" folder. They only get moved from "ready" to "success" if the document goes through the entire pipeline. Since this pipeline is composed of three smaller pipelines, I had the idea to move the documents based on whether they fail a part of the pipeline. Essentially, we do not want to go through the dagster UI to see what passed the pipeline and what failed