Hello, I'm trying to run a specific step of a pipe...
# announcements
b
Hello, I'm trying to run a specific step of a pipeline using
execute_pipeline
with
step_keys_to_execute
defined in the
RunConfig
however, I'm running into this error:
Copy code
dagster.core.errors.DagsterStepOutputNotFoundError: When executing transform.compute discovered required outputs missing from previous step: [StepOutputHandle(step_key='load.copy_django_logs_to_redshift.compute', output_name='result')]
I thought maybe I could fix it by passing in
rexecution_config
to RunConfig, but when I do I get this error:
Copy code
TypeError: __new__() got an unexpected keyword argument 'rexecution_config'
Could someone give me pointers on how to run a specific pipeline step that depends on previous steps? Thanks so much really appreciate all the help.
a
can you provide some context on your overall goal? You may find
execute_solid
to be appropriate if you are trying to test a single solid
b
I’d like to be able to test a single solid in my pipeline, assuming the previous solid have already completed successfully, but without having to rerun the previous steps each time
I had used execute_solid before but ran into an issue that it doesn’t work with aliased solids so I’ve been sticking to the step key method
a
ah ok
I think
pipeline.build_sub_pipeline([solid_name])
might do the trick
(and probably should be what i suggested before)
b
So I see that function within PipelineDefinition, but how do I pass in environment configs, mode etc in the same way I can with execute_pipeline?
Maybe there's an example somewhere you could point me to?
Thanks again
a
you will end up with
execute_pipeline(pipeline.build_sub_pipeline([solid_name]), ...)
- which assumes the mode you need is already on the full pipeline
b
Ahh I see
ok thanks
b
Ok I was able to get that to work, thanks
👍 1
a
sorry for the run around with the
step_keys_to_execute
business
b
No worries!
doing it that way does require me to change the config each time though
Meaning
build_sub_pipeline
seems error if I pass in the config for the entire pipeline, rather than passing in only the config for that solid. I think I can work around this but it would be nice if it just knew how to disregard the config for other solids w/out erroring
a
ya this is also problematic when using this subsetting feature via dagit
b
Cool—I'll keep my eye on this ticket.
Thanks for the help