19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
d
Dan Corbiani
06/02/2021, 12:51 PM@chris We've been continuing our memoization work. All is mostly good except for the DagsterNoStepsToExecute error. Is it intentional that an exception is raised if there is nothing to do? Am I calling the pipeline incorrectly?
I end up having to wrap a bunch of our execution processes in try blocks like below:
It makes me wonder how to access the output from a previously run memoized pipeline.
c
chris
06/02/2021, 2:12 PMYea, this is the (currently) correct behavior. If there are no steps to execute, then this error will be thrown. Essentially dagster isn't designed to run with no steps, and so to prevent more esoteric failures later on, we just fail fast here.
We don't currently have a super convenient API to access run results to my knowledge, and this is a known feature gap. In combination with the no step keys to execute error though, I'm seeing that it makes for a bit of an incongruency.
one potential workaround is directly instantiating your io manager, and using build_input_context/build_output_context to pass to the
handle_inputload_outputd
Dan Corbiani
06/02/2021, 5:49 PMhmm. That's interesting. I'm glad to hear I interpreted it correctly. Do you anticipate this making it's way onto the roadmap at some point? It also seems like I could put a dummy step into the pipeline that would never get saved.
c
chris
06/02/2021, 5:50 PMthat seems like a good workaround for now. A better solution is definitely on the roadmap, whether that is a better output-retrieval API or being able to execute without any steps, or both
d
Dan Corbiani
06/03/2021, 1:14 AMI realized this is a bit more frustrating than I originally thought. I've been playing around with it more and it seems like if something is loaded, I can't access it as output at all. Is that also correct?
The use case is if I rerun a pipeline I can not access the previous output for a step. Is there something I can do to help make that possible?
I also looked through the instantiation docs but I'm not sure how to get that to work. It seems like I'm also missing the version information when I create that context.
I can get around the limitation by creating a dummy solid that calls that output. As long as it isn't saved, I can access the output.
c
chris
06/03/2021, 2:20 PMHmm. Are you saying that from the result object returned by
execute_pipelineI'll investigate solutions around this space. I think both of these problems should be fixable: not failing if we have no steps to execute, and retrieving outputs from previously memoized runs.
d
Dan Corbiani
06/03/2021, 6:04 PMyes. that's the conclusion I've come to. I've ended up creating a dummy step that won't be memoized. That allows us to basically point back to the memoized dataframe. Happy to share whatever I can to help.
we fully realize this is experimental. It's a fantastic start.