hi all, liking dagster a lot from the tutorial, planning to use with

dbt

to manage a workflow consisting of transforming a dataset from bigquery, then running ML on it, and putting results back in bigquery. I know this was asked before, but one question — when re-running a pipeline containing heavy computations, are there mechanisms for avoiding re-computation of parts whose inputs haven’t changed? Let’s say the solids’ inputs [outputs] are results read from [dumped to] files Essentially some type of caching/invalidation mechanism.

cc @chris

Hey @Prasad Chalasani! We actually have an experimental feature for versioning/memoization of pipeline runs. We have an example intended to get someone up and running with these features coming out later today with our mini-release.

great, thanks for the fast response!

Once the docs are updated, I'd be happy to link that to you, and would love to get your feedback on the user experience

happy to give feedback

you can also re-execute a solid subselection, using the upstream outputs from a previous run. let me find the docs..

will take a look, thank you!

Oh also, this only works if the intermediates are written to persistent storage (ie not in memory), so S3 for example would work