https://dagster.io/ logo
#ask-community
Title
# ask-community
x

x.z.

07/06/2022, 12:33 PM
Hello everyone, I am exploring the possibility of replacing notebook-based data analysis workflows with a Dagster-based one. A crucial loop in such workflows is to be able to re-execute only the changes/additions to a pipeline while reusing existing results whenever possible. To this end, I have experimented with a "edit, reload, re-execute" approach, but I have yet to find a way to (re-)execute newly appended ops to a job based on the cached results from a previous run. What would be the best practice for such use case? Perhaps a more general question is: do people think Dagster is suited for such usage in general? Thanks!
s

sandy

07/06/2022, 3:14 PM
Hey @x.z. - this should be possible if you use software-defined assets. I.e. you can add a new asset that depends on an existing asset and then materialize it, and it will use the contents of the existing asset
x

x.z.

07/06/2022, 4:06 PM
Thanks @sandy, will give it try.
2 Views