Also if I use an IOManager say S3 then can I re run specific

Question

Also if I use an IOManager say S3 then can I re run specific solids or just a part of the pipeline If I do that the intermediate data will be fetched from S3 right

yuhan · Answer

yes as long as it s a persistent IO Manager dagster machinery would handle the logic for loading from previously persisted intermediate data

madhurt · Answer

so the `s3` IOManager is a persistent one right

madhurt · Answer

I am asking because there will be no data that will be stored locally All intermediate outputs will be in S3 through this IOManager Now if a pipeline has solids A gt B gt C and if B fails and now I need to re run B onwards i e B and C So B should get A s data through S3 right Is that expected with the `s3` IOManager

madhurt · Answer

cc < yuhan>

yuhan · Answer

yes the data will go through s3 see details in <https docs dagster io deployment guides aws using s3 for io management>