Hey, anyone used ray to accelerate dagster pipelines?
I'm thinking that i can use ray in my asset definitions and then use the plasma store as an iomanager. I'm not sure if this will work though since I think that after materialisation the process will finish and the ref counter will get decremented and this will result in garbage collection of the object.
I am not super confident in my Ray or Dagster knowledge here so this may just work.
Otherwise maybe a ray Aactor could be initialised to maintain the ref I guess? Any thoughts about this?