Hey, anyone used ray to accelerate dagster pipelin...
# ask-community
o
Hey, anyone used ray to accelerate dagster pipelines? I'm thinking that i can use ray in my asset definitions and then use the plasma store as an iomanager. I'm not sure if this will work though since I think that after materialisation the process will finish and the ref counter will get decremented and this will result in garbage collection of the object. I am not super confident in my Ray or Dagster knowledge here so this may just work. Otherwise maybe a ray Aactor could be initialised to maintain the ref I guess? Any thoughts about this?
🤖 1