https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Pierre Pasquet

02/08/2024, 3:47 PM
hey- is the InMemoryIOManager broken? It seems the
values
dictionary is not being persisted across different asset classes. Apparently, a separate instance of the class (hence key value store) is being created for every EXECUTION STEP (
STEP_WORKER_STARTED
). Is this expected behavior?
i

Ion Koutsouris

02/08/2024, 5:02 PM
The asset class is not passed through because each step is executed in a new process, super annoying ofcourse when you want to use Ops this way
👍 1
p

Pierre Pasquet

02/08/2024, 5:02 PM
this is wild
i

Ion Koutsouris

02/08/2024, 5:03 PM
So it only works together with a single executor
p

Pierre Pasquet

02/08/2024, 5:03 PM
yeah thats what i ve reverted to
i

Ion Koutsouris

02/08/2024, 5:03 PM
They basically need to allow to run all ops in a single graph asset in the same process
p

Pierre Pasquet

02/08/2024, 5:03 PM
do you know if multiple jobs that are all defined with a single executor will share the same process?
or will they be parallelised?
i

Ion Koutsouris

02/08/2024, 5:06 PM
@Pierre Pasquet that I'm not sure, I think each seperate job has its own process
p

Pierre Pasquet

02/08/2024, 5:06 PM
ok sweet
i

Ion Koutsouris

02/08/2024, 5:06 PM
But that executor is only useful in a local device
p

Pierre Pasquet

02/08/2024, 5:07 PM
i think you are still better off running in a single process than having to deal with I/O operations at each step
lol ok
i

Ion Koutsouris

02/08/2024, 5:09 PM
Or we need to switch to pushing everything on a single Vm
Which could be possibly but then we lose flexibility because steps/assets require different amount of resources
3 Views