Hello! Is there a best practice to lock a resource across ops in a graph? Two ops in the middle of my graph require exclusive access to a mutex; the mutex is acquired by the first op and cannot be released until the second one has finished.
In an ideal world, I'd write something like that in my graph and have the gRPC server hold the lock while the ops execute:
a = op1()
But of course, this is not as easy. 😉
03/04/2022, 2:55 PM
in case you use the dagster queued launcher you can have queues and workers and assign queues to ops. If one tagged queue has only one worker you would achieve the desired result
03/04/2022, 2:57 PM
Hi! We do use the queued launcher.
One important detail that I forgot to mention is that the resource that needs to be locked is not known in advance and is determined dynamically by another op earlier in the graph.
I’ve looked into using a subgraph with a specific tag to leverage the concurrency options offered by the run coordinator, but it does not seem possible to set the tag associated to the subgraph dynamically at runtime.
In a nutshell, my pipeline is as follows:
1. Start the run, examine initial parameters.
2. Determine I need to lock A.
3. Lock A.
4. Run a bunch of tasks in parallel.
5. Collect the results.
6. Unlock A.