19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello! Is there a best practice to lock a resource...
# dagster-feedbackv
VxD
03/04/2022, 2:49 PMHello! Is there a best practice to lock a resource across ops in a graph? Two ops in the middle of my graph require exclusive access to a mutex; the mutex is acquired by the first op and cannot be released until the second one has finished.
VxD
03/04/2022, 2:51 PMIn an ideal world, I'd write something like that in my graph and have the gRPC server hold the lock while the ops execute:
Copy code
with lock():
a = op1()
op2(b)VxD
03/04/2022, 2:52 PMBut of course, this is not as easy. 😉
g
geoHeil
03/04/2022, 2:55 PMin case you use the dagster queued launcher you can have queues and workers and assign queues to ops. If one tagged queue has only one worker you would achieve the desired result
v
VxD
03/04/2022, 2:57 PMHi! We do use the queued launcher.
One important detail that I forgot to mention is that the resource that needs to be locked is not known in advance and is determined dynamically by another op earlier in the graph.
VxD
03/04/2022, 3:01 PMI’ve looked into using a subgraph with a specific tag to leverage the concurrency options offered by the run coordinator, but it does not seem possible to set the tag associated to the subgraph dynamically at runtime.
VxD
03/04/2022, 3:10 PMIn a nutshell, my pipeline is as follows:
1. Start the run, examine initial parameters.
2. Determine I need to lock A.
3. Lock A.
4. Run a bunch of tasks in parallel.
5. Collect the results.
6. Unlock A.
2 Views