Hi. I have a graph containing a bunch of ops. By default, dagster executes (whatever possible) in parallel but there is one particular op in this graph which I'd like to run solo (no ops running in parallel to that) when the execution comes to it since this op needs most of the system's memory to itself otherwise it crashes. Is it possible to do such thing while keeping the op still in the same graph?
dagster bot responded by community 2
05/26/2022, 11:18 PM
I don't think there a solution for this use case built into dagster's framework, but you could probably model this into the graph by adding extra dependencies between ops, in other words, creating inputs/outputs between this heavy op and ops that are able to run parallel to it.
05/27/2022, 12:44 AM
yeah, one way you could do this is to have a
input, and then pass in a list of all necessary preceding inputs
05/27/2022, 11:54 AM
Yeah but that basically changes the logical sequence of the graph which I don't want to do.
05/27/2022, 1:16 PM
I don’t think that we yet have a way to constrain op execution order outside of using dependencies, but cc @alex for confirmation.
But you can definitely achieve what you’re after using Nothing dependencies, which are an admittedly imperfect solution.