Mykola Palamarchuk
11/02/2022, 2:50 PMinitial_data = initial_op()
data1 = processor1_op(initial_data)
persist1_op(data1)
data2 = processor2_op(initial_data)
persist2_op(data2)
I was expecting that the order of ops will remain the same as in job definition, but Dagster run them like this: initial_op -> processor1_op -> processor2_op -> persist1_op -> persist2_op
, which is kind of bad: results of processorX_op
will be stored in io_manager (in memory) till the very end of the job execution, but that memory could be released probably if the op execution order was preserved.
Is there any way to set some execution ordering strategies or priorities? I know there are "nothing" dependencies, but this is a bit different situation as there is no implicit dependency.alex
11/02/2022, 3:26 PMIs there any way to set some execution ordering strategies or priorities?you can set
dagster/priority
tag on ops to influence execution order
https://docs.dagster.io/_apidocs/execution#dagster.in_process_executorMegan Beckett
11/15/2022, 9:28 AM@job(
metadata={
"owner": "data team", # will be converted to MetadataValue.text
"docs": MetadataValue.url("<https://docs.dagster.io>"),
}
)
def my_job_with_metadata():
my_op()
But, trying something similar with ops doesn't work as the metadata argument is not recognised:
@op(
required_resource_keys={"database"},
metadata = {
"dagster/priority": 2
}
)
def my_op(context):
Mykola Palamarchuk
11/15/2022, 11:08 AMMegan Beckett
11/15/2022, 11:08 AMMykola Palamarchuk
11/15/2022, 11:49 AMMegan Beckett
11/15/2022, 12:11 PMExecution priority can be configured using thetag via solid/op metadata, where the higher the number the higher the priority. 0 is the default and both positive and negative numbers can be used.dagster/priority
Mykola Palamarchuk
11/15/2022, 12:14 PMalex
11/15/2022, 3:29 PMknow the algorithm behind the priorities graph resolutionEach time the executor can select ops that are ready to execute (dependencies completed) to fill open execution slots - it will sort them based on the priority tag as the description above cites.