Hi team! I'm running some job which looks like this:

initial_data = initial_op()

data1 = processor1_op(initial_data)
persist1_op(data1)

data2 = processor2_op(initial_data)
persist2_op(data2)

I was expecting that the order of ops will remain the same as in job definition, but Dagster run them like this:

initial_op -> processor1_op -> processor2_op -> persist1_op -> persist2_op

, which is kind of bad: results of

processorX_op

will be stored in io_manager (in memory) till the very end of the job execution, but that memory could be released probably if the op execution order was preserved. Is there any way to set some execution ordering strategies or priorities? I know there are "nothing" dependencies, but this is a bit different situation as there is no implicit dependency.

Is there any way to set some execution ordering strategies or priorities?

you can set

dagster/priority

tag on ops to influence execution order https://docs.dagster.io/_apidocs/execution#dagster.in_process_executor

Hi, is there an example for this? I found adding metadata tags to a job with this:

@job(
    metadata={
        "owner": "data team",  # will be converted to MetadataValue.text
        "docs": MetadataValue.url("<https://docs.dagster.io>"),
    }
)
def my_job_with_metadata():
    my_op()

But, trying something similar with ops doesn't work as the metadata argument is not recognised:

@op(
    required_resource_keys={"database"},
    metadata = {
        "dagster/priority": 2
    }
    )
def my_op(context):

I used "tags" on ops.

I see, thanks!

But, unfortunately, it doesn't seems to work 😞 . I'm checking it right now. @alex, could you please assist a little bit?

Ok, thanks - so if you have a priority tag for one op, you have to have them for all ops in a job, even though the default is set to 0? As it says here:

Execution priority can be configured using the

dagster/priority

tag via solid/op metadata, where the higher the number the higher the priority. 0 is the default and both positive and negative numbers can be used.

I'm not sure. Should be tested more to confirm. I'd like to know the algorithm behind the priorities graph resolution to make sure.

know the algorithm behind the priorities graph resolution

Each time the executor can select ops that are ready to execute (dependencies completed) to fill open execution slots - it will sort them based on the priority tag as the description above cites.