👋 is there a clean pattern for running a dynamic mapping in job, inan ordered way? basically I want something like

@op(out=DynamicOut(int), config_schema={"start": int, "stop": int)
def generate_things(context) -> int:
    for i in range(context.op_config["start"], context.op_config["stop"]):
        yield DynamicOutput(value=i, mapping_key=f"{i}")

@graph
def graph_of_complex_things(i):
    ... # do graph of things with i

@job
def dynamic_size_ordered_job():
    for i in generate_things():
        graph_of_complex_things(i)

I was poking around and

generate_things().map(graph_of_complex_things)

is the closest thing I can find but isn't ordered

in an ordered way

can you be more specific about what you mean here?

Hey sorry just saw this. Basically have each op depend on the previous but have a dynamic number of ops executed

@Erik Did you discover any other solutions for this? I'm trying to achieve something similar, and have been trying to use GraphDefinition to build a job at runtime to define the dependencies, but it seems you can't define graphs/jobs at runtime unless I'm missing something.

I don’t believe there is a way to achieve this currently

you can’t define graphs/jobs at runtime

correct

Not sure if this is the correct way to ask the question, but is there a way to reinitialize a graph/job definition immediately prior to running a job? Let's say I have a job I run through dagit, and in the definition for that job I reach out to a DB, and use the information returned to create a GraphDefinition. When I rerun the job, will it reinitialize that GraphDefinition?

it will - you may run in to some weird product experiences when the shape of the graph changes at unexpected times, but we always load the job/graph from source

When the job/graph is loaded, is the run_config available at definition time? Or is there some other way to get run_config accessible to definition time?

nope,

run_config

targets a well defined schema that is a property of the

JobDefinition

. If you want to make a yaml -> job/graph system, you will have to manage that yourself outside of dagit