hello, i want to generate a dynamic graph from job configuration such as

1. read table 1,2 from a resource (postgres or snowflake)  
2. join tables together with id column with join-op 
3. OR union table together by simple appending with append-op 
4. writes back to another resource,

i read through the dynamic graph section in the doc but don’t seem to be able to programmatically pick which ops to use for step 2 or 3, i am see a way to access user config from

@job

or

@repository

, any idea how to achieve this? Thanks!

Hi Wen. One workaround I can think of with a programmatically defined job is to create your job with optional outputs. If a downstream op's input comes from an upstream op's optional output, if the optional output is not yielded, the downstream op will be skipped. You can do something like this:

@op(
    out={"downstream_join": Out(is_required=False), "downstream_union": Out(is_required=False)},
    config_schema={"downstream_join": boolean},
)
def my_op(context):
    my_output = 1
    if context.op_config["downstream_join"] == True:
        yield Output(my_output, "downstream_join")
    else:
        yield Output(my_output, "downstream_union")


@op
def join(table):
    return 1


@op
def union(table):
    return 1


@job
def my_job():
    downstream_join, downstream_union = my_op()
    join(downstream_join)
    union(downstream_union)

The example above defines two optional outputs, and from op config will select which downstream op to run. Then, it will only return an output for the selected downstream op.

got it, it’s workable. thanks. is there a strong reason or literature of why

@job

or

@repository

do not have access to runtime config?

Dagster jobs basically function as a wrapper to define dependencies between your different operations/resources, so it's not a configurable entity. The configurable entities are ops/assets/resources, which the job will create an execution context for that contains the validated run configuration for that specific op/asset/resource

thanks for explaining.