https://dagster.io/ logo
#ask-community
Title
# ask-community
w

Wen Hou

06/17/2022, 6:49 PM
hello, i want to generate a dynamic graph from job configuration such as
Copy code
1. read table 1,2 from a resource (postgres or snowflake)  
2. join tables together with id column with join-op 
3. OR union table together by simple appending with append-op 
4. writes back to another resource,
i read through the dynamic graph section in the doc but don’t seem to be able to programmatically pick which ops to use for step 2 or 3, i am see a way to access user config from
@job
or
@repository
, any idea how to achieve this? Thanks!
c

claire

06/17/2022, 7:41 PM
Hi Wen. One workaround I can think of with a programmatically defined job is to create your job with optional outputs. If a downstream op's input comes from an upstream op's optional output, if the optional output is not yielded, the downstream op will be skipped. You can do something like this:
Copy code
@op(
    out={"downstream_join": Out(is_required=False), "downstream_union": Out(is_required=False)},
    config_schema={"downstream_join": boolean},
)
def my_op(context):
    my_output = 1
    if context.op_config["downstream_join"] == True:
        yield Output(my_output, "downstream_join")
    else:
        yield Output(my_output, "downstream_union")


@op
def join(table):
    return 1


@op
def union(table):
    return 1


@job
def my_job():
    downstream_join, downstream_union = my_op()
    join(downstream_join)
    union(downstream_union)
The example above defines two optional outputs, and from op config will select which downstream op to run. Then, it will only return an output for the selected downstream op.
w

Wen Hou

06/17/2022, 7:47 PM
got it, it’s workable. thanks. is there a strong reason or literature of why
@job
or
@repository
do not have access to runtime config?
c

claire

06/17/2022, 8:02 PM
Dagster jobs basically function as a wrapper to define dependencies between your different operations/resources, so it's not a configurable entity. The configurable entities are ops/assets/resources, which the job will create an execution context for that contains the validated run configuration for that specific op/asset/resource
w

Wen Hou

06/17/2022, 9:19 PM
thanks for explaining.
4 Views