https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Matthew Koh

07/14/2022, 5:51 PM
hi! i’m evaluating dagster and trying to figure out if there’s any way to set configuration for an Op’s dependencies based on values computed from the downstream Op’s configuration. If my job looks like
Copy code
@job
def job():
  c_op(a_op(), b_op())
c_op has some config and those in turn change how a_op and b_op should run How would I write the
Job
to do this?
y

yuhan

07/14/2022, 8:49 PM
There are several ways: • conditional branching can model conditional execution e.g. skip c_op if a_op and b_op don’t have outputs ◦ doc: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#with-conditional-branching • dynamic output can dynamically generate downstream steps (fan out) ◦ doc: https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs
m

Matthew Koh

07/15/2022, 1:37 PM
Thanks. Dynamic graphs and conditional execution seems close to what I want, but not sure how to dynamically configure the ops. Is there something I’m missing? Basically I want to set the configuration for all the ops based on a single config. Our graphs are always going to condense down into a single node that runs last. Currently, each “op’s” configuration is determined by walking backwards up the tree. At each recursive step, the dependencies and their configs are determined by the current node’s configuration. Is there some way to mimic that type of setup?
@yuhan is my requirement covered by dynamic graphs and I just need to read the docs better? 🙂 or is there some other way to achieve this setup?
3 Views