Hi Team, I have a calculation service that consumes a calculator config, transforms it into a sql, submits, and then returns the result. My @job is to consume the list of calculators config and a dependencies dict, and turn them into a DAG. Every calculator config represents an op, so I use the jinja2 template to create many ops with different calculator ids. Then use another jinja2 template to create @job with DAG by consuming those ops and dependencies. I feel this way awkward, any other better idea, thanks in advance.

one option could be to use dynamic ops - you could have a single op at the top that reads in the configuration and returns a

DynamicOutput

for each individual calculation with the necessary config. then a downstream op will be created for each of those dynamic outputs https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs

@jamie Thanks for the quick response. I did try DynamicOutput, but it seems like it cannot handle dependencies between every single op. In my case, I need to specify every auto_generate_calculator_xxx(@op) result is the input of another auto_generate_calculator_xxx(@op), rather than simply collect all results.

ah i see. yes in that case Dynamic ops/graphs won’t work. in that case i feel like your current solution is fine! I haven’t seen much use of jinja to template ops but the general concept of generating a bunch of ops from a template is super common. and no reason jinja can’t be used for that

Really appreciate your professional advice.