Hi folks! Is there a way to have two similar sub-graphs with different resources in one job? My case is pretty simple - I have to ETL sub-graphs which pulls data from the same source (but from different tables, configured in configs) but perform slightly different transformation. So I use different "transformer"-resource in every usage of this graph. But now I need to ETL two tables and start dbt jobs after both sub-graphs are completed. So my thought is to nest these two ETL subgraphs and dbt-operator in one job. But the issue is - how can I provide different resources for ETL with similar resource key?

Configured

API does not deal with resources unfortunately.

hi @Ivan Tsarev! I see what you mean. I think there are two main options here. The first would be to switch to encode the different behaviors you want from the transformer resource in the config (rather than by passing in a different resource definition). You'd pass some bit of config into that single transformer resource's function, or branch what function you call based on the config. The other option would be to switch to a "graph factory" approach, where you actually generate different graphs in each case, i.e.

def get_graph(resource_key: str):
    @op(
        name=f"op_for_{resource_key}",
        required_resource_keys={resource_key}
    )
    def some_op():
        ...
    
    @graph(name=f"graph_for_{resource_key}")
    def some_graph():
         return some_op()
    
    return some_graph

Hi @owen! Thank you for response! I've thought about configuring resources but did not find elegant enough way to do it. I mean the most intuitive thing for me would be having family of

Transformer

classes and simple

TransformResource

wrapper around them which can be configurable for every job. But I didn't manage to make resource config param to be a arbitrary python class. I suppose there are strict restrictions in legacy system which mostly preserved in new pydantic-based configs as well (However I did not completely give up on making such "architecture" works inside pydantic config but no luck yet). I understand that its possible to make less elegant solution with choosing transform logic inside resource based on string config but tbh it does not feel that good 🙂 And graph factory is a completely new approach for me, maybe I should give it a try as well, thank you!