https://dagster.io/ logo
#ask-community
Title
# ask-community
i

Ivan Tsarev

03/16/2023, 6:17 PM
Hi everyone! Is there any way to parametrize one of graph`s op with callable? I mean, I have "template" graph with three ops - loading data from GCS -> transforming data -> upload data to BQ. So both first and last ops takes string params like GCS uri and BQ table id, but to make graph itself completely reusable without any changes in dagster-coupled code I need to change transformation logic inside the second op. The only way I found yet is to pass there strings argument with module and function (class) name to import "transform" logic inside the op itself. It works but looks dirty as hell, so I wonder is there other options to do so, since I cant pass Callable as part of the op config for sure.
s

Spencer Nelson

03/16/2023, 6:19 PM
I think your code might be cleaner if you use Resources for that
i

Ivan Tsarev

03/16/2023, 6:32 PM
Hm, I should probably think about Resources once more. Initially I decided that it has same issue and cannot be configured by callable, but probably I should just create separate resources for every transformer-function. Even if it exposes some of dagster logic for someone using template graph, its still cleaner to understand
👍 1
s

Spencer Nelson

03/16/2023, 6:37 PM
exactly. Then your op has something like
required_resource_keys={"transformer_1", "transformer_2", ...}
Your job can set
"transformer_1": actual_implementation_1
or
"transformer_1": actual_implementation_2
1
so it will feel pretty configurable in the way you describe, i think
i

Ivan Tsarev

03/16/2023, 7:48 PM
Yeah, sounds great. I probably can even hide Resource definition somewhere in parent meta-class if really want to :)
💯 1