https://dagster.io/ logo
Powered by
#ask-community
Title
# ask-community
s

Stephen Bailey

05/16/2022, 3:20 PM
is there a way for me to create a parameterized partitioned config? I have a graph that executes an etl job for a table, and i want to create a partitioned job per table. the intuitive way to do this is:
Copy code
@daily_partitioned_job(...)
def daily_config(table_name, start, end):
   return config_for_table

job_list = []
for ii in [table1, table2, table3]:
    j = my_graph.to_job(name=f"replicate_{table}", config=daily_config(table_name = ii))
    job_list.append(j)
but that doesnt quite work, because it instantiates the config... is there a recommended practice?
🤖 1
✅ 1
ah, got it! what i needed to do was create a config generator function, then instantiate that within the loop ... cool
Copy code
def daily_config_generator(table_name: str):
    @daily_partitioned_config(...)
    def daily_config_fn(start: datetime, _end: datetime):
        ...
        config["table_name"] = table_name
        return config
    return daily_config_fn

job_list = []
for ii in [table1, table2, table3]:
    config = daily_config_generator(ii)
    j = my_graph.to_job(name=f"replicate_{table}", config=config)
    job_list.append(j)
Powered by