https://dagster.io/ logo
#announcements
Title
# announcements
d

Daniel Kim

12/03/2020, 1:49 PM
Hello! I am new to dagster. I have a question. Is there a way to run a pipeline multiple times based on a list of values that I can pass in somehow? For example, I slightly modified the hello world example from the main github page where I am executing 2 tasks that don't depend on each other.
from dagster import (
    
execute_pipeline,
      
pipeline,
      
solid
)
@solid
def get_name(_, plant_code: str):
    
print(f"Plant code: {plant_code}")
@solid
def hello(context, plant_code: str):
<http://context.log.info|context.log.info>('Hello, {name}!'.format(name=plant_code))
@pipeline
def hello_pipeline():
    
get_name()
    
hello()
if __name__ == "__main__":
    
run_config = {
        
"solids": {
            
"get_name": {"inputs": {"plant_code": {"value": "Plant1"}}},
            
"hello": {"inputs": {"plant_code": {"value": "Plant1"}}}
        
}
    
}
    
execute_pipeline(hello_pipeline, run_config=run_config)
# want to do something like this:     value = ["Plant1", "Plant2", "Plant3"]     for plant_code in value:         execute_my_pipeline(plant_code) etc... I can create a pipeline for one plant code "Plant1", but let's say I want to execute this set of tasks in this pipeline for each additional plant code. With the run_config, not sure if it is possible to loop through a list of values passed into the "value" parameter. I hope my question makes sense. Any help is much appreciated!
d

daniel

12/03/2020, 3:18 PM
Hi Daniel - is it not an option to use different run config for each execution to accomplish this? i.e. you could take in plant_code as an argument to your function that calls execute_pipeline, and set the run config that's passed into execute_pipeline based on the value of plant_code?
d

Daniel Kim

12/03/2020, 3:36 PM
Hi @daniel I see what you mean. I can do what you suggest for now. My list of values or plant codes could be dynamic or a user may wish to be able to pass in any number of values for the pipeline input. I realize that the proper terminology for what I think I am after is perhaps called creating "dynamic pipelines". I think I found a GH issue related to what I'm after: https://github.com/dagster-io/dagster/issues/2077 I'll take a deeper look at Nick's suggestion and let it sink in for me for a while. Thanks for your response!
d

daniel

12/03/2020, 3:38 PM
Ah great - you might also find https://github.com/dagster-io/dagster/discussions/2902 interesting, that's a bit more recent and outlines future work to make pipelines parametrizable in the same way that solids are
👍 1