jasono's question above got me wondering. What is the best way or usage pattern in dagster where I can pass in a list of file names or paths where every solid in the pipeline will be given a file path in the list as input then with each subsequent pipeline run, it will loop through the list of file paths. So the structure of the pipeline is the same or not dynamic, but instead, the input to the pipeline is what will be dynamic.

Hi Daniel- not sure I have a full grasp on your use case, but you could look in to partition sets https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions

Thanks @johann! I think partition set maybe is something I can use. When I saw the docs for partition set earlier, I was under the impression it is for dates and not like file names that I can pass in or generate. But I guess I can pass in or create a list of file names. I also wasn't sure how to use the partition set definition, but it turns out the documentation isn't very explicit on how to use partition set. I found out later it is used in the repository definition per this GH issue: https://github.com/dagster-io/dagster/issues/4074 Also, the latest docs (0.11.x) no longer shows example of "lazy" construction of schedules. Does that mean "lazy" construction is no longer supported? Here is legacy doc for repository example: https://legacy-docs.dagster.io/overview/repositories-workspaces/repositories

Date ranges are the most common use case for partitions, but they are intentionally generic to support other usages