https://dagster.io/ logo
Title
j

jeremy

05/17/2021, 8:23 PM
Hey there, wondering if there is a recommended way to handle pipeline versions? For example in a situation where the pipeline changes on the more frequent side, but you also would want to be able to run the older versions of the pipeline if needed. One way I can think of right now is to add new solids to the grpc and dynamically create new pipelines with the DSL using a set of YAML files, but that could potentially get pretty messy. Would there be a better pattern to organize this?
e

Eduardo Santizo

05/18/2021, 4:00 PM
You could perhaps construct the pipeline programatically, pulling certain versions of solids and constructing the pipeline based on a configuration number in a YAML file? Its the first thing that came to mind, but it might also be kinda messy.