Hi dagster team! We are using dagster in our produ...
# announcements
l
Hi dagster team! We are using dagster in our production pipeline. Our pipelines have different yaml files depending on the environment (e.g. staging, production). Currently we are facing a problem when changing our yaml files for the run config. It already happened some times that our yaml files aren’t valid, but we only discovered it when the pipeline was executed in production. Is there any possibility to check for each pipeline if the yaml files contain valid configurations for the pipeline before the pipeline is executed?
👍 1
s
Hey Lee! This is a very reasonable request and a bit of an oversight on our part. Will file an issue!
https://github.com/dagster-io/dagster/issues/2875 Feel free to add anything I missed!
l
Hi schrockn, thanks for the fast response! the github issue looks good. we are looking forward for this feature to be released!
s
in the mean time you should be able to do something this like:
Copy code
from dagster.core.definitions.run_config_schema import create_environment_type 
 from dagster.config.validate import process_config
 from dagster.core.errors import DagsterInvalidConfigError

def check_run_config(pipeline_def, run_config, mode=None):
        mode = mode or pipeline_def.get_default_mode_name()
        environment_type = create_environment_type(pipeline_def, mode)

        config_evr = process_config(environment_type, run_config)
        if not config_evr.success:
            raise DagsterInvalidConfigError(
                "Error in config for pipeline {}".format(pipeline_def.name),
                config_evr.errors,
                run_config,
            )
👍 1
l
thanks a lot, we will try it