Alexander Shirokov
09/07/2021, 1:04 PMconfiguration.yaml
loggers:
console:
config:
log_level: ERROR
i’ ve found the only way to use this configuration in Python program:
1. load dictionary from yaml
import yaml
with open('configuration.yaml'. 'r\) as f:
log_config = yaml.load(f)
2. pass this log_config into execute_pipeline function
import dagster as d
d.execute_pipeline(pipeline, run_config=log_config)
but this way is not suitable - way better to use repositories. not using yaml configuration generates new problems - in my example, my custom logger won’t overwrite default dagster colored_logger and a won’t reach expected effect.
so, the question is - how to load yaml configuration and use it when using repositories?
thanks in advance!from dagster import op, graph, config_from_files
@op(config_schema=str)
def my_op(_):
pass
@graph
def basic():
my_op()
basic.to_job(config=config_from_files("/path/to/yaml.yaml"))
This config becomes the default config for the job, and can be overridden.Alexander Shirokov
09/07/2021, 1:12 PMchris
09/07/2021, 1:25 PMAlexander Shirokov
09/09/2021, 4:00 PMchris
09/09/2021, 4:16 PM@schedule(...)
def my_schedule():
return my_preset.run_config