https://dagster.io/ logo
Title
a

Alexander Shirokov

09/07/2021, 1:04 PM
good day! in dagster documentation there are a couple of examples, that use yaml configuration, like:
configuration.yaml

loggers:
  console:
    config:
      log_level: ERROR
i’ ve found the only way to use this configuration in Python program: 1. load dictionary from yaml
import yaml
with open('configuration.yaml'. 'r\) as f:
    log_config = yaml.load(f)
2. pass this log_config into execute_pipeline function
import dagster as d

d.execute_pipeline(pipeline, run_config=log_config)
but this way is not suitable - way better to use repositories. not using yaml configuration generates new problems - in my example, my custom logger won’t overwrite default dagster colored_logger and a won’t reach expected effect. so, the question is - how to load yaml configuration and use it when using repositories? thanks in advance!
1
c

chris

09/07/2021, 1:07 PM
With the current APIs, you can do this using presets. You can use the `.from_files`(link) method to create a preset from yaml, and pass that to a pipeline. This will make it available from every execution entrypoint.
With the new APIs, you can load the config in using config_from_files, and then pass the result to a job:
from dagster import op, graph, config_from_files

@op(config_schema=str)
def my_op(_):
    pass

@graph
def basic():
    my_op()

basic.to_job(config=config_from_files("/path/to/yaml.yaml"))
This config becomes the default config for the job, and can be overridden.
👍 1
a

Alexander Shirokov

09/07/2021, 1:12 PM
thank you, Chris! i will try to implement this into code. new api with graph’s and op’s is going to be OP :)
😛owerup: 1
c

chris

09/07/2021, 1:25 PM
feel free to let us know if any other questions come up !
💫 1
a

Alexander Shirokov

09/09/2021, 4:00 PM
@chris, i would like to clarify one more question about presets - is there a way to use custom PresetDefinition by default in schedule. I’m using my custom logger, but it works only when i’m running pipeline in playground, but in schedule runs it doesn’t work. Thanks in advance!
c

chris

09/09/2021, 4:16 PM
We don't have a way to use a preset in a schedule directly - the pattern you'll see a lot is pulling the run config off of their preset and returning it from the schedule:
@schedule(...)
def my_schedule():
    return my_preset.run_config