Hi all I have a question about managing configura...
# ask-community
c
Hi all I have a question about managing configuration files. So Dagit has great config editor build-in in launchpad. It validates types using config schema from code. This is all great. But when it comes to configurate for scheduled or sensored jobs API is forcing to pass
run_config
as a dict inside a code. I think this is not so good becouse this combines configuration inside code logic.
Copy code
@schedule(job=configurable_job, cron_schedule="0 0 * * *")
def configurable_job_schedule(context: ScheduleEvaluationContext):
    scheduled_date = context.scheduled_execution_time.strftime("%Y-%m-%d")
    return RunRequest(
        run_key=None,
        run_config={
            "ops": {"configurable_op": {"config": {"scheduled_date": scheduled_date}}}
        },
        tags={"date": scheduled_date},
    )
So what I'm ending doing is having a folder of yaml config files and poiting to them inside the code to separeate confids, passwords etc from code. (Alternative way is to use env variables)
Copy code
@schedule(job=calculate_items_stock_job, cron_schedule="0 0 * * *")
def calculate_daily(context):
    
    config_yaml = r'project/graphs/conf/job_config.yaml'

    config = yaml.safe_load(open(config_yaml).read())
    
    return RunRequest(
        run_key=None,
        run_config=config,
        tags={"date": scheduled_date, "source": "gam_prod"}
    )
But where is the point of using config schema then? Can I use Dagit to manage config files for schedules and sensors to laverage config schema? Or maybe I misunderstand this concept? I understand that this is avaible to generate config dynamicly like
date_from
, and
date_to
to pass to ops or something. But resource configuration is mainly static.
s
The way my mental model works is that I think of: • ops, jobs, graphs are code • sensors and schedules are config this is not strictly true, as sensor generators and partitioned schedules can be pretty sophisticated. but i find myself focusing on building out capabilities within the ops/jobs/graphs, then trying to put variations of it into sensors/schedules. all of that to say that your pattern makes sense to me -- the value of
config_schema
is in controlling the possible variations of an op/graph by downstream scehdules/sensors
y
@Cezary Pukownik @Stephen Bailey thanks for the writeup! I think the confusion you flagged is real and is something we’re discussing internally - I created a github discussion here to collect further feedback you may have.