Hi guys, I have a resource that performs some post execution DQ checks on our pipeline. THey are simple SQL commands that evaluate to True/False and then raise an Error that we capture and fail ops if they are severe enough. We have recently found that in TEST/PROD we might want to suppress a couple of the non critical checks. Currently I have added this functionality through resource configs and can pass in the list of suppressed checks either through the launchpad or ENV variables. I have been asked to see if there is a way to automate this on specific checks? Perhaps through scheduling params or sensors. There would be a specific subset of checks to suppress. Any guidance would be much appreciated.

Hi @Philip Gunning I'm not 100% sure i fully understand your use case, so please correct me if i'm making any incorrect assumptions. From what I understand, you want to run some of the data quality checks on a schedule, and modify which checks are suppressed. That should be relatively simple as you can set the configuration for your resources in the RunRequests returned by your schedule. This docs example shows setting the run config in the RunRequest returned by a sensor. in your case you would ensure that the resources section of run config is included. Something like

@schedule(job=my_job, cron_schedule="0 0 * * *")
def my_schedule(context):
    return RunRequest(
       run_key=None,
       run_config={"resources": {"foo_resource": {"bar": 1}}}
    )

let me know if that doesn't fully fit what you're trying to do and i can help out some more!