Jay Sharma
03/31/2022, 6:41 PMdef get_schedules_from_config(schedule_config):
schedule_list = []
for pipeline_name in schedule_config:
schedule = ScheduleDefinition(cron_schedule=schedule_config[pipeline_name], pipeline_name=pipeline_name,
run_config={"resources": [{"file_manager": {"config": {"s3_bucket": "s3-pipeline"}}},
{"io_manager": {"config": {"s3_bucket": "s3-pipeline @"}}}]},
mode='aws',execution_timezone="US/Eastern")
schedule_list.append(schedule)
return schedule_list
I get an error like:
Error 1: Value at path root:resources must be dict. Expected: "{ file_manager: { config: { endpoint_url?: (String | { env: String }) max_attempts?: Int profile_name?: String region_name?: String s3_bucket: (String | { env: String }) s3_prefix?: (String | { env: String }) use_unsigned_session?: Bool } } io_manager: { config: { s3_bucket: (String | { env: String }) s3_prefix?: (String | { env: String }) } } postgres_connection?: { config?: Any } s3?: { config?: { endpoint_url?: (String | { env: String }) max_attempts?: Int profile_name?: String region_name?: String use_unsigned_session?: Bool } } }". (Screenshot #2)
Ok, it is expecting a dictionary, not a list of dictionaries.
However, when I try to pass in the resources like this:
run_config={"resources": {"file_manager": {"config": {"s3_bucket": "s3-pipeline"}}},
"io_manager": {"config": {"s3_bucket": "s3-pipeline"}}}
I get an error like this (Screenshot #3): dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline Error 1: Received unexpected config entry "io_manager" at the root. Expected: "{ execution?:
Thanks for the help.owen
03/31/2022, 6:47 PM{
"resources": {
"file_manager": {
"config": {"s3_bucket": "s3-pipeline"}
},
"io_manager": {
"config": {"s3_bucket": "s3-pipeline"}
},
},
}
(with the config you shared, "io_manager" and "resources" were at the same level)Jay Sharma
04/06/2022, 1:50 PM