Scott Hood
07/21/2022, 5:52 PMops
impossible. However we would like to not have to set io manager on every given job or need to create a helper around it. Wondering if there is a location in dagster I can set the default IO manager once and then have every job utilize that.Daniel Gafni
07/21/2022, 7:05 PMwith_resources
on a list of assets and set the “io_manager” key thereDaniel Gafni
07/21/2022, 7:06 PMprod_assets = with_resources(
assets,
resource_defs={
"io_manager": s3_pickle_io_manager,
"parquet_io_manager": s3_parquet_io_manager,
"s3": s3_resource,
},
resource_config_by_key={
"io_manager": {
"config": {
"s3_bucket": "",
}
},
"parquet_io_manager": {
"config": {
"s3_bucket": "",
}
},
"s3": {"config": {"endpoint_url": "", "region_name": ""}},
},
)
Daniel Gafni
07/21/2022, 7:07 PMScott Hood
07/21/2022, 7:07 PM