Anthony Reksoatmodjo
08/12/2022, 12:30 AMdagster.yaml
values from init_context
? My hope is to specify non-run-specific config in dagster.yaml
, and keep it separate from run_config
.claire
08/12/2022, 5:21 PMcontext
object. Is there a reason why you need to read from the context object? Because you could specify non-run-specific config in another yaml file and read the config in within an op or resource (we offer a utility method here that reads config from yaml).Anthony Reksoatmodjo
08/12/2022, 7:19 PMIs there a reason why you need to read from the context object?No specific reason. The problem is that
init_context.instance.storage_directory()
is too small for my outputs (10-100s of GB), so I have a separate /path/to/large/disk
for larger files (logs and stuff still get written to default storage
). However, this path gets defined multiple times in my `run_config`:
"resources": {
"local_temp_dir": {
"config": {
"path": "/path/to/large/disk/temp"
}
},
"my_io_manager": {
"config": {
"base_dir": "/path/to/large/disk"
}
},
},
So I'm looking for a place within Dagster where I can define /path/to/large/disk
once, but have it available during resource
init and ops
We originally had our own yaml-to-config-object parser, but I was hoping there was a more Dagster-esque way of doing things.