Alex Service
01/03/2022, 9:58 PMPath
object works pretty well# config.yaml
resources:
storage:
config:
local: data/dag
and in my pipeline code:
cached_path = Path(context.resources.storage['local']) / 'cache' / 'cached_raw.snappy.parquet'
io_manager
a little more intelligently, so there are probably better ways 🙂Thomas Mignon
01/03/2022, 10:03 PMops:
purify:
command_param:
'inputpath=/home/datawork/ 20200101'
partition:
command_param: 'inputpath=/home/datawork/inputpath=/home/datawork/ 20200101'
to something like that
date: 20200101
ops:
purify:
command_param: 'inputpath=/home/datawork/ *date*'
partition:
command_param: 'inputpath=/home/datawork/inputpath=/home/datawork/ *date*'