Alex Service
01/03/2022, 9:58 PMPath
object works pretty wellAlex Service
01/03/2022, 10:00 PM# config.yaml
resources:
storage:
config:
local: data/dag
and in my pipeline code:
cached_path = Path(context.resources.storage['local']) / 'cache' / 'cached_raw.snappy.parquet'
Alex Service
01/03/2022, 10:01 PMio_manager
a little more intelligently, so there are probably better ways 🙂Thomas Mignon
01/03/2022, 10:03 PMThomas Mignon
01/03/2022, 10:04 PMThomas Mignon
01/03/2022, 10:05 PMops:
purify:
command_param:
'inputpath=/home/datawork/ 20200101'
partition:
command_param: 'inputpath=/home/datawork/inputpath=/home/datawork/ 20200101'
to something like that
date: 20200101
ops:
purify:
command_param: 'inputpath=/home/datawork/ *date*'
partition:
command_param: 'inputpath=/home/datawork/inputpath=/home/datawork/ *date*'