https://dagster.io/ logo
Title
a

Alex Service

01/03/2022, 9:58 PM
@Thomas Mignon Given it’s a yaml file, I don’t think yaml properly supports string concatenation, weirdly enough, but I’ve found that defining a root path in a resource and then using a python
Path
object works pretty well
e.g.
# config.yaml
resources:
  storage:
    config:
      local: data/dag
and in my pipeline code:
cached_path = Path(context.resources.storage['local']) / 'cache' / 'cached_raw.snappy.parquet'
:dag-eyes: 1
This also plays nicely when switching between environments (local resources vs cloud), though I’m just starting to explore using the
io_manager
a little more intelligently, so there are probably better ways 🙂
t

Thomas Mignon

01/03/2022, 10:03 PM
Seems a nice solution but unfortunately i want to use the var i define in the Yaml inside the same yaml not from the python code
But I have the case of Path, but i have i also have the case of dates which i'm repeating a lot of times in my yaml, and if i could foung a solution to variabilize the date it would be much better
From actually this
ops:
   purify:
      command_param:
'inputpath=/home/datawork/ 20200101'
   partition:
      command_param: 'inputpath=/home/datawork/inputpath=/home/datawork/ 20200101'
to something like that
date: 20200101
ops:
   purify:
      command_param: 'inputpath=/home/datawork/ *date*'
   partition:
      command_param: 'inputpath=/home/datawork/inputpath=/home/datawork/ *date*'