So i have some text template files that are used in our pipeline and occasionally get updated. Should I:
1. bake them into the git repo and make sofware @assets out of them (read in text pass it down)?
2. OR should i be loading them in from an external source like S3?
3. OR should i just read them in from the local file system and not make them assets.
I'm not anticipating rerunning anything when they change but would be good to track which version is used with each run.
dagster bot responded by community 1
or should i use resources for this?
07/19/2022, 8:38 AM
from my understanding you can achive same result either ways, it depends whats works for you best.
I use resources for things that is want to configure once in pipeline like 'source_folder' or 'api_client', then use them in my ops/graphs.
If you want to track source file version maybe you sould update this files as assets when its get updated. Then pass that assets t jobs that uses its as source file.
to handle saving and writing you could use io_managers, but I use resources to handle this is ops to load and save using coonfigured resource.
07/19/2022, 10:29 PM
thanks man, that really helpful. I'll look into resources