Tobias Macey

03/15/2021, 8:55 PM
I'm working on reimplementing a daily uploads folder within a pipeline and figured I should write it as a FileManager. Curious if anyone has gone down that road already? Specifically I create a directory on local disk named from the current date (e.g. 20210315) and then as the last stage of my pipeline I write the contents of the folder to S3. The data that goes into the folder is a few different types of thing. Some of it is CSV files, there are also some output files from mongodump, and there will also be some .tar.gz files. Any thoughts on best practice given the mechanisms that Dagster has available?
👀 1