Daniel Galea12/13/2022, 1:37 PM
in a different way other than
? I would like to write my files to S3 in the following manner
s3://<bucket>/dagster/storage/<job run id>/<op name>.compute
. I guess that the <job run id> is used so that re-running the same Run over and over again will allow any downstream tasks to process the exact same data. This is similar to what I want but year/month_day would be a bit more human readible than a run ID. I am processing my data on EMR and I don't want to couple my Spark code to Dagster. Therefore, a year/month/day partition style would allow Spark to read data independent of Dagster.
jamie12/13/2022, 5:47 PM
and change the logic that determines the path
Daniel Galea12/14/2022, 8:15 AM