https://dagster.io/ logo
#ask-community
Title
# ask-community
e

Emmanuel Ogunwede

06/05/2023, 1:45 PM
Hi! Can I use an asset to represent a folder/directory such that every time the asset gets materialized the contents of the folder get refreshed? For context, I have an op that performs an action based on a YAML file. The content of the YAML is generated/managed/updated by another system. Every time there's a change, the YAML file is uploaded to an s3 bucket to overwrite the previous file in the s3 bucket. I want to always have the latest version of this file available in a path/directory accessible to Dagster and was wondering if I can use an Asset and a sensor to achieve the file sync or if this is something meant specifically for an OP? Thanks in advance for sharing your thoughts!
s

sandy

06/05/2023, 7:04 PM
Do you drop and recreate the entire folder each time? Or do you operate on individual files within the folder? Are there multiple YAML files?
e

Emmanuel Ogunwede

06/05/2023, 7:06 PM
The entire folder...I dumbed down my context. I actually need to unzip an entire folder into a path Dagster can access
The yaml file allows me to install a package (a Meltano tap)
s

sandy

06/05/2023, 7:08 PM
here's an example that unzips an entire folder: https://docs.dagster.io/tutorial/managing-your-own-io#tutorial-part-7-managing-your-own-io. would this work for you?
e

Emmanuel Ogunwede

06/05/2023, 7:11 PM
Sweet! Thanks, Sandy!
I'll read through it and let you know!
@sandy Yea, this should work! - The asset will be responsible for making the latest changes available then, I have a separate Job that expects the directory (and its content) to always be available and up to date; the Job uses dagster_meltano to install and execute meltano jobs. Instead of materializing the asset every time the Job has to run, I'll want to use a sensor to keep my asset up to date and only materialize it when the zip file has been updated/overwritten in s3. Do you think this pattern works or should I consider using a different approach? I'm looking to use a sensor, an Asset, and a job with a few ops. any thoughts?
s

sandy

06/05/2023, 10:28 PM
Using a sensor to detect when the zip file has been overwritten and trigger the asset sounds like a good idea to me
e

Emmanuel Ogunwede

06/06/2023, 8:05 AM
Thank you 🙏🏻
2 Views