Hi! 👋 Looking to confirm some understanding of a sensor against an SFTP using assets. Sounds like the trend is to move towards asset based features, so given that what would the best approach be to poll for new files in an SFTP? The example of sensors uses

ops

and the dynamic partition example (which polls for images, but the concept tracks) seems to lean towards dynamic partitions for picking up new assets, adding them to the partition then triggering a run. I like the approach of the dynamic assets as it'll give a good view of what file name has been collected and be fairly observable if a particular load fails (or not). Final question here is when the file is removed from the SFTP (at an unknown interval) would you remove the partition, or would you keep that partition as is. Someone could (in theory) try and rematerialize the partition at a future date, which would cause that partition to show as failed. Removing it would avoid that, but it would also remove the visibility of it being loaded historically. Thoughts? Drop partitions, keep partitions, stick with

ops

? Thanks!

Sounds to me like a dynamically partitioned asset is the way to go here.

Removing it would avoid that, but it would also remove the visibility of it being loaded historically.

Removing the partition shouldn’t delete past materialization events from history.

Awesome. I'll give the dynamic partition a go. Appreciate the help sean!