I've got maybe a case where treating observable source assets as a multi-asset would be useful. I don't think this is actually doable right now, though, so I'm just going to materialize the remote assets rather than observe them.
02/13/2023, 5:50 PM
Hey Chris - would you mind expanding on this a little bit? Is the idea that you'd like to be able to absorb a set of assets with a single function call?
cc @schrockn and @sean, who have been thinking about related topics recently
02/13/2023, 6:38 PM
There's a list of 51 files on a remote server. The file names change when the files do but not every file gets updated all the time. I'd like to figure out whether they've changed before downloading and extracting them, which happens in a single directory list operation. As I've been working through this, I've been treating the list as one asset and then consuming that with a partitioned asset, but if I model them as a multi-asset I can maybe track staleness.
This seems like a good use case for observable source assets, but I don't see any way to combine those with multi-asset decorators save maybe doing something fancy with ops, asset observations and explicitly initialized source assets.
So what I'm hoping is that just using multi-asset functionality and materializing the assets (actually pointers to the assets) will get me most of the way there.
02/15/2023, 12:56 AM
Got it - would it ideally make sense to track it as a partitioned asset and version each of the partitions separately? That's something we'd like to support soon