This is about how to structure my code: I’m taking data from an API that export XML. The API has (let’s say) 10 endpoint, I want to extract each endpoint, convert the data to json, and load the data as text files on my local harddrive. The way I understand it: I have to create 10 extract

assets

for each endpoint, and 10 load

assets

for each file that should be exported to my harddrive. Is that correct? What do I do with the transform from xml to json? As I understand it, this is an asset. But the code is the exact same for each endpoint. Should I duplicate code and also create 10 transform

assets

?

Maybe you can create a resource to wrap the API, define the extract and transformation logic there and use the resource in your assets. Also, if you don’t need the intermediary xml files just the transformed json, maybe you don’t even have to create an asset for them. Perhaps you can even put the whole thing under a

@multi_asset

which produces 10 assets, one for each endpoint, and use the logic from the resource to return the json files.

Thanks @Andras Somi 🙏 I will read up on

resources

and

multi-asset

. You wrote: if you don’t need the intermediary xml files just the transformed json, maybe you don’t even have to create an asset for them How could I implement that? Do you mean having a the transformation part within either the extract or load

asset

?

Yes, that's what I meant. For a similar task I recently put the xml->json parsing logic to the resource itself, because I thought it's inherently connected to the wrapped API anyway.