Hello everyone. I have a use case that has mixed assets and non-asset computations and I'm having trouble understanding now to conceptualize it. If anyone has time I would appreciate some ideas. This is the short story: Get list of data orders and order info from databases, for each order download files from storage archive, execute user defined processing which can include, sub-setting, conversion between data formats, calculating and formatting quality control info, among others.
I've implemented some of this using jobs and ops and such but all my jobs end up as single nodes in the graph and I can't see any execution graph or dependencies.
Please forgive my ignorance if this is an easy task. Data pipe lining with a tool such as this isn't in my usual wheelhouse.