Hello everyone. I have a use case that has mixed assets and non-asset computations and I'm having trouble understanding now to conceptualize it. If anyone has time I would appreciate some ideas. This is the short story: Get list of data orders and order info from databases, for each order download files from storage archive, execute user defined processing which can include, sub-setting, conversion between data formats, calculating and formatting quality control info, among others. I've implemented some of this using jobs and ops and such but all my jobs end up as single nodes in the graph and I can't see any execution graph or dependencies. Please forgive my ignorance if this is an easy task. Data pipe lining with a tool such as this isn't in my usual wheelhouse.

Hi! Seems like you might have already found the info you need here, but it sounds like this is a case for graph-backed assets. You can have a graph of op that all work together to produce a single (or multiple) assets. https://docs.dagster.io/concepts/assets/graph-backed-assets#graph-backed-assets

Thank you so much Owen. I really love dagster and I think it'll revolutionize our data processing and delivery system. I just have to wrap my head around the way it works. what you sent is almost exactly what I need. On another note, is there a way to have optional ops or skip one if it fails and continue a graph?

glad to hear it! I think the conditional branching feature sounds similar to what you're looking for, but maybe not exactly (if the op with an optional output doesn't emit an output, anything downstream of that op will not execute) in terms of op failure, any time an op fails that will result in the entire job failing, but to get around that limitation, one pattern would be to wrap the command you expect to fail in a try/except block

Thank you so much for your help. Do you know if there are any more examples of complex workflows? I've gone through all the dagster project examples and found some on GitHub with the dagster tag but I haven't found something that does quite what I'm looking for.

no problem! and unfortunately I'm not aware of any examples outside of the ones on github -- but this channel is a great place to help brainstorm if you have a specific thing you're trying to accomplish

Thank you so much. I really appreciate your help. I'll play around with dagster a bit more then ask if I still can't figure it out.