During development though, I use

execute_solid

a lot.

It was kind of cool.

By I, I mean someone who worked for me. He did the magic.

Another way of thinking about it was the when run in the context of a dag, your results were evaluated on the dag, with memoization at nodes, etc. But it could also be wired up to a forward streaming set of data - same code, different context.

Also, it’s not clear how to handle optional inputs in this way.

The benefits are that you can map over outputs natively. This is basically how autograd works.

erm, that is you can support

__iter___

; I think data-dependent mapping is still out of the question. You can’t map over a quoted dict for example.

I did this by writing a Quote class that wraps a real value, and then operations on this Quoted object would be recorded on the graph.

An example is Dask’s delayed decorator https://docs.dask.org/en/latest/delayed.html

Re-reading this, I realize I’m not explaining myself very well. There’s a simpler explanation. In autograd/autodiff, one way to implement is to define a “box” type. It holds the result of computations (value), but also a link to the gradient function computation. Every function you call on this box type creates more boxed types. When you call “backward” on the final result, the framework walks back through the dag you’ve created using these boxed handles and gives you the gradient wrt the inputs. In pytorch, this “boxed” type was called Variable. It would wrap a Tensor to allow autograd to work. (It’s been merged with Tensor in recent editions).

These task graphs are a similar concept, except you never need to go backward, but you do need to define the graph somehow. By “boxing”/quoting you can do that in python instead of using some DSL by just calling functions.