https://dagster.io/ logo
#announcements
Title
# announcements
m

Michael T

02/11/2021, 9:38 PM
Right - but that ends up constructing a parallel hierarchy in a way, of “solid”-ified functions.
a

antonl

02/11/2021, 9:38 PM
Yes, but I think that’s unavoidable.
During development though, I use
execute_solid
a lot.
m

Michael T

02/11/2021, 9:41 PM
There is something that I had done a few years ago in Julia that was kind of magic. Basically modeled after taint analysis - so that an operation on something in the dag would itself be on the dag, without needing to hard-code something.
It was kind of cool.
By I, I mean someone who worked for me. He did the magic.
Another way of thinking about it was the when run in the context of a dag, your results were evaluated on the dag, with memoization at nodes, etc. But it could also be wired up to a forward streaming set of data - same code, different context.
a

antonl

02/11/2021, 9:43 PM
I tried building something like this by quoting every object with a proxy and recording the graph that way. It works, but gets pretty tricky.
Also, it’s not clear how to handle optional inputs in this way.
The benefits are that you can map over outputs natively. This is basically how autograd works.
erm, that is you can support
__iter___
; I think data-dependent mapping is still out of the question. You can’t map over a quoted dict for example.
m

Michael T

02/11/2021, 9:45 PM
What do you mean by “quoting every object”? And yes, autograd is part of the thoughts. In 2015, I was trying to embed automatic differentiation at the core of our analytics 😉
a

antonl

02/11/2021, 9:48 PM
Term of art in language design. You need to distinguish between a value and the name of that value. Something like lvalue vs rvalue in C++. Dagster defines a
SolidOutputHandle
for each output. What you could do is to overload this handle to support all python operations and generate implicit graph nodes on the fly.
I did this by writing a Quote class that wraps a real value, and then operations on this Quoted object would be recorded on the graph.
An example is Dask’s delayed decorator https://docs.dask.org/en/latest/delayed.html
Re-reading this, I realize I’m not explaining myself very well. There’s a simpler explanation. In autograd/autodiff, one way to implement is to define a “box” type. It holds the result of computations (value), but also a link to the gradient function computation. Every function you call on this box type creates more boxed types. When you call “backward” on the final result, the framework walks back through the dag you’ve created using these boxed handles and gives you the gradient wrt the inputs. In pytorch, this “boxed” type was called Variable. It would wrap a Tensor to allow autograd to work. (It’s been merged with Tensor in recent editions).
These task graphs are a similar concept, except you never need to go backward, but you do need to define the graph somehow. By “boxing”/quoting you can do that in python instead of using some DSL by just calling functions.
m

Michael T

02/11/2021, 10:01 PM
Yes. I figured you meant this. And you have hit on the libraries I also was hoping could be leveraged.
s

sandy

02/11/2021, 10:16 PM
@Michael T @antonl have you looked at jax? https://github.com/google/jax
a

antonl

02/11/2021, 10:32 PM
Yes, it’s pretty nice. They use the same autograd mindset to do more general optimizations. That’s how they implement
vmap
. I really wish theano wasn’t dead.
3 Views