https://dagster.io/ logo
d

Daniel Fernandez

10/22/2020, 6:38 PM
Hi, I am new to Dagster and I have a short Q, what's the back-end that stores each Node information on the DAG (i.e., a node/solid status, metadata about the solid, run status, code that the solid runs, etc.) - I am looking to add some custom information to the nodes/solids and was wondering what'd be the best way to do something like this.
s

sashank

10/22/2020, 6:42 PM
Hey @Daniel Fernandez, what kind of information are you looking to add? If it’s just simple metadata, you can just use solid tags: https://docs.dagster.io/_apidocs/solids#dagster.solid: It’s a argument to the
@solid
decorator
To answer your question about what the backend looks like: we have a few “storages” (such as
RunStorage
and
EventLogStorage
) which are DB tables that store information about past runs and the events they emitted.
d

Daniel Fernandez

10/22/2020, 6:46 PM
Nice. Thanks, yes this is similar to what we'd store but we have some custom requirements for what we currently store about each solid/node and was wondering more of the backend architecture and if we could play a bit with it
currently we are storing everythign in elastic search
for our current pipeline that we want to migrate to dagster
m

max

10/22/2020, 6:48 PM
if it's metadata that is generated by each execution of the solid, you can also yield structured events from the bodies of the solids
d

Daniel Fernandez

10/22/2020, 6:48 PM
mmmm interesting... how would i do that?
There’s also a UI that comes along with this that let’s you easily lookup materializations by key
d

Daniel Fernandez

10/22/2020, 6:51 PM
interesting... this could be what i use. I am a bit afraid of having two places where I do all my storage about the pipeline as ideally it's better to keep it all in one but for migration this can be a really good start
s

sashank

10/22/2020, 6:52 PM
More than happy to spend some time better understanding what exactly what you want to do/what your use case is to come up with a solution that works for you
1