Hi, I am new to Dagster and I have a short Q, what...
# announcements
Hi, I am new to Dagster and I have a short Q, what's the back-end that stores each Node information on the DAG (i.e., a node/solid status, metadata about the solid, run status, code that the solid runs, etc.) - I am looking to add some custom information to the nodes/solids and was wondering what'd be the best way to do something like this.
Hey @Daniel Fernandez, what kind of information are you looking to add? If it’s just simple metadata, you can just use solid tags: https://docs.dagster.io/_apidocs/solids#dagster.solid: It’s a argument to the
To answer your question about what the backend looks like: we have a few “storages” (such as
) which are DB tables that store information about past runs and the events they emitted.
Nice. Thanks, yes this is similar to what we'd store but we have some custom requirements for what we currently store about each solid/node and was wondering more of the backend architecture and if we could play a bit with it
currently we are storing everythign in elastic search
for our current pipeline that we want to migrate to dagster
if it's metadata that is generated by each execution of the solid, you can also yield structured events from the bodies of the solids
mmmm interesting... how would i do that?
There’s also a UI that comes along with this that let’s you easily lookup materializations by key
interesting... this could be what i use. I am a bit afraid of having two places where I do all my storage about the pipeline as ideally it's better to keep it all in one but for migration this can be a really good start
More than happy to spend some time better understanding what exactly what you want to do/what your use case is to come up with a solution that works for you