https://dagster.io/ logo
#announcements
Title
# announcements
m

Manas Jain

09/18/2020, 8:04 AM
Hi Team, I need to pass a dictionary throughout the solids of a pipeline to capture certain parameters in each solid... is there a better way of doing it apart from using output definitions to output it from a solid and again passing it as input to the next one?
s

sashank

09/18/2020, 1:47 PM
Could you explain what you mean by “capture certain parameters in each solid”? Do you mean that you have to build up a dictionary over several solids?
m

Manas Jain

09/19/2020, 10:10 AM
@sashank, Yes so I am capturing the inputs passed on to each solid, the start time and end time for each solid, some other query execution data, success/failure, etc. Currently I have to return this dictionary out from each solid and pass it as input in the next solid... Is there a better way? May be if I can use context somehow for doing the same?
s

sashank

09/19/2020, 6:20 PM
One thing to realize is that you’ll need to potentially use a persistent data store (like a filesystem or a DB) to write data to – depending on which executor you’re using the solids may not be running in the same process or even the same machine.
cc @alex @max do you have any ideas here? I wonder if this is something we should provide
capturing the inputs passed on to each solid, the start time and end time for each solid, some other query execution data, success/failure, etc
It also might help to understand what your overall goal here is. Dagster already most of these things, and you could use Assets or Materializations to track the query execution data: https://docs.dagster.io/overview/asset-materializations
Also if you need to do things conditionally on success/failure, you might find some value in hooks: https://docs.dagster.io/examples/hooks
a

alex

09/21/2020, 2:51 PM
I would recommend a using a
resource
- then the implementation can be as complex as your execution substrate requires. If you are just doing in process execution you can just use an in memory dictionary, but if its distributed you will need to store things externally