19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
m
Manas Jain
09/18/2020, 8:04 AMHi Team,
I need to pass a dictionary throughout the solids of a pipeline to capture certain parameters in each solid... is there a better way of doing it apart from using output definitions to output it from a solid and again passing it as input to the next one?
s
sashank
09/18/2020, 1:47 PMCould you explain what you mean by “capture certain parameters in each solid”? Do you mean that you have to build up a dictionary over several solids?
m
Manas Jain
09/19/2020, 10:10 AM@sashank, Yes so I am capturing the inputs passed on to each solid, the start time and end time for each solid, some other query execution data, success/failure, etc. Currently I have to return this dictionary out from each solid and pass it as input in the next solid... Is there a better way? May be if I can use context somehow for doing the same?
s
sashank
09/19/2020, 6:20 PMYou could write a resource for this: https://docs.dagster.io/overview/modes-resources-presets/modes-resources#defining-resources
One thing to realize is that you’ll need to potentially use a persistent data store (like a filesystem or a DB) to write data to – depending on which executor you’re using the solids may not be running in the same process or even the same machine.
cc @alex @max do you have any ideas here? I wonder if this is something we should provide
capturing the inputs passed on to each solid, the start time and end time for each solid, some other query execution data, success/failure, etcIt also might help to understand what your overall goal here is. Dagster already most of these things, and you could use Assets or Materializations to track the query execution data: https://docs.dagster.io/overview/asset-materializations
Also if you need to do things conditionally on success/failure, you might find some value in hooks: https://docs.dagster.io/examples/hooks
a
alex
09/21/2020, 2:51 PMI would recommend a using a
resource