https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Mhd Mousa Hamad

03/06/2023, 11:26 AM
Hi all, We are building multiple jobs based on Software-Defined Assets. The jobs are generated once per client where the clients are fetched through an endpoint. The graph is the same for all clients but the configuration might differ. 1. Next to the data flowing through the graph, we would like to pass some context which might be extended in each op. I know we can use multiple outs and ins for the assets, but I was thinking if there is a key-value bag in some already existing context which we can use out of the box? 2. Some information like client id and similar information is fixed per job, what is the best approach to pass this information through all assets. I found this comment recommending using resources, but I don't think this is the best approach for our case as this will lead to a resource per client. AFAIK, the Configured API is the best fit for our need but it doesn't seem to support Software-Defined Assets. Any recommendations? Thank you!
c

chris

03/06/2023, 9:47 PM
you can provide config to resources, so you can define a resource, use it with every asset, and then configure that resource separately for each client_id.
Regarding a kvs store, we do have something like that used internally within the system, but wouldn’t recommend extending that. Instead i would recommend using a resource to write in / out of as well - maybe some sort of simple cloud kv store that you could use for retrieval
m

Mhd Mousa Hamad

03/07/2023, 9:34 AM
Thank you Chris for your reply. To understand this better, would you actually recommend using a resource for this kind of information, and how this differs from using run configuration?
Assuming you are gonna support AssetDefinitions with the ConfiguredAPI, would you recommend switching to that approach? Thank you!