question. a lot of my solids use the same set of c...
# announcements
c
question. a lot of my solids use the same set of configuration params. is it possible to have pipeline-level config params instead of having to re-define these configs for each solid or pass them from one solid to the next?
s
The most straightforward way to do this would be to utilize the resource system
Just have the "resource" in this case be the config dictionary itself
c
hmm the problem is that i am using the same config for every solid in my pipeline, but that this config is different for every partition
m
if it's just cumbersome to have repeated blocks in the config for each solid definition, you could just assign the block to a variable and reuse it across your definitions
if it's cumbersome to have the repetition in your environment dict, your resource config could be produced as part of your partition
c
ooo that's a good point. nice.
m
PartitionSetDefinition
takes an argument,
user_defined_environment_dict_fn_for_partition
which can generate resource config
or indeed config for your solids of whatever kind
c
yes! using that for something already so very easy to add it. good call. thank you
👌 1
m
if it's easy to compute the config values from some single key, you could also have a
compute_important_values()
fn or equiv on the resource, so that the solid logic calls
resource.compute_important_values(partitioned_key)
or equiv