https://dagster.io/ logo
c

Chris Roth

04/28/2020, 11:48 PM
question. a lot of my solids use the same set of configuration params. is it possible to have pipeline-level config params instead of having to re-define these configs for each solid or pass them from one solid to the next?
s

schrockn

04/29/2020, 12:57 AM
The most straightforward way to do this would be to utilize the resource system
Just have the "resource" in this case be the config dictionary itself
c

Chris Roth

04/29/2020, 1:05 AM
hmm the problem is that i am using the same config for every solid in my pipeline, but that this config is different for every partition
m

max

04/29/2020, 1:09 AM
if it's just cumbersome to have repeated blocks in the config for each solid definition, you could just assign the block to a variable and reuse it across your definitions
if it's cumbersome to have the repetition in your environment dict, your resource config could be produced as part of your partition
c

Chris Roth

04/29/2020, 1:10 AM
ooo that's a good point. nice.
m

max

04/29/2020, 1:10 AM
PartitionSetDefinition
takes an argument,
user_defined_environment_dict_fn_for_partition
which can generate resource config
or indeed config for your solids of whatever kind
c

Chris Roth

04/29/2020, 1:11 AM
yes! using that for something already so very easy to add it. good call. thank you
👌 1
m

max

04/29/2020, 1:11 AM
if it's easy to compute the config values from some single key, you could also have a
compute_important_values()
fn or equiv on the resource, so that the solid logic calls
resource.compute_important_values(partitioned_key)
or equiv