https://dagster.io/ logo
#ask-community
Title
# ask-community
e

Eric Loreaux

03/09/2023, 11:51 PM
Sorry if this is naive, but can multiple ops share the same config parameters? The dagit launchpad design seems to imply that config parameters must be supplied as values where a single unique op is the key.
s

Spencer Nelson

03/09/2023, 11:54 PM
You might be looking for ConfigMappings https://docs.dagster.io/concepts/ops-jobs-graphs/jobs#config-mapping which let you write a function for this
e

Eric Loreaux

03/09/2023, 11:56 PM
Oh, that's cool! Would the config values still need to be duplicated if entering them via launchpad?
c

chris

03/09/2023, 11:56 PM
^ - but yes in general each op is configured separately. One thing you can do is define a function which returns a config schema, and use it for multiple ops
If using config mapping, config values only need to be provided once
e

Eric Loreaux

03/10/2023, 12:03 AM
So let's say I had two ops which took in the same param:
Copy code
@op(config_schema={"config_param": str})
def op1(context):
  ...

@op(config_schema={"config_param": str})
def op2(context):
  ...
With the naive approach, the launchpad configuration for these ops would look like:
Copy code
ops:
  op1:
    config:
      config_param: X
  op2:
    config:
      config_param: X
But now with the new approach, I use config mapping:
Copy code
@config_mapping(config_schema={"config_param": str})
def distribute_config(val):
  return {
    "ops": {
      "op1": {
        "config": {"config_param": val["config_param"]}
      },
      "op2": {
        "config": {"config_param": val["config_param"]}
      },
    }
  }
But how would this look in launchpad?
seems like it can only be used if you're calling
execute_in_process
c

chris

03/10/2023, 12:35 AM
Can be provided to the
config
argument of the job
In the launchpad it would look like this:
Copy code
config_param: "blah"
that’s it
e

Eric Loreaux

03/10/2023, 12:37 AM
Oh so job config values are just key-value pairs at the highest level of the launchpad config? Like
Copy code
config_param: "blah"
ops:
 ...
...
s

Spencer Nelson

03/10/2023, 6:59 PM
@Eric Loreaux The job’s configmapping will just “take over” so it’s just
config_param: "blah"
. You can’t also override the ops, no
ops: …
lines at all.
c

chris

03/10/2023, 7:17 PM
^ exactly what spencer said. Think of it as a function that takes an outer config schema, and produces the full job’s config from that
e

Eric Loreaux

03/12/2023, 8:22 AM
It sounds like the config mapping was exactly what I was looking for! With all your help, my team can now use a much simpler interface. Thanks everyone
One follow up question - can configmappings still be assigned default values?
c

chris

03/13/2023, 12:34 PM
Yea you can specify defaults in the outer config schema:
outer_config = {"foo": Field(str, default_value="bar")}
for example
e

Eric Loreaux

04/13/2023, 4:32 AM
To follow up on this thread, I'm not seeing the default values show up in launchpad when I specify them. Here is a very simple example I put together which demonstrates the problem:
Copy code
@asset(config_schema={
    "price": float
})
def asset1(context):
    return context.op_config["price"] * 2

@asset(config_schema={
    "price": float
})
def asset2(context):
    return context.op_config["price"] / 2

@config_mapping(config_schema={
    "price": Field(
        config=float,
        default_value=2
    )
})
def shared_config(val):
    return {
        "ops": {
            "asset1": {"config": {"price": val["price"]}},
            "asset2": {"config": {"price": val["price"]}},
        }
    }

job1 = define_asset_job(
    name="job1",
    selection=["asset1", "asset2"],
    config=shared_config
)
When I materialize all on dagit, the launchpad is blank and doesn't include the default value of 2 that I configured
c

chris

04/14/2023, 1:35 AM
Hm interesting. I’ll put it on my docket to investigate what’s going on here
🙌 1
e

Eric Loreaux

05/02/2023, 5:51 AM
Just checking in - any chance this has been investigated further? Would love to use
config_mapping
but without being able to populate it with default values, it makes my pipeline less user-friendly
c

chris

05/02/2023, 5:27 PM
blob facepalm got caught up in other stuff - will do now
Yea was able to reproduce this: I think it’s related to https://github.com/dagster-io/dagster/issues/11828. Feel free to upvote, I think this is on our roadmap to get fixed sooner rather than later. I think we ideally want to have a scaffold-default-config button.
👍 1
4 Views