Sorry if this is naive, but can multiple ops share the same config parameters? The dagit launchpad design seems to imply that config parameters must be supplied as values where a single unique op is the key.

You might be looking for ConfigMappings https://docs.dagster.io/concepts/ops-jobs-graphs/jobs#config-mapping which let you write a function for this

Oh, that's cool! Would the config values still need to be duplicated if entering them via launchpad?

^ - but yes in general each op is configured separately. One thing you can do is define a function which returns a config schema, and use it for multiple ops

So let's say I had two ops which took in the same param:

@op(config_schema={"config_param": str})
def op1(context):
  ...

@op(config_schema={"config_param": str})
def op2(context):
  ...

With the naive approach, the launchpad configuration for these ops would look like:

ops:
  op1:
    config:
      config_param: X
  op2:
    config:
      config_param: X

But now with the new approach, I use config mapping:

@config_mapping(config_schema={"config_param": str})
def distribute_config(val):
  return {
    "ops": {
      "op1": {
        "config": {"config_param": val["config_param"]}
      },
      "op2": {
        "config": {"config_param": val["config_param"]}
      },
    }
  }

But how would this look in launchpad?

Can be provided to the

config

argument of the job

Oh so job config values are just key-value pairs at the highest level of the launchpad config? Like

config_param: "blah"
ops:
 ...
...

@Eric Loreaux The job’s configmapping will just “take over” so it’s just

config_param: "blah"

. You can’t also override the ops, no

ops: …

lines at all.

^ exactly what spencer said. Think of it as a function that takes an outer config schema, and produces the full job’s config from that

It sounds like the config mapping was exactly what I was looking for! With all your help, my team can now use a much simpler interface. Thanks everyone

Yea you can specify defaults in the outer config schema:

outer_config = {"foo": Field(str, default_value="bar")}

for example

To follow up on this thread, I'm not seeing the default values show up in launchpad when I specify them. Here is a very simple example I put together which demonstrates the problem:

@asset(config_schema={
    "price": float
})
def asset1(context):
    return context.op_config["price"] * 2

@asset(config_schema={
    "price": float
})
def asset2(context):
    return context.op_config["price"] / 2

@config_mapping(config_schema={
    "price": Field(
        config=float,
        default_value=2
    )
})
def shared_config(val):
    return {
        "ops": {
            "asset1": {"config": {"price": val["price"]}},
            "asset2": {"config": {"price": val["price"]}},
        }
    }

job1 = define_asset_job(
    name="job1",
    selection=["asset1", "asset2"],
    config=shared_config
)

When I materialize all on dagit, the launchpad is blank and doesn't include the default value of 2 that I configured

Hm interesting. I’ll put it on my docket to investigate what’s going on here

Just checking in - any chance this has been investigated further? Would love to use

config_mapping

but without being able to populate it with default values, it makes my pipeline less user-friendly

blob facepalm got caught up in other stuff - will do now