https://dagster.io/ logo
Title
e

Eric Loreaux

04/13/2023, 3:39 AM
How do I use a config_mapping for an asset-based job, but also create multiple versions of that job where the values provided to that mapping are different? It feels like you can only do one or the other. You can either pass a different fixed dictionary as the
config
arg for each
define_asset_job
call, or you can pass a config_mapping to this arg, but that config_mapping is only associated with a single set of default values and cannot be easily tweaked for multiple jobs
To make this concrete, I have 2 assets that both take in a
price
parameter:
@asset(config_schema={
    "price": float
})
def asset1(context):
    return context.op_config["price"] * 2

@asset(config_schema={
    "price": float
})
def asset2(context):
    return context.op_config["price"] / 2
And I want to have 2 asset-based jobs, each of which has a different default price configured. The first approach would look like this:
job1 = define_asset_job(
    name="job1",
    config={"ops": {
        "asset1": {"config": {"price": 1}},
        "asset2": {"config": {"price": 1}},
    }}
)

job2 = define_asset_job(
    name="job2",
    config={"ops": {
        "asset1": {"config": {"price": 2}},
        "asset2": {"config": {"price": 2}},
    }}
)
It's fine, but it's clunky. The same number must be repeated for both assets, which is also clunky for users who want to overwrite values in launchpad. Instead, I might define a simple config_mapping which only exposes a single price parameter:
@config_mapping(config_schema={
    "price": float
})
def shared_config(val):
    return {
        "ops": {
            "asset1": {"config": {"price": val["price"]}},
            "asset2": {"config": {"price": val["price"]}},
        }
    }
And I could put this into a job:
job1 = define_asset_job(
    name="job1",
    config=shared_config
)
But it's not clear to me how I would set different default values for
job1
and
job2
using this config_mapping