How do I use a config mapping for an asset based job but als dagster #ask-community

How do I use a config_mapping for an asset-based j...

Eric Loreaux

04/13/2023, 3:39 AM

How do I use a config_mapping for an asset-based job, but also create multiple versions of that job where the values provided to that mapping are different? It feels like you can only do one or the other. You can either pass a different fixed dictionary as the

config

arg for each

define_asset_job

call, or you can pass a config_mapping to this arg, but that config_mapping is only associated with a single set of default values and cannot be easily tweaked for multiple jobs

Eric Loreaux

04/13/2023, 3:48 AM

To make this concrete, I have 2 assets that both take in a

price

parameter:

Copy code

@asset(config_schema={
    "price": float
})
def asset1(context):
    return context.op_config["price"] * 2

@asset(config_schema={
    "price": float
})
def asset2(context):
    return context.op_config["price"] / 2

And I want to have 2 asset-based jobs, each of which has a different default price configured. The first approach would look like this:

Copy code

job1 = define_asset_job(
    name="job1",
    config={"ops": {
        "asset1": {"config": {"price": 1}},
        "asset2": {"config": {"price": 1}},
    }}
)

job2 = define_asset_job(
    name="job2",
    config={"ops": {
        "asset1": {"config": {"price": 2}},
        "asset2": {"config": {"price": 2}},
    }}
)

It's fine, but it's clunky. The same number must be repeated for both assets, which is also clunky for users who want to overwrite values in launchpad. Instead, I might define a simple config_mapping which only exposes a single price parameter:

Copy code

@config_mapping(config_schema={
    "price": float
})
def shared_config(val):
    return {
        "ops": {
            "asset1": {"config": {"price": val["price"]}},
            "asset2": {"config": {"price": val["price"]}},
        }
    }

And I could put this into a job:

Copy code

job1 = define_asset_job(
    name="job1",
    config=shared_config
)

But it's not clear to me how I would set different default values for

job1

and

job2

using this config_mapping

23 Views

Open in Slack

Previous Next