I want to make it look like a particular plugin based op is dagster #ask-community

I want to make it look like a particular plugin-ba...

Josh Lloyd

09/05/2023, 8:30 PM

I want to make it look like a particular plugin-based op is producing a set of dbt source assets (because it is producing them). But this obviously doesn’t work:

Copy code

@multi_asset(
    outs={
        name: AssetOut(key=asset_key)
        for name, asset_key in get_asset_keys_by_output_name_for_source(
            [edp_dbt_assets], "gainsightpx"
        ).items()
    }
)
def gainsightpx(context):

    meltano_run_op(f"--force tap-gainsightpx target-snowflake")()

    output_names = list(context.selected_output_names)
    for output_name in output_names:
        yield Output(value=..., output_name=output_name)

Where do I go from here?

jamie

09/05/2023, 9:13 PM

what do you mean by “obviously doesnt work”? are you seeing errors? the UI doesn’t look like you want? something else?

Josh Lloyd

09/05/2023, 9:24 PM

well, you can’t run an op with an asset/multi_asset definition

Josh Lloyd

09/05/2023, 9:27 PM

I’m starting to get somewhere with

@graph_multi_asset

as opposed to

@multi_asset

but it still won’t run. It will at least get me to the UI, but when I try to materialize them I get:

Copy code

dagster._core.errors.DagsterInvalidSubsetError: When building job, the AssetsDefinition 'gainsightpx' contains asset keys [AssetKey(['raw_gainsightpx', 'accounts']), AssetKey(['raw_gainsightpx', 'custom_events']), ... ], but attempted to select only [AssetKey(['raw_gainsightpx', 'users'])]. This AssetsDefinition does not support subsetting. Please select all asset keys produced by this asset.

Josh Lloyd

09/05/2023, 9:28 PM

That’s the error with the latest iteration of code which is:

Copy code

@graph_multi_asset(
    resource_defs={"meltano": meltano_resource},
    outs={
        name: AssetOut(key=asset_key)
        for name, asset_key in get_asset_keys_by_output_name_for_source(
            [edp_dbt_assets], "raw_gainsightpx"
        ).items()
    }
)
def gainsightpx():

    result = meltano_run_op(f"--force tap-gainsightpx target-snowflake")()

    asset_keys = get_asset_keys_by_output_name_for_source([edp_dbt_assets], "raw_gainsightpx")
    outs = {}
    for asset_key in asset_keys:
        outs[asset_key] = result
    return outs

Josh Lloyd

09/05/2023, 9:29 PM

Screenshot 2023-09-05 at 3.29.27 PM.png

claire

09/08/2023, 7:07 PM

When executing graph-backed assets and multi-assets, by default, every output asset must be selected for execution. This is because in these cases multiple assets are frequently outputted from the same step. In order to enable subsetting, you'll have to define your ops to yield conditional outputs based on the assets selected for execution: • for graph backed assets: https://docs.dagster.io/concepts/assets/graph-backed-assets#advanced-subsetting-graph-backed-assets • for multi assets: https://docs.dagster.io/concepts/assets/multi-assets#subsetting-multi-assets

Open in Slack

Previous Next