https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Josh Lloyd

09/05/2023, 8:30 PM
I want to make it look like a particular plugin-based op is producing a set of dbt source assets (because it is producing them). But this obviously doesn’t work:
Copy code
@multi_asset(
    outs={
        name: AssetOut(key=asset_key)
        for name, asset_key in get_asset_keys_by_output_name_for_source(
            [edp_dbt_assets], "gainsightpx"
        ).items()
    }
)
def gainsightpx(context):

    meltano_run_op(f"--force tap-gainsightpx target-snowflake")()

    output_names = list(context.selected_output_names)
    for output_name in output_names:
        yield Output(value=..., output_name=output_name)
Where do I go from here?
j

jamie

09/05/2023, 9:13 PM
what do you mean by “obviously doesnt work”? are you seeing errors? the UI doesn’t look like you want? something else?
j

Josh Lloyd

09/05/2023, 9:24 PM
well, you can’t run an op with an asset/multi_asset definition
I’m starting to get somewhere with
@graph_multi_asset
as opposed to
@multi_asset
but it still won’t run. It will at least get me to the UI, but when I try to materialize them I get:
Copy code
dagster._core.errors.DagsterInvalidSubsetError: When building job, the AssetsDefinition 'gainsightpx' contains asset keys [AssetKey(['raw_gainsightpx', 'accounts']), AssetKey(['raw_gainsightpx', 'custom_events']), ... ], but attempted to select only [AssetKey(['raw_gainsightpx', 'users'])]. This AssetsDefinition does not support subsetting. Please select all asset keys produced by this asset.
That’s the error with the latest iteration of code which is:
Copy code
@graph_multi_asset(
    resource_defs={"meltano": meltano_resource},
    outs={
        name: AssetOut(key=asset_key)
        for name, asset_key in get_asset_keys_by_output_name_for_source(
            [edp_dbt_assets], "raw_gainsightpx"
        ).items()
    }
)
def gainsightpx():

    result = meltano_run_op(f"--force tap-gainsightpx target-snowflake")()

    asset_keys = get_asset_keys_by_output_name_for_source([edp_dbt_assets], "raw_gainsightpx")
    outs = {}
    for asset_key in asset_keys:
        outs[asset_key] = result
    return outs
Screenshot 2023-09-05 at 3.29.27 PM.png
c

claire

09/08/2023, 7:07 PM
When executing graph-backed assets and multi-assets, by default, every output asset must be selected for execution. This is because in these cases multiple assets are frequently outputted from the same step. In order to enable subsetting, you'll have to define your ops to yield conditional outputs based on the assets selected for execution: • for graph backed assets: https://docs.dagster.io/concepts/assets/graph-backed-assets#advanced-subsetting-graph-backed-assets • for multi assets: https://docs.dagster.io/concepts/assets/multi-assets#subsetting-multi-assets