Hi I need help I am trying to use `define asset job` with bo dagster #ask-community

Hi! I need help. I am trying to use `define_asset...

Remi Gabillet

03/28/2023, 3:41 PM

Hi! I need help. I am trying to use

define_asset_job

with both source & regular assets. I have an asset group with both types of assets in it. f I use

AssetSelection.groups

I get the error:

Asset selection specified both regular assets and source assets. This is not currently supported. Selections must be all regular assets or all source assets.

I'm even more confused because I have a different job that points to another group mixing both types of assets. That one loads successfully. 😕

🤖 1

Remi Gabillet

03/28/2023, 3:47 PM

I think I may be misunderstanding what source assets are. I have a dbt project and I assumed that all assets loaded with

load_assets_from_dbt_project

would be source assets. When I look at the job that is working, the dbt assets are coming up as regular assets, but not on the job that is not working.

Remi Gabillet

03/28/2023, 4:00 PM

so, the dbt assets that are explicitly used as inputs on dagster software-defined assets get categorized as "SourceAsset", causing this issue.

Remi Gabillet

03/28/2023, 4:00 PM

how can I work around it?

sean

03/28/2023, 4:06 PM

Hi Remi, thanks for reaching out. We recently started allowing

define_asset_job

to specify source assets, but it is a (temporary) limitation that we can’t allow both observable source and regular assets to be included in the same job, so selections including both will error. This is not meant to happen unless you explicitly specify a key for both a source asset and regular asset. Looks like the issue with

groups

was just an oversight, it should resolve only to regular assets. We are releasing tmrw and I can patch this for that release.

Remi Gabillet

03/28/2023, 4:09 PM

thanks for the quick reply @sean Actually, I just solved out my issue! I was mistakenly using

SourceAsset

to reference dbt models instead of

AssetIn

sean

03/28/2023, 4:09 PM

ah great-- well you found a bug in any case

Remi Gabillet

03/28/2023, 4:10 PM

yeah I see that 🙂

Remi Gabillet

03/28/2023, 4:11 PM

I was doing:

Copy code

dbt_asset = SourceAsset(key=["warehouse", DBT_SCHEMA, "dbt_asset"])

@asset
def es_index_profiles_v3b(context,
                          dbt_asset):
    ...

but I meant:

Copy code

dbt_asset = AssetIn(key=["warehouse", DBT_SCHEMA, "dbt_asset"])

@asset
def es_index_profiles_v3b(context,
                          dbt_asset):
    ...

Remi Gabillet

03/28/2023, 4:11 PM

I'm still unclear what

SourceAsset

really is

sean

03/28/2023, 4:22 PM

SourceAsset

represents a data asset that you want to use as an input but that Dagster doesn’t know how to compute. e.g. maybe the result of a request from some web api

Remi Gabillet

03/28/2023, 4:24 PM

I see. So it's similar to dbt sources right? I can declare it to document that a dagster depends on an external asset.

sean

03/28/2023, 4:35 PM

I’m not very familiar with dbt so I can’t speak to its similarity, but yes, you can “declare it to document that dagster depends on an external asset”.

Remi Gabillet

03/28/2023, 4:38 PM

thank you. I appreciate the help @sean!

16 Views

Open in Slack

Previous Next