Hey there I m having some trouble with asset dependencies be dagster #ask-community

Hey there, I’m having some trouble with asset dep...

Simrun Basuita

05/08/2023, 6:00 PM

Hey there, I’m having some trouble with asset dependencies between multiple modules. I have a central

Definitions

which looks like this:

Copy code

from dagster import Definitions, load_assets_from_modules

from .assets import (
  pipeline_classifications,
  pipeline_emissions,
  pipeline_valuations,
)

defs = Definitions(
  assets=load_assets_from_modules([
    pipeline_classifications,
    pipeline_emissions,
    pipeline_valuations,
  ])
)

But I’m getting the error:

Copy code

UserWarning: Error loading repository location spectacle:dagster._core.errors.DagsterInvalidDefinitionError: Input asset '["company_emissions_cost"]' for asset '["archetype_financials"]' is not produced by any of the provided asset ops and is not one of the provided sources

company_emissions_cost

is defined in

pipeline_emissions

as:

Copy code

@asset()
def company_emissions_cost(clean_cdp_emissions, companies_w_dtics):

archetype_financials

is defined in

pipeline_classifications

as:

Copy code

@asset(ins={'company_emissions_cost': AssetIn(key='company_emissions_cost')})
def archetype_financials(companies_w_dtics, company_emissions_cost):

What am I doing wrong? Help / guidance much appreciated! Thanks.

🤖 1

sean

05/08/2023, 7:16 PM

Hi Sim, I think we should confirm that the assets you think are being loaded by

load_assets_from_module

are actually being loaded. Can you try this:

Copy code

assets=load_assets_from_modules([
    pipeline_classifications,
    pipeline_emissions,
    pipeline_valuations,
])

for defn in assets:
    if isinstance(defn, AssetsDefinition):
        print(defn.keys)

And confirm that what you expect is printed?

Simrun Basuita

05/09/2023, 6:59 AM

thanks sean will try it and get back to you

Simrun Basuita

05/09/2023, 12:21 PM

Thanks for your help on this. For some reason I forgot that at the end of the day these are all just python objects which can be inspected 🙂 I actually realised the problem was from

pipeline_classifcations

having its own Definitions variable. Which was getting read for some reason. Not sure if this is expected behaviour, but I've removed that extra Definitions and it's working now. Cheers! Refactor continues 🙂

🙌 1

Open in Slack

Previous Next