https://dagster.io/ logo
Powered by
#ask-community
Title
# ask-community
a

Alec Ryan

07/13/2022, 12:21 PM
Not sure exactly why I am seeing this error
Copy code
dagster.core.errors.DagsterExecutionLoadInputError: Error occurred while loading input "game_finals" of step "generate_dbt_artifacts":
The above exception was caused by the following exception:
IsADirectoryError: [Errno 21] Is a directory: '/opt/dagster/dagster_home/storage/game_finals'
I'm creating the asset "game_finals" and using it as an AssetIn for a downstream model. I can see the asset key in my RDS database
My asset def looks like this
Copy code
@asset(
    required_resource_keys={'s3', 'dbt'},
    compute_kind='python',
    ins={"game_finals": AssetIn("game_finals")}
)
def generate_dbt_artifacts(context, game_finals):
    '''
    dbt generate command for refresh of dbt artifacts:
    - index.html
    - manifest.json
    - catalog.json
    '''
y

yuhan

07/13/2022, 5:02 PM
The error indicates that you’re using 
/opt/dagster/dagster_home/storage
as the storage. Were you running it on a container?
a

Alec Ryan

07/14/2022, 12:36 AM
Yeah, I am
though my storage should be RDS..
I found the problematic behavior, it's from using the new dbt partitioning arguments
Copy code
#def partition_key_to_dbt_vars(partition_key):
#    return {"run_date": partition_key}

dbt_assets = load_assets_from_dbt_project(
    project_dir = dbt_dir, 
    profiles_dir = dbt_dir,
    #partitions_def=daily_partitions_def,
    #partition_key_to_vars_fn=partition_key_to_dbt_vars,

    )
When I change the configuration by commenting out the lines above, the models run as expeceted
FYI @owen
o

owen

07/14/2022, 5:37 PM
ah interesting -- I think I've seen something like this before, I'll see if I can track down what's happening
one quick question: are you using a special IO manager to allow you to load game_finals as a dataframe (or similar)? if not, then the game_finals input into generage_dbt_artifacts will not have any useful information in it (it'll just be a value of None), and so you should be able to circumvent this issue by setting game_finals as a 
non_argument_dep
to generate_dbt_artifacts
this will tell dagster that you don't care about the contents, and not to attempt loading that output value. this would look like:
Copy code
@asset(
    required_resource_keys={'s3', 'dbt'},
    compute_kind='python',
    non_argument_deps={"game_finals"}
)
def generate_dbt_artifacts(context):
    '''
    dbt generate command for refresh of dbt artifacts:
    - index.html
    - manifest.json
    - catalog.json
    '''
I think this would probably solve your problem
a

Alec Ryan

07/14/2022, 7:06 PM
Thanks, I'll give this a try. Tbh I haven't fully wrapped my head around I/O managers yet lol
I do plan on reading up on that soon
That does seem to work for me. Thanks @owen!
🙌 1
17 Views
Powered by