https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jonny Wray

02/16/2023, 2:46 PM
Question regarding an error I'm getting at the moment. If anyone has any insight that would be great. I have an Airbyte asset (one of many, but all are showing same behaviour) configured via:
Copy code
airbyte_nasdaq_assets = build_airbyte_assets(
    connection_id=AIRBYTE_NASDAQ_CONNECTION_ID,
    destination_tables=["nasdaq_end_of_day_data"],
    asset_key_prefix=["timescale_source"],
)
and this (and two others) are feeding to a downstream dependency which is a dbt asset. So very similar to the modern data stack example project. When I use the UI to materialize the assets - either selectively or the full workflow - I get an error
Copy code
2023-02-16 14:34:16 +0000 - dagster - DEBUG - __ASSET_JOB - 91cb0a21-d1d0-4a94-896f-da41410b8e52 - 12712 - airbyte_sync_b9e15 - ASSET_MATERIALIZATION - Materialized value timescale_source nasdaq_end-of-day-data.
2023-02-16 14:34:16 +0000 - dagster - INFO - __ASSET_JOB - 91cb0a21-d1d0-4a94-896f-da41410b8e52 - airbyte_sync_b9e15 - op 'airbyte_sync_b9e15' did not fire outputs {'nasdaq_end_of_day_data'}
2023-02-16 14:34:16 +0000 - dagster - ERROR - __ASSET_JOB - 91cb0a21-d1d0-4a94-896f-da41410b8e52 - 12712 - airbyte_sync_b9e15 - STEP_FAILURE - Execution of step "airbyte_sync_b9e15" failed.

dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "airbyte_sync_b9e15" did not return an output for non-optional output "nasdaq_end_of_day_data"

Stack Trace:
  File "C:\Work\MTM Data Research Unified\data-engineering\data-pipelines\env\lib\site-packages\dagster\_core\execution\plan\execute_plan.py", line 265, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "C:\Work\MTM Data Research Unified\data-engineering\data-pipelines\env\lib\site-packages\dagster\_core\execution\plan\execute_step.py", line 382, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "C:\Work\MTM Data Research Unified\data-engineering\data-pipelines\env\lib\site-packages\dagster\_core\execution\plan\execute_step.py", line 175, in _step_output_error_checked_user_event_sequence
    raise DagsterStepOutputNotFoundError(
I assume I have a configuration and/or dependency wire up error but I can't find it. Any pointers gratefully received. I should also say - the Airbyte connection sync is triggered fine and the Dagster console logs contain a bunch of messages suggesting successful progress.
o

owen

02/16/2023, 5:55 PM
hi @Jonny Wray! can you confirm that the stream name in airbyte exactly matches
nasdaq_end_of_day_data
? the destination_tables argument is somewhat misnamed, and sometimes the table that data ends up in does not actually match the stream name in the UI. you might also be able to take advantage of load_assets_from_airbyte_instance which should save you some of the manual work of writing out those destination tables
j

Jonny Wray

02/16/2023, 6:37 PM
Thanks for the reply. I thought they matched but indeed I can’t guarantee it. I probably won’t be able to look until next week but great place for me to start. Cheers. Will let you know what I find.
Just checked this and you are correct. The stream name is
nasdaq_end-of-day-data
- uses hyphen and not underscore between the last four words. However, when I change to use the correct stream name, or use the
load_assets_from_airbyte_instance
method, I get an exception that looks like it is related to dictionary keys. I need to dig into that deeper. But - any reason why that stream name with mixed hyphen and underscore would not be valid/cause issues in the load asset phase?
I think I may have the answer but it’s a bit of an extrapolation from another error I saw and fixed. Looks like hyphens are not a valid character for Dagster names (that’s the error I saw and fixed). Does that mean the autogenerated asset names from the Airbyte integration could fail if the Airbyte stream names contain hyphens? Error I got didn’t say that, but it makes sense it could cause a key related error.
o

owen

02/21/2023, 6:53 PM
hm this is a good question -- cc @ben
3 Views