Jonny Wray
02/16/2023, 2:46 PMairbyte_nasdaq_assets = build_airbyte_assets(
connection_id=AIRBYTE_NASDAQ_CONNECTION_ID,
destination_tables=["nasdaq_end_of_day_data"],
asset_key_prefix=["timescale_source"],
)
and this (and two others) are feeding to a downstream dependency which is a dbt asset. So very similar to the modern data stack example project.
When I use the UI to materialize the assets - either selectively or the full workflow - I get an error
2023-02-16 14:34:16 +0000 - dagster - DEBUG - __ASSET_JOB - 91cb0a21-d1d0-4a94-896f-da41410b8e52 - 12712 - airbyte_sync_b9e15 - ASSET_MATERIALIZATION - Materialized value timescale_source nasdaq_end-of-day-data.
2023-02-16 14:34:16 +0000 - dagster - INFO - __ASSET_JOB - 91cb0a21-d1d0-4a94-896f-da41410b8e52 - airbyte_sync_b9e15 - op 'airbyte_sync_b9e15' did not fire outputs {'nasdaq_end_of_day_data'}
2023-02-16 14:34:16 +0000 - dagster - ERROR - __ASSET_JOB - 91cb0a21-d1d0-4a94-896f-da41410b8e52 - 12712 - airbyte_sync_b9e15 - STEP_FAILURE - Execution of step "airbyte_sync_b9e15" failed.
dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "airbyte_sync_b9e15" did not return an output for non-optional output "nasdaq_end_of_day_data"
Stack Trace:
File "C:\Work\MTM Data Research Unified\data-engineering\data-pipelines\env\lib\site-packages\dagster\_core\execution\plan\execute_plan.py", line 265, in dagster_event_sequence_for_step
for step_event in check.generator(step_events):
File "C:\Work\MTM Data Research Unified\data-engineering\data-pipelines\env\lib\site-packages\dagster\_core\execution\plan\execute_step.py", line 382, in core_dagster_event_sequence_for_step
for user_event in check.generator(
File "C:\Work\MTM Data Research Unified\data-engineering\data-pipelines\env\lib\site-packages\dagster\_core\execution\plan\execute_step.py", line 175, in _step_output_error_checked_user_event_sequence
raise DagsterStepOutputNotFoundError(
I assume I have a configuration and/or dependency wire up error but I can't find it. Any pointers gratefully received.
I should also say - the Airbyte connection sync is triggered fine and the Dagster console logs contain a bunch of messages suggesting successful progress.owen
02/16/2023, 5:55 PMnasdaq_end_of_day_data
? the destination_tables argument is somewhat misnamed, and sometimes the table that data ends up in does not actually match the stream name in the UI.
you might also be able to take advantage of load_assets_from_airbyte_instance which should save you some of the manual work of writing out those destination tablesJonny Wray
02/16/2023, 6:37 PMJonny Wray
02/20/2023, 3:29 PMnasdaq_end-of-day-data
- uses hyphen and not underscore between the last four words. However, when I change to use the correct stream name, or use the load_assets_from_airbyte_instance
method, I get an exception that looks like it is related to dictionary keys. I need to dig into that deeper. But - any reason why that stream name with mixed hyphen and underscore would not be valid/cause issues in the load asset phase?Jonny Wray
02/20/2023, 5:15 PMowen
02/21/2023, 6:53 PM