https://dagster.io/ logo
Powered by
#dagster-airbyte
Title
# dagster-airbyte
g

Guy McCombe

12/19/2022, 4:47 PM
Hi! After turning off a stream in airbyte, my replication jobs are failing due to no output from the switched-off stream. The assets are loaded using the 
load_assets_from_airbyte_instance
function, is there a way to get dagster to refresh the assets it’s loading from the instance? keanu thanks
r

rex

12/19/2022, 4:49 PM
If you go over to your assets in Dagit, you can hit “Reload Definitions”
b

ben

12/19/2022, 4:49 PM
Here’s the button to do so ^
g

Guy McCombe

12/19/2022, 4:50 PM
Thanks - I guess it’s cached cos the assets are still hanging around 😕
b

ben

12/19/2022, 4:51 PM
You may also need to refresh the page (they might be cached on the page even if the asset itself was removed)
g

Guy McCombe

12/19/2022, 4:52 PM
I’ve refreshed, reloaded and redeployed but no joy yet blob sad
b

ben

12/19/2022, 4:54 PM
Could you share the error message you are seeing?
g

Guy McCombe

12/19/2022, 4:55 PM
Copy code
dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "airbyte_sync_54095" did not return an output for non-optional output "contacts"

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_plan.py", line 225, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
,  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 361, in core_dagster_event_sequence_for_step
    _step_output_error_checked_user_event_sequence(step_context, user_event_sequence)
,  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 156, in _step_output_error_checked_user_event_sequence
    output_name=step_output_def.name,
b

ben

12/19/2022, 5:01 PM
Hmm, the reload operation should invalidate any caching on our end, but it does seem like that asset is still being registered. Just to try to replicate - you had a stream enabled and turned off the sync via the UI toggle?
g

Guy McCombe

12/19/2022, 5:01 PM
Yep!
b

ben

12/19/2022, 6:46 PM
One thing we can try is to force our internal cache key to change, just to be sure that’s not the issue. If you’re not specifying a 
connection_filter
to 
load_assets_from_airbyte_instance
, you can give it a no-op 
lambda _: True
, which should be enough to change the cache key.
g

Guy McCombe

12/19/2022, 6:51 PM
and if I am? 😬
b

ben

12/19/2022, 6:57 PM
I believe it uses the 
inspect.getsource()
of the function, so changing its definition (in a no-op manner) should be enough
g

Guy McCombe

12/19/2022, 7:08 PM
daggy success That worked, thanks very much for your help, Ben 😁
b

ben

12/19/2022, 7:09 PM
🙌 amazing
Will need to investigate more deeply on our end to see why that cache wasn’t getting invalidated, but hopefully that unsticks you for now
daggy love 1
g

Guy McCombe

12/19/2022, 7:09 PM
Another reason to bring ingestion as code higher up the backlog priority 😅
2 Views
Powered by