https://dagster.io/ logo
Title
g

Guy McCombe

12/19/2022, 4:47 PM
Hi! After turning off a stream in airbyte, my replication jobs are failing due to no output from the switched-off stream. The assets are loaded using the
load_assets_from_airbyte_instance
function, is there a way to get dagster to refresh the assets it’s loading from the instance? :keanu-thanks:
r

rex

12/19/2022, 4:49 PM
If you go over to your assets in Dagit, you can hit “Reload Definitions”
b

ben

12/19/2022, 4:49 PM
Here’s the button to do so ^
g

Guy McCombe

12/19/2022, 4:50 PM
Thanks - I guess it’s cached cos the assets are still hanging around 😕
b

ben

12/19/2022, 4:51 PM
You may also need to refresh the page (they might be cached on the page even if the asset itself was removed)
g

Guy McCombe

12/19/2022, 4:52 PM
I’ve refreshed, reloaded and redeployed but no joy yet :blob_sad:
b

ben

12/19/2022, 4:54 PM
Could you share the error message you are seeing?
g

Guy McCombe

12/19/2022, 4:55 PM
dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "airbyte_sync_54095" did not return an output for non-optional output "contacts"

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_plan.py", line 225, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
,  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 361, in core_dagster_event_sequence_for_step
    _step_output_error_checked_user_event_sequence(step_context, user_event_sequence)
,  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 156, in _step_output_error_checked_user_event_sequence
    output_name=step_output_def.name,
b

ben

12/19/2022, 5:01 PM
Hmm, the reload operation should invalidate any caching on our end, but it does seem like that asset is still being registered. Just to try to replicate - you had a stream enabled and turned off the sync via the UI toggle?
g

Guy McCombe

12/19/2022, 5:01 PM
Yep!
b

ben

12/19/2022, 6:46 PM
One thing we can try is to force our internal cache key to change, just to be sure that’s not the issue. If you’re not specifying a
connection_filter
to
load_assets_from_airbyte_instance
, you can give it a no-op
lambda _: True
, which should be enough to change the cache key.
g

Guy McCombe

12/19/2022, 6:51 PM
and if I am? 😬
b

ben

12/19/2022, 6:57 PM
I believe it uses the
inspect.getsource()
of the function, so changing its definition (in a no-op manner) should be enough
g

Guy McCombe

12/19/2022, 7:08 PM
:daggy-success: That worked, thanks very much for your help, Ben 😁
b

ben

12/19/2022, 7:09 PM
🙌 amazing
Will need to investigate more deeply on our end to see why that cache wasn’t getting invalidated, but hopefully that unsticks you for now
:daggy-love: 1
g

Guy McCombe

12/19/2022, 7:09 PM
Another reason to bring ingestion as code higher up the backlog priority 😅