An issue that I m experiencing with my self hosted Airbyte i

Question

An issue that I m experiencing with my self hosted Airbyte instance that has multiple connections is that with a multiprocess executor Dagster will try to trigger multiple syncs simultaneously and several of them will fail with a 409 error which appears to be related to <https github com airbytehq airbyte issues 21208> One thought I had was to try introducing some skew to the API calls from the Dagster integration but I m not seeing any built in way to do that Is there some option that folks have used in the multiprocess executor to do something similar

Adam Bloom · Answer

409s are also triggered when you try to trigger a sync for a connection while one is already in progress airbyte has no concept of queues for that

Adam Bloom · Answer

so is it possible that you have your assets set up such that one connection is duplicated and trying to trigger multiple times when you materialize or it already had a sync in progress outside of dagster

Tobias Macey · Answer

Yeah looking in the logs that s what I m seeing I might know what the problem is It s probably because the connections are set to run every 24 hours from the Airbyte side so the Airbyte and Dagster schedulers are in a race condition

Adam Bloom · Answer

oh yeah you want to disable the airbyte scheduler if you re triggering syncs with dagster set everything in airbyte to manual

Tobias Macey · Answer

+1 Just did that Now to keep an eye on the regular nightly runs slightly smiling face

Tobias Macey · Answer

Closing the loop here that was the source of the problem Thanks again