An issue that I'm experiencing with my self-hosted Airbyte instance that has multiple connections is that, with a multiprocess executor Dagster will try to trigger multiple syncs simultaneously, and several of them will fail with a 409 error, which appears to be related to https://github.com/airbytehq/airbyte/issues/21208. One thought I had was to try introducing some skew to the API calls from the Dagster integration, but I'm not seeing any built-in way to do that. Is there some option that folks have used in the multiprocess executor to do something similar?
02/22/2023, 6:07 PM
409s are also triggered when you try to trigger a sync for a connection while one is already in progress (airbyte has no concept of queues for that)
so, is it possible that you have your assets set up such that one connection is duplicated and trying to trigger multiple times when you materialize? or it already had a sync in progress outside of dagster?
02/22/2023, 6:08 PM
Yeah, looking in the logs that's what I'm seeing. I might know what the problem is... It's probably because the connections are set to run every 24 hours from the Airbyte side, so the Airbyte and Dagster schedulers are in a race condition.
02/22/2023, 6:09 PM
oh yeah, you want to disable the airbyte scheduler if you're triggering syncs with dagster. set everything in airbyte to manual
02/22/2023, 6:18 PM
👍 Just did that. Now to keep an eye on the regular nightly runs 🙂
Closing the loop here, that was the source of the problem. Thanks again!