19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
An issue that I'm experiencing with my self-hosted...
# integration-airbytet
Tobias Macey
02/22/2023, 4:27 PMAn issue that I'm experiencing with my self-hosted Airbyte instance that has multiple connections is that, with a multiprocess executor Dagster will try to trigger multiple syncs simultaneously, and several of them will fail with a 409 error, which appears to be related to https://github.com/airbytehq/airbyte/issues/21208. One thought I had was to try introducing some skew to the API calls from the Dagster integration, but I'm not seeing any built-in way to do that. Is there some option that folks have used in the multiprocess executor to do something similar?
dagster bot responded by community 1
a
Adam Bloom
02/22/2023, 6:07 PM409s are also triggered when you try to trigger a sync for a connection while one is already in progress (airbyte has no concept of queues for that)
Adam Bloom
02/22/2023, 6:08 PMso, is it possible that you have your assets set up such that one connection is duplicated and trying to trigger multiple times when you materialize? or it already had a sync in progress outside of dagster?
t
Tobias Macey
02/22/2023, 6:08 PMYeah, looking in the logs that's what I'm seeing. I might know what the problem is... It's probably because the connections are set to run every 24 hours from the Airbyte side, so the Airbyte and Dagster schedulers are in a race condition.
a
Adam Bloom
02/22/2023, 6:09 PMoh yeah, you want to disable the airbyte scheduler if you're triggering syncs with dagster. set everything in airbyte to manual
t
Tobias Macey
02/22/2023, 6:18 PM👍 Just did that. Now to keep an eye on the regular nightly runs 🙂
Tobias Macey
02/23/2023, 6:49 PMClosing the loop here, that was the source of the problem. Thanks again!
7 Views