https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
m

Miguel Caballero Pinto

06/15/2023, 4:11 PM
Hi there! Im getting this error when I run multiple executions of the same job at the same time:
Copy code
google.api_core.exceptions.Forbidden: 403 Exceeded rate limits: too many table update operations for this table. For more information, see <https://cloud.google.com/bigquery/docs/troubleshoot-quotas>
It seems the jobs try to make updates too aggressively, but they shouldn’t since the schema of the assets did not change. Have you encounteres this before? Any workaround?
o

owen

06/16/2023, 8:38 PM
Hi @Miguel Caballero Pinto! Operations that insert records into the table are counted as "table update operations", so it's not just changes to the schema which would count against this limit. One option to reduce this issue might be to set a concurrency limit for that job: https://docs.dagster.io/guides/limiting-concurrency-in-data-pipelines#configuring-run-level-concurrency. See also: https://cloud.google.com/knowledge/kb/exceeded-rate-limits-too-many-table-update-operations-when-running-load-jobs-in-bigquery-000004366
3 Views