19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hey dagster team im getting this weird error on da...
# dagster-serverlesss
Son Do
08/18/2023, 7:19 PMHey dagster team im getting this weird error on dagster serverless I haven't seen it before:
Copy code
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListTasks operation: Rate exceeded
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
self._wait_for_new_multipex_server(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 557, in check_service_has_running_task
running = self.ecs.list_tasks(
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_namep
prha
08/18/2023, 7:27 PMHi Son. We’re investigating right now.
🙏 1
s
Son Do
08/18/2023, 7:40 PMSeems to work now
Son Do
08/18/2023, 9:21 PMLooks like it is back @prha
Copy code
Copy
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation: Rate exceeded
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1339, in _reconcile
self._wait_for_new_server_ready(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 457, in _wait_for_new_server_ready
self._wait_for_dagster_server_process(
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1721, in _wait_for_dagster_server_process
self._wait_for_server_process(
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1770, in _wait_for_server_process
additional_check()
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 460, in <lambda>
additional_check=lambda: self.client.assert_task_not_stopped(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 543, in assert_task_not_stopped
task = self.ecs.describe_tasks(cluster=self.cluster_name, tasks=[task_arn]).get("tasks")[0]
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name)p
prha
08/18/2023, 9:24 PMHi Son… we identified a couple callsites that were hitting some rate limits and rolled a change to help with some backoff for calls to
ListTasks❤️ 1
4 Views