https://dagster.io/ logo
#dagster-serverless
Title
# dagster-serverless
s

Son Do

08/18/2023, 7:19 PM
Hey dagster team im getting this weird error on dagster serverless I haven't seen it before:
Copy code
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListTasks operation: Rate exceeded
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
    self._wait_for_new_multipex_server(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
    task_arn = self.client.wait_for_new_service(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
    return self.check_service_has_running_task(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 557, in check_service_has_running_task
    running = self.ecs.list_tasks(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
    raise error_class(parsed_response, operation_name
p

prha

08/18/2023, 7:27 PM
Hi Son. We’re investigating right now.
🙏 1
s

Son Do

08/18/2023, 7:40 PM
Seems to work now
Looks like it is back @prha
Copy code
Copy
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation: Rate exceeded

  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1339, in _reconcile
    self._wait_for_new_server_ready(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 457, in _wait_for_new_server_ready
    self._wait_for_dagster_server_process(
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1721, in _wait_for_dagster_server_process
    self._wait_for_server_process(
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1770, in _wait_for_server_process
    additional_check()
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 460, in <lambda>
    additional_check=lambda: self.client.assert_task_not_stopped(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 543, in assert_task_not_stopped
    task = self.ecs.describe_tasks(cluster=self.cluster_name, tasks=[task_arn]).get("tasks")[0]
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
    raise error_class(parsed_response, operation_name)
p

prha

08/18/2023, 9:24 PM
Hi Son… we identified a couple callsites that were hitting some rate limits and rolled a change to help with some backoff for calls to
ListTasks
. We’ll take a look at doing something similar with `DescribeTasks`…
❤️ 1