Pete Hunt
03/08/2023, 4:29 AMSterling Paramore
04/05/2023, 8:27 PMZach P
04/06/2023, 6:08 PMdagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC
with either Error Code: Unreachable
or Error Code: Unavailable
. Is there some service issue going? I’ll post more of the errors in the thread.David Merritt
04/10/2023, 4:55 AMDavid Merritt
04/14/2023, 3:44 PMSterling Paramore
04/14/2023, 4:31 PMFélix Tremblay
05/23/2023, 10:23 PMAgent xxxxxxxx detected that run worker failed.
Over the past 7 days, we have encountered three instances of such errors, which account for approximately 0.1% of the total runs.
I was wondering if anyone has investigated the root causes behind these worker failures.
Additionally, I wanted to suggest the possibility of implementing an automatic process wherein the agent can initiate the launch of a new worker when such failures occur. This could help mitigate the impact of these issues and ensure smoother operations.
Thanks!Manish Khatri
06/06/2023, 8:31 AMRUN_FAILURE
Error: dagster_cloud_cli.core.errors.GraphQLStorageError
Trace stack in thread…
``````Simon Weber
06/21/2023, 9:53 PMartemis
.Joel Olazagasti
06/26/2023, 4:43 PMGraphQLStorageError
that's blocking our auto-materialize jobs, trace in thread:Simon Weber
06/27/2023, 9:30 PMmanaged_python_loggers
dagster setting and I think I'm getting mixed up by the different config files. it didn't work putting it in my dagster_cloud.yaml (where we have settings like code_location). putting it in a separate dagster.yaml works locally, but it's not picked up by the cloud cli when I deploy. I went looking in the web ui and didn't see a place to configure it either -- it looks like a copy of the dagster cloud configPierre Cadman
07/20/2023, 12:36 PMJoel Olazagasti
07/25/2023, 2:22 PMJin Kim
08/01/2023, 11:34 PMView the status of your locations at <http://miracle.dagster.cloud/475ac3c997508b328529e52c352390fa84d1f93f/workspace>.
Waiting for agent to sync changes to miracle-dagster...
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Still waiting for agent to sync changes to miracle-dagster. This can take a few minutes.
Error: No Dagster Cloud agent is actively heartbeating. Make sure that you have a Dagster Cloud agent running.
Traceback (most recent call last):
File "/home/runner/.pex/venvs/4237725ac0312343e87c13daa25f529bb537503a/bb8d7c46f1a0a639d46b6e06c454ccfb042eac7b/pex", line 263, in <module>
exec(ast, globals_map, locals_map)
File "src/create_or_update_comment.py", line 81, in <module>
main()
File "src/create_or_update_comment.py", line 33, in main
pr = repo.get_pull(pr_id)
File "/home/runner/.pex/venvs/4237725ac0312343e87c13daa25f529bb537503a/bb8d7c46f1a0a639d46b6e06c454ccfb042eac7b/lib/python3.8/site-packages/github/Repository.py", line 2835, in get_pull
headers, data = self._requester.requestJsonAndCheck(
File "/home/runner/.pex/venvs/4237725ac0312343e87c13daa25f529bb537503a/bb8d7c46f1a0a639d46b6e06c454ccfb042eac7b/lib/python3.8/site-packages/github/Requester.py", line 398, in requestJsonAndCheck
return self.__check(
File "/home/runner/.pex/venvs/4237725ac0312343e87c13daa25f529bb537503a/bb8d7c46f1a0a639d46b6e06c454ccfb042eac7b/lib/python3.8/site-packages/github/Requester.py", line 423, in __check
raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "<https://docs.github.com/rest/pulls/pulls#get-a-pull-request>"}
Ignoring failure to update PR comment: None
None
Error: Failed to deploy Python Executable. Try disabling fast deploys by setting `ENABLE_FAST_DEPLOYS: 'false'` in your .github/workflows/*yml.
Error: Process completed with exit code 1.
Todd de Quincey
08/07/2023, 10:24 AMError: pg_config executable not found.
error in GitHub actionsJoel Olazagasti
08/07/2023, 7:52 PMdbt deps
to install my dbt packages? We're currently on the docker deployment with a post install scriptTodd de Quincey
08/12/2023, 5:30 PMStefan Adelbert
08/18/2023, 2:53 AMSon Do
08/18/2023, 7:19 PMbotocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListTasks operation: Rate exceeded
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
self._wait_for_new_multipex_server(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 557, in check_service_has_running_task
running = self.ecs.list_tasks(
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name
Kyle Montag
08/21/2023, 1:47 PMSébastien Duguay
08/23/2023, 5:23 PMCopy
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation: Rate exceeded
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
self._wait_for_new_multipex_server(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 587, in check_service_has_running_task
task = self.ecs.describe_tasks(
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
Todd de Quincey
09/01/2023, 6:04 PMAWS_REGION
env var being overridden in ServerlessJacob Marcil
09/07/2023, 3:24 PMretry_policy
of an asset?
I need to query API that are not very reliable (The data can show up with 1 hour to 3 days of delay), and was wondering if I configure my retry_policy
to something like 1 or 2 days, would I pay for those days while doing nothing since my run
is technically in progress?Son Do
09/18/2023, 3:06 PMDavid Merritt
09/18/2023, 11:40 PMCharles
09/19/2023, 8:29 AMJacob Marcil
09/19/2023, 9:41 PMpartitions
to keep track of them. T
hat would create something around 500k partitions for that specific asset growing at a rate of 1000 partitions per day. Would that highly increase the cost of Dagster Serverless even if the compute time if very small?David Merritt
09/21/2023, 12:08 AMManish Khatri
09/21/2023, 10:42 AMAllocating compute resources to ensure run isolation. This can take a few minutes.
Still waiting for compute resources to spin up.
Still waiting for compute resources to spin up.
...
Still waiting for compute resources to spin up.
Agent b48510b6 detected that run worker failed.
In terms of a better experience around this, I believe that Dagster should catch this specific exception and have some type of retry mechanism in place. It feels lazy that Dagster tried once, encountered this failure and then gave up so quickly. If our impacted job was a very important daily job, it could have been much more disruptive.Anthony Yim
09/21/2023, 10:50 PM