19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
hey team, been getting this a lot recently for a c...
# dagster-plusc
Charlie Bini
07/28/2023, 3:33 PMhey team, been getting this a lot recently for a couple of my sensors, but the evaluation function runs ~20 seconds during testing. I'm thinking something may be up with out sensor daemon
Copy code
dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function. One way to avoid this error is to break up the sensor work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.j
jordan
07/28/2023, 3:34 PMyou probably have more memory/cpu allocated locally than we do on serverless but i’ll take a look
jordan
07/28/2023, 3:34 PMthe severless grpc servers are pretty under-provisioned - which can cause some challenges around heavier sensor workloads. it’s something we’re still deciding the best way to address.
jordan
07/28/2023, 3:35 PMoh wait you’re not on serverless, are you?
in that case, same basic answer but hopefully more solvable
can you take a look at your grpc server pods and see if they’re resource constrained?
c
Charlie Bini
07/28/2023, 3:35 PMyeah I'm on a hybrid deployment
Charlie Bini
07/28/2023, 3:36 PMwould that be the code location pod, or dagster agent pod, or something different?
j
jordan
07/28/2023, 3:40 PMcode location pod
c
Charlie Bini
07/28/2023, 3:47 PMwrong one before
Charlie Bini
07/28/2023, 3:53 PMrunning it on the deployed pod is still ~20 seconds
Charlie Bini
07/28/2023, 4:02 PMas a workaround, I tried setting this env var to increase the timeout time yesterday and it doesn't appear to have made it to the pod
j
jordan
07/28/2023, 7:58 PMWe’re going to have to keep digging into this - I don’t have an answer yet.
a
alex
07/31/2023, 8:06 PMthe
agentc
Charlie Bini
07/31/2023, 9:13 PMthanks @alex, I use helm yeah
Charlie Bini
07/31/2023, 9:14 PMI tried adding the env to the chart but I don't think it worked
Charlie Bini
07/31/2023, 9:15 PMI'm using GKE autopilot, which doesn't have a set node size, but the default pod size is 0.5 vCPU/2 GiB https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-resource-requests#defaults
Charlie Bini
07/31/2023, 9:16 PMon my helm chart, I set
Copy code
dagsterCloudAgent:
env:
DAGSTER_GRPC_TIMEOUT_SECONDS: "120"Charlie Bini
07/31/2023, 9:17 PMit's visible on the config map, but the timeout still seems to be 60 according to errors
a
alex
07/31/2023, 9:48 PMhmm, did the agent pod restart when you made this change?
alex
07/31/2023, 9:48 PMwe may be missing some machinery to force helm/k8s to cause a restart on env var change, since its indirected via the config map
c
Charlie Bini
07/31/2023, 11:21 PMI thought it did, but I deleted the agent pod and started a new one
Charlie Bini
07/31/2023, 11:22 PMI'm seeing the env var show up correctly when I connect to the pod, so I'll see if the error persists after that
a
alex
08/01/2023, 3:41 PMgot it keep me posted
in terms of why this takes longer, the user code server is a grpc server that handles executing your code in threads. When a sensor evaluation happens its happening in a thread with any other requests that user code server is handling. If you have multiple sensors that start to overlap, they can contend for compute in the process. Potential mitigations are:
• increase the resources allocated to the user code server
• split across code locations (each code location gets its own code server)
Another option if you have the bandwidth to do so is perform some profiling on the user code server using a tool like py-spy. This can reveal exactly what is slowing things down.
2 Views