19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
My jobs (which are supposed to spawn every 5 minut...
# ask-communityl
Liezl Puzon
04/20/2022, 7:40 PMMy jobs (which are supposed to spawn every 5 minutes and take ~1-3mins to complete) are stuck in “Starting” as far back as 3 days even though I have an autoscale GKE cluster
I’m also seeing an extremely slow load then error when trying to go to this sensor page https://dagit.nautilusapp.xyz/workspace/example_repo@custom-example-user-code17/sensors/spawn_train_workers and this page https://dagit.nautilusapp.xyz/workspace/example_repo@custom-example-user-code17/sensors
Do I need to beef up or repair my cluster? I’m still getting a bunch of completed runs even til now, so I know my sensor is functioning
✅ 1
Liezl Puzon
04/20/2022, 7:46 PMalthough I am having trouble finding (via tags) all the jobs that I KNOW were supposed to have been spawned via sensor
(in both the All Runs and In Progress page)
Liezl Puzon
04/20/2022, 7:46 PMso I’m not confident that the sensor is fully functioning and issuing all the `RunRequest`s that it’s supposed to
a
alex
04/20/2022, 7:47 PMit appears your database has consumed all the disk it has access to which would cause writes to fail putting the app in to this broken state you are observingDiskFull
👍 1
l
Liezl Puzon
04/20/2022, 7:54 PMgotcha, looks like this is the workload
Noob question, but is this correct: I can beef this up by setting the
resourcespostgresql:Liezl Puzon
04/20/2022, 7:57 PMIs there any downside to creating a separate persistent, managed postgres instance with Google Cloud SQL and pointing to that (vs configuring it to live in the GKE cluster via helm here)?
it seems more stable and easier to upgrade separately if I have an external postgres instance
hopefully latency isn’t a big factor since I’m using a regional cluster anyway. And this is what a typical “Managed Dagster Cloud” setup might look like anyways?
👍 1
a
alex
04/20/2022, 7:57 PMI would recommend using a fully managed database if you are not comfortable with DB management
https://cloud.google.com/blog/products/databases/to-run-or-not-to-run-a-database-on-kubernetes-what-to-consider
❤️ 1
l
Liezl Puzon
04/20/2022, 7:58 PMahh gotcha! thanks for the resource