we've been having some intermittent issues this morning with submitting jobs... from around 8-8:10 PST jobs that were submitted (5-6) got stuck in "Not started", and never started (interestingly, these records only show when viewing runs from the "All runs" tab, and not in the job-specific runs view). now at 9:00 PST we just had another job get stuck in "Not started", although I was able to relaunch it just now. it doesn't look like my agent has gone down recently... anyone have any ideas why this could be happening or pointers for debugging?

Hey Zach - just sent you a couple of followup questions in a private channel

@Zach are you using GKE? I'm seeing something similar

no, I'm using AWS ECS - it turned out to be from abusing the run tags to hold too much data