tree f108c21b66278211169209c067365aaac6562328
parent 097da61406fdf8f7decf0092191db98b9d93a3fe
author Ian Barkley-Yeung <iby@google.com> 1595037052 -0700
committer Commit Bot <commit-bot@chromium.org> 1595906913 +0000

crash: Investigate missed crash spike

We've seen a recent spike in missed Chrome crashes
(Crash.Chrome.MissedCrashes / Crash.Chrome.CrashesFromKernel) on some
boards in stable. While we can see the spike, we don't have enough info
to really investigate the spike.

Add a 1-in-1000 capture of some relevant logs and system status
information when crashes are missed.

This is a little tricky because the log capture must happen at the time
the "ignoring call from kernel" message happens, but the actual miss is
only confirmed 30 seconds later. So our normal log capturing scheme
inside crash_reporter doesn't work. Instead, we speculatively capture
logs when we see the "ignoring call from kernel" message, and then pass
them to crash_reporter whole.

BUG=b:160903152
TEST=Ran anomaly_detector on eve, spammed 20000 "ignoring call from
kernel" messages, confirmed we got ~20 crashes and the crashes looked
correct. Email addresses were redacted from logs. Uploaded crashes and
checked the logs were present on crash-staging/

Change-Id: Ia674d1724f2fb8197eb899fc974668750a0ecbb5
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform2/+/2314919
Tested-by: Ian Barkley-Yeung <iby@chromium.org>
Auto-Submit: Ian Barkley-Yeung <iby@chromium.org>
Commit-Queue: Ian Barkley-Yeung <iby@chromium.org>
Commit-Queue: Jorge Lucangeli Obes <jorgelo@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Reviewed-by: Miriam Zimmerman <mutexlox@chromium.org>