gce_au_worker: Release allocated resources before terminating
Recent GCP resource leaks(e.g., chromium:579811) show that when
cbuildbot.stages.test_stages.GCETestStage times out, the subprocess that calls
out to ctest and thus this module is aborted, without getting a chance to clean
things up, resulting in leaked GCP resources and various quotas being hit.
This CL patches gce_au_worker to intercept SIGINT and SIGTERM and release
allocated resources before terminating the test process. It also updates `ctest`
to wait for a certain amount of time (10 minutes) before sending SIGKILL to the
test subprocess. Similarly, the cbuildbot code has been updated to suspend the
termination of the `ctest` subprocess.
BUG=b:26489739
TEST=gce_au_worker_unittest, manual ctest run passes, and ran a trybot against
lakitu-pre-cq
Change-Id: I1a174e7ce16f7f52c2cd1b80e1f936d83b2ac966
Reviewed-on: https://chromium-review.googlesource.com/323867
Commit-Ready: Daniel Wang <wonderfly@google.com>
Tested-by: Daniel Wang <wonderfly@google.com>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
5 files changed
