gce_au_worker: Release allocated resources before terminating
Recent GCP resource leaks(e.g., chromium:579811) show that when
cbuildbot.stages.test_stages.GCETestStage times out, the subprocess that calls
out to ctest and thus this module is aborted, without getting a chance to clean
things up, resulting in leaked GCP resources and various quotas being hit.
This CL patches gce_au_worker to intercept SIGINT and SIGTERM and release
allocated resources before terminating the test process. It also updates `ctest`
to wait for a certain amount of time (10 minutes) before sending SIGKILL to the
test subprocess. Similarly, the cbuildbot code has been updated to suspend the
termination of the `ctest` subprocess.
TEST=gce_au_worker_unittest, manual ctest run passes, and ran a trybot against
Commit-Ready: Daniel Wang <email@example.com>
Tested-by: Daniel Wang <firstname.lastname@example.org>
Reviewed-by: Simran Basi <email@example.com>
Reviewed-by: Don Garrett <firstname.lastname@example.org>
5 files changed