tree 8a5a1ac23ab0a115fe09845cd035e957afd66f5e
parent dec23c1c7b62ac4797c828a329f78220fc5ea895
author Daniel Wang <wonderfly@google.com> 1453774556 -0800
committer chrome-bot <chrome-bot@chromium.org> 1454980056 -0800

gce_au_worker: Release allocated resources before terminating

Recent GCP resource leaks(e.g., chromium:579811) show that when
cbuildbot.stages.test_stages.GCETestStage times out, the subprocess that calls
out to ctest and thus this module is aborted, without getting a chance to clean
things up, resulting in leaked GCP resources and various quotas being hit.

This CL patches gce_au_worker to intercept SIGINT and SIGTERM and release
allocated resources before terminating the test process. It also updates `ctest`
to wait for a certain amount of time (10 minutes) before sending SIGKILL to the
test subprocess. Similarly, the cbuildbot code has been updated to suspend the
termination of the `ctest` subprocess.

BUG=b:26489739
TEST=gce_au_worker_unittest, manual ctest run passes, and ran a trybot against
    lakitu-pre-cq

Change-Id: I1a174e7ce16f7f52c2cd1b80e1f936d83b2ac966
Reviewed-on: https://chromium-review.googlesource.com/323867
Commit-Ready: Daniel Wang <wonderfly@google.com>
Tested-by: Daniel Wang <wonderfly@google.com>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>