Reland: "autotest: Add timeout to _ping_check_status"
In Bluetooth MTBF tests, we encountered a scenario where a ping command
that should have completed in 1s (with a failure) completed almost
8 seconds later with success. Either ping failed to enforce the deadline
(set via -w1) or the subprocess call got stuck somewhere.
Here is an example log of the issue:
09/12 23:43:15.056 DEBUG| utils:0219| Running 'ping chromeos15-row5-rack5-host6 -w1 -c1'
09/12 23:43:23.067 DEBUG| utils:1993| [rc=0] 1 packets transmitted, 1 received, 0% packet loss, time 0ms; rtt min/avg/max/mdev = 2.923/2.923/2.923/0.000 ms
To enforce a more consistent ping check, also use the timeout parameter
when running the ping request. Either the ping will finish quickly (less
than deadline), finish at the deadline (which will likely be before the
timeout) or after the timeout (at which point the subprocess will be
killed).
Also, add a check in the return value of ping to make sure we don't act
on a None result.
BUG=b:168428728
TEST=Ran a bluetooth specific suspend/resume stress test that uses
cros_hosts.test_wait_for_sleep and passed 50 iterations without the
issue.
Change-Id: I9bcd863dc7e39ccb963c5b962549d8f33198325c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/2446878
Tested-by: Abhishek Pandit-Subedi <abhishekpandit@chromium.org>
Reviewed-by: Yu Liu <yudiliu@google.com>
Commit-Queue: Abhishek Pandit-Subedi <abhishekpandit@chromium.org>
diff --git a/client/common_lib/utils.py b/client/common_lib/utils.py
index 8b01889..77c4fca 100644
--- a/client/common_lib/utils.py
+++ b/client/common_lib/utils.py
@@ -1942,7 +1942,12 @@
return 'NoSerialNumber'
-def ping(host, deadline=None, tries=None, timeout=60, user=None):
+def ping(host,
+ deadline=None,
+ tries=None,
+ timeout=60,
+ ignore_timeout=False,
+ user=None):
"""Attempt to ping |host|.
Shell out to 'ping' if host is an IPv4 addres or 'ping6' if host is an
@@ -1962,6 +1967,8 @@
@param deadline: seconds within which |tries| pings must succeed.
@param tries: number of pings to send.
@param timeout: number of seconds after which to kill 'ping' command.
+ @param ignore_timeout: If true, timeouts won't raise CmdTimeoutError.
+ @param user: Run as a specific user
@return exit code of ping command.
"""
args = [host]
@@ -1976,10 +1983,24 @@
args = [user, '-c', ' '.join([cmd] + args)]
cmd = 'su'
- result = run(cmd, args=args, verbose=True,
- ignore_status=True, timeout=timeout,
+ result = run(cmd,
+ args=args,
+ verbose=True,
+ ignore_status=True,
+ timeout=timeout,
+ ignore_timeout=ignore_timeout,
stderr_tee=TEE_TO_LOGS)
+ # Sometimes the ping process times out even though a deadline is set. If
+ # ignore_timeout is set, it will fall through to here instead of raising.
+ if result is None:
+ logging.debug('Unusual ping result (timeout)')
+ # From man ping: If a packet count and deadline are both specified, and
+ # fewer than count packets are received by the time the deadline has
+ # arrived, it will also exit with code 1. On other error it exits with
+ # code 2.
+ return 1 if deadline and tries else 2
+
rc = result.exit_status
lines = result.stdout.splitlines()
diff --git a/server/hosts/cros_host.py b/server/hosts/cros_host.py
index 47910d4..2005ea9 100644
--- a/server/hosts/cros_host.py
+++ b/server/hosts/cros_host.py
@@ -1798,7 +1798,11 @@
(i.e. both True or both False).
"""
- ping_val = utils.ping(self.hostname, tries=1, deadline=1)
+ ping_val = utils.ping(self.hostname,
+ tries=1,
+ deadline=1,
+ timeout=2,
+ ignore_timeout=True)
return not (status ^ (ping_val == 0))
def _ping_wait_for_status(self, status, timeout):
