[autotest] Do not ABORT even if reboot ssh command timed out.
If it is blocking call i.e. wait=True is passed to reboot(),
do not ABORT even if reboot ssh command is failed.
Running reboot in remote machine can trigger flaky behavior on ssh,
like timeout, even if reboot is successfully in progress.
It means, even if reboot is successfully done, the ssh command
may fail, which causes unexpected ABORT on bot status.
Instead, with this CL, it starts to rely on restart polling
(wait_for_restart()), if wait=True is passed.
BUG=b:37652392
TEST=Ran bots.
Change-Id: I9d3e226a6a9352df3b963651e67da4cf6ef56739
Reviewed-on: https://chromium-review.googlesource.com/487925
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
(cherry picked from commit 3d512d3e8a063772e90f452d334473a0136d337a)
Reviewed-on: https://chromium-review.googlesource.com/491227
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>
Commit-Queue: Hidehiko Abe <hidehiko@chromium.org>
Trybot-Ready: Hidehiko Abe <hidehiko@chromium.org>
diff --git a/server/hosts/remote.py b/server/hosts/remote.py
index 337fc24..af77a8d 100644
--- a/server/hosts/remote.py
+++ b/server/hosts/remote.py
@@ -115,6 +115,8 @@
Args:
timeout - How long to wait for the reboot.
wait - Should we wait to see if the machine comes back up.
+ If this is set to True, ignores reboot_cmd's error
+ even if occurs.
fastsync - Don't wait for the sync to complete, just start one
and move on. This is for cases where rebooting prompty
is more important than data integrity and/or the
@@ -142,9 +144,17 @@
self.run_background(reboot_cmd)
except error.AutoservRunError:
- self.record("ABORT", None, "reboot.start",
- "reboot command failed")
- raise
+ # If wait is set, ignore the error here, and rely on the
+ # wait_for_restart() for stability, instead.
+ # reboot_cmd sometimes causes an error even if reboot is
+ # successfully in progress. This is difficult to be avoided,
+ # because we have no much control on remote machine after
+ # "reboot" starts.
+ if not wait:
+ # TODO(b/37652392): Revisit no-wait case, later.
+ self.record("ABORT", None, "reboot.start",
+ "reboot command failed")
+ raise
if wait:
self.wait_for_restart(timeout, old_boot_id=current_boot_id,
**dargs)
