tree 5ce0ffccc6db9d5336de2b163c978290724c9e03
parent 6221a4a06b0b8400fa190914227996bde937daa3
author Grant Grundler <grundler@chromium.org> 1502238644 -0700
committer chrome-bot <chrome-bot@chromium.org> 1518162139 -0800

check_ethernet: try each recovery method only once

Reboot has proven to be the most effective "recovery" method.
15 minutes was too long to get to reboot.

Restructured the main() recovery loop with that in mind:
1) execute each method only once before rebooting.
2) drop methods which have not proven effective.
3) look for PAUSE_FILE before trying any method.
4) Stop avoiding cdc_ether driver. This is the
   NIC driver for Linksys USB3GIGV1 on older kernels.

For (2) provision jobs logs from bvt/cq pools was used to determine which
methods were effective for recovery. "Survivorship bias" was considered
and in fact a primary reason some methods were dropped.

I believe (4) was added to avoid bouncing 3G/4G/LTE modems.
Tests that depend on LTE and bring down the link to lab network will need
to use the open(PAUSE_ETHERNET_HOOK_FILE)/flock() also used by sys_power.py
to disable these checks.

BUG=chromium:758781
TEST=copy script to /tmp on DUT
       $ stop recover_duts
    Disconnect LAN cable. Then:
       $ /tmp/check_ethernet.hook
    And verify the DUT tries each method and then reboots.

Change-Id: I296d5565d264442a9033d6416a8262b1865961af
Reviewed-on: https://chromium-review.googlesource.com/611391
Commit-Ready: Grant Grundler <grundler@chromium.org>
Tested-by: Grant Grundler <grundler@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>