Re-land: Autotest: rewrite machine_install to be robust to devserver flakiness.

This CL both simplifies machine_install as well as adds protections
when devservers are flaky to provisioning.

1) I simplified machine_install's repair logic by having it in one place
rather than scattered throughout machine_install and using the same
pre-condition state for all machine_install's (rather than a heavier one just
for repair).

2) Added a loop around the actual full update. Basically, I've added a #
of attempts for update. If an update fails, I will run a health check against
its devserver and re-attempt on another devserver if it's failed. This includes
re-staging the image on the other devserver as well as updating from it.

BUG=chromium:470581,chromium:471294
TEST=Moblab + Dynamic Suite + Instrumented Code to fail Autoupdate once to
check fallback + unittests in autoupdater.py + pyflakes/pylint on all
affected files.

Change-Id: I5b73a41db32ec832202f6f09f51604b62640b2b6
Reviewed-on: https://chromium-review.googlesource.com/267121
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
3 files changed