toolchain-utils: provide option to migrate nightly tests to recipe

This patch provides two new options to buildbot_test_toolchain script,
which launches nightly performance tests for toolchain.

Option `--recipe` introduces new location and mechanism to find latest
image from recipe bucket. The recipe builder for each board monitored by
toolchain team builds an image every day and store results to certain gs
locaton. When this option specified, the script no longer launches
tryjob to get a new image but tries to access the lastest existing image
in the last two days from the gs location.

Option `--test` makes it easier to test locally, it blocks all locations
in chrotomation machine and do not send or store unnecessary results
after tests finish.

A small change to lock_machine.py fixes an issue while related to
swarming when testing this change.

BUG=chromium:1071271
TEST=./buildbot_test_toolchains.py --board=lulu
     --remote=chromeos2-row9-rack8-host7.cros
     --chromeos_root=$chromiumos --test=True --recipe=True

Change-Id: I724a31ce174955c1f002bb7e330919f23535e105
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/toolchain-utils/+/2151717
Reviewed-by: Tiancong Wang <tcwang@google.com>
Tested-by: Zhizhou Yang <zhizhouy@google.com>
Auto-Submit: Zhizhou Yang <zhizhouy@google.com>
Commit-Queue: Zhizhou Yang <zhizhouy@google.com>
4 files changed