cheets_CTS/GTS: Mitigation for a broken download cache.
This is a tentative workaround until we find a fundamental way to avoid
breaking the cache. When a cache entry with empty content (which should
happen normally) is found, the file is not considered as cached and
re-downloaded.
BUG=chromium:800657
TEST=run cheets_CTS_N after manually deleting the cached adb binary.
Change-Id: I6a53c3f5f875bc5ebf5f68d195c68ff5509c71a9
Reviewed-on: https://chromium-review.googlesource.com/875525
Tested-by: Kazuhiro Inaba <kinaba@chromium.org>
Trybot-Ready: Kazuhiro Inaba <kinaba@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
(cherry picked from commit a62cc8f967c51de3c373601e0819eb871db152c0)
Reviewed-on: https://chromium-review.googlesource.com/877359
Reviewed-by: Kazuhiro Inaba <kinaba@chromium.org>
diff --git a/server/cros/tradefed_test.py b/server/cros/tradefed_test.py
index 7f97f96..f614578 100644
--- a/server/cros/tradefed_test.py
+++ b/server/cros/tradefed_test.py
@@ -790,9 +790,15 @@
# instead of file existence, so that _install_bundle can delete original
# zip files to save disk space.
if os.path.exists(output_dir):
- logging.info('Skipping download of %s, reusing content of %s.', uri,
- output_dir)
- return output
+ # TODO(crbug.com/800657): Mitigation for the invalid state. Normally
+ # this should not happen, but when a lock is force borken due to
+ # high IO load, multiple processes may enter the critical section
+ # and leave a bad state permanently.
+ if os.listdir(output_dir):
+ logging.info('Skipping download of %s, reusing content of %s.',
+ uri, output_dir)
+ return output
+ logging.error('Empty cache entry detected %s', output_dir)
self._safe_makedirs(output_dir)
if parsed.scheme not in ['gs', 'http', 'https']:
