init: Give mount-encrypted umount 10 times to retry

There is a lazy umount -n -l /var/run /var/lock before mount-encrypted
umount. But since it is a lazy umount, we don't know when it really
clears all reference points.
By experiment, some delay is needed to make mount-encrypted umount to
succeed. Without this delay, it will fail with fail rate 1 out of 10.
This CL lets mount-encrypted umount to retry 10 times with 0.1 seconds delay
between each retries to make it succeed earlier with certain delay.

BUG=chrome-os-partner:21345
TEST=run shutdown -r now, check there is no
/mnt/stateful_partition/shutdown_stateful_umount_failure
nor /mnt/stateful_partition/umount-encrypted.log.
Redo the test for 1000 times.

Original Change-Id: Ic598d045d53525a1d39e6b4f87d1cb4fdecb8bf5
Original Reviewed-on: https://chromium-review.googlesource.com/168911
Original Reviewed-by: Kees Cook <keescook@chromium.org>
Original Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Original Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Change-Id: I9945744607008a0326f517aa2c9afe37fa8c9a8a
Reviewed-on: https://chromium-review.googlesource.com/219007
Reviewed-by: Kees Cook <keescook@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
diff --git a/init/chromeos_shutdown b/init/chromeos_shutdown
index 5163a1b..01ef67d 100755
--- a/init/chromeos_shutdown
+++ b/init/chromeos_shutdown
@@ -65,8 +65,18 @@
 # Lazy umount the tmpfs dirs under /var so we can immediately umount it.
 umount -n -l /var/run /var/lock
 if [ -e /dev/mapper/encstateful ]; then
-  mount-encrypted umount >/tmp/umount-encrypted.log 2>&1
-  if [ $? -ne 0 ] ; then
+  # Give mount-encrypted umount 10 times to retry, otherwise
+  # it will fail with 'device is busy' because lazy umount does not finish
+  # clearing all reference points yet. Check crosbug.com/p/21345.
+  for i in 1 2 3 4 5 6 7 8 9 10; do
+    mount-encrypted umount > /tmp/umount-encrypted.log 2>&1
+    rc=$?
+    if [ $rc -eq 0 ]; then
+      break
+    fi
+    sleep 0.1
+  done
+  if [ $rc -ne 0 ] ; then
     mv /tmp/umount-encrypted.log /mnt/stateful_partition/
   fi
   umount -n /usr/local /home /mnt/stateful_partition