FAFT (Fully Automated Firmware Tests) is a collection of tests and related infrastructure that exercise and verify capabilities of Chrome OS. The features tested by FAFT are implemented through low-level software (firmware/BIOS) and hardware. FAFT evolved from SAFT (Semi-Automated Firmware Tests) and you can locate tests in the FAFT suite in the Autotest tree as directories with the prefix
The founding principles of FAFT are:
To access some of these low-level capabilities, the tests require a servod instance running and executing controls with the help of physical servo board (servo v2, servo v4 with servo micro or servo v4 Type-C)
The servo board is connected directly to the DUT (Device Under Test) to enable access to low-level hardware interfaces, as well as staging areas for backup software (on a USB drive).
The FAFT framework runs the tests with a tool called test that and it is based on a client-server architecture, where the client runs on the DUT and the server runs on the host machine.
The tests may corrupt various states in the EC, firmware, and kernel to verify recovery processes. In these cases you can almost always use FAFT to restore the system to its original state. The FAFT suite of tests can be invoked locally or remotely. This document describes how to set up the local configuration only.
The Chrome OS firmware controls, among other things, the initial setup of the system hardware during the boot process. They are necessarily complicated, providing reliability against various corruption scenarios and security to ensure trusted software is controlling the system. Currently, the purpose of FAFT is to exercise EC firmware and BIOS firmware functionality and performance.
The hardware configuration for running FAFT on a servo v4 Type-A with servo micro includes:
Figure 1 shows a diagram of how to connect the latest debug boards, servoV4 Type-A and servo micro, to the test controller, DUT, and network. It is important to ensure the DUT is powered off before plugging in cables and components to the servo.
Figure 1.Diagram of hardware configuration for a ServoV4 Type-A with servo micro.
Details of servoV4 Type-A with micro connections:
The hardware configuration for running FAFT with a servo v4 type-C includes:
Figure 2 shows a diagram of how to connect a servoV4 Type-C, to the test controller, DUT, and network. It is important to ensure the DUT is powered off before plugging in cables and components to the servo and DUT.
Figure 2.Diagram of hardware configuration for a ServoV4 Type-C.
Details of servoV4 Type-C connections in Figure 2:
Make sure to use the following servo type and configuration for running the faft_pd suite or the faft_cr50 suite (note: the cr50 suite requires special images so is not runnable outside of Google). This setup requires servod to be in “DUAL_V4” mode. You should generally only use this setup for faft_pd and faft_cr50, faft_ec and faft_bios do not expect servod to be in DUAL_V4 mode.
Figure 3.Diagram of hardware configuration for a ServoV4 Type-C with servo micro.
Details about FAFT PD's ServoV4 Type-C + servo micro setup (Figure 3):
To check or upgrade the FW on the servo v4 and servo micro, respectively, before kicking off the FAFT PD suite:
(Deprecated) The following photo shows the details how to connect the older, deprecated servo v2 board to the test controller, test device, and network.
Figure 4.Diagram of hardware configuration for a ServoV2 board.
Details of servo v2 connections:
After the hardware components are correctly connected, prepare and install a test Chromium OS image:
To run FAFT you use the
test_that tool, which does not automatically start a
servod process for communicating with the servo board. Running FAFT is easiest with
test_that running in separate terminals inside the SDK, using either multiple SDK instances (
cros_sdk --enter --no-ns-pid) or a tool such as
screen inside an SDK instance. Before running any tests, go into chroot:
$ sudo servod --board=$BOARDwhere
$BOARDis the code name of the board you are testing. For example:
$ sudo servod --board=eve
firmware_FAFTSetuptest to verify basic functionality and ensure that your setup is correct.
/usr/bin, the syntax is
$ /usr/bin/test_that --board=$BOARD $DUT_IP firmware_FAFTSetup
It is important to note that this syntax will work only if the correct packages for the DUT have been built. To build the packages, which usually takes a few hours, run the following from chroot:
$ ./build_packages --board=$BOARD where
$BOARD is the code name of the board under test
If packages have not been built, the command won't work unless a path to the autotest directory is included in the command as follows:
$ test_that --autotest_dir ~/trunk/src/third_party/autotest/files/ --args="servo_host=localhost servo_port=9999" -b $BOARD $IP $TEST_NAME
A few sample invocations of launching tests against a DUT:
Running FAFT test with test case name
$ /usr/bin/test_that --board=$BOARD $DUT_IP f:.*DevMode/control
Some tests can be run in either normal mode or dev mode, specify the control file
$ /usr/bin/test_that --board=$BOARD $DUT_IP f:.*TryFwB/control.dev
FAFT can install Chrome OS image from the USB when image filename is specified
$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "image=$IMAGE_FILE" f:.*RecoveryButton/control.normal
To update the firmware using the shellball in the image, specify the argument firmware_update=1
$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "image=$IMAGE_FILE firmware_update=1" f:.*RecoveryButton/control.normal
Run the entire faft_bios suite
$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_bios
Run the entire faft_ec suite
$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_ec
Run the entire faft_pd suite
$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_pd
To run servod in a different host, specify the servo_host and servo_port arguments.
$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "servo_host=$SERVO_HOST servo_port=$SERVO_PORT" suite:faft_lv1
To run multiple servo boards on the same servo host (labstation), use serial and port number.
$ sudo servod --board=$BOARD --port $port_number --serial $servo_serial_number
$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "servo_host=localhost servo_port=$port_number faft_iterations=5000" f:.*firmware_ConsecutiveBoot/control
Q: All of my FAFT tests are failing. What should I do?
firmware_FAFTSetupas a single test. Once it fails, check the log and determine which step failed and why.
Q: A few of my FAFT tests failed, but most tests are passing. What should I do?
Q: I still need help. Who can help me?
Q: I got an error while running FAFT:
AutoservRunError: command execution error: sudo -n which flash_ec . What's wrong?
sudo emerge chromeos-ecinside your chroot.
Q: All tests are failing to run, saying that python was not found. What's wrong?
A: This happens when the stateful partition that holds Python is wiped by a powerwash.
It is usually caused by the stateful filesystem becoming corrupted, since Chrome OS performs a powerwash instead of running
fsck like a standard Linux distribution would.
Q: What causes filesystem corruption?
A1: Most cases of corruption are triggered by a test performing an EC reset, because the current sync logic in Autotest doesn't fully guarantee that all writes have been completed, especially on USB storage devices.
A2: If the outer stateful partition (
/mnt/stateful_partition) becomes full, the inner loop-mounted DM device (
/mnt/stateful_partition/encrypted) will encounter write errors, likely corrupting the filesystem.
Note: Running out of space only tends to happens when running FAFT tests that leave the DUT running from the USB disk, and only if the image's stateful partition is too small.
Q: Can I compare the results obtained with a Type-C servo to those obtained with a Type-A servo + micro?