client/site_tests/suite_HWQual/README.txt

Copyright (c) 2010 The Chromium OS Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.


This document describes the steps to go through in order to run Chrome OS
hardware qualification on a device under test.

================================================================================
Glossary
================================================================================

- $: command line prompt
- $HOME: home directory of current user.
- AC: alternating current, implies device is not powered by battery.
- DUT: device under test
- Semi-Automated Test: test that runs with automation but requires manual
  intervention.

================================================================================
Test Setup
================================================================================

- Setup a Linux machine to serve as the Autotest server. The Autotest
  server requires Python, Wireless access to the DUT and basic Linux
  shell utilities. The setup has been tested on Ubuntu 9.10 available
  for download at http://www.ubuntu.com/getubuntu/download/.


- Create an installation directory on the Autotest server for the
  Chrome OS hardware qualification package. The rest of the
  instructions assume that you're installing the package in the
  current user home directory ($HOME/).


- Contact your Google technical support person and download the Chrome
  OS hardware qualification package chromeos-hwqual-TAG.tar.bz2 for
  your device in $HOME/.


- Install the package on the server:

  $ cd $HOME/ && tar xjf chromeos-hwqual-TAG.tar.bz2


- Install the Chrome OS test image on the DUT. The USB test image is
  available in:

  $HOME/chromeos-hwqual-TAG/chromeos-hwqual-usb.img

  Here are sample steps to install the test image.

  - Plug a USB storage device into the Autotest server. Note that all
    data on your USB stick will be destroyed.

  - Unmount any mounts on the USB device.

  - Copy the USB image to a USB storage device by executing:

    $ sudo dd if=$HOME/chromeos-hwqual-TAG/chromeos-hwqual-usb.img \
              of=/dev/sdX

  - where /dev/sdX is your USB device.

  - Plug the USB device into the DUT and boot from it.

  - Log in to Chrome OS.  Start the Chrome OS shell by pressing Ctrl-Alt-T.
    Install Chrome OS on the DUT:

    crosh> install


- Cold boot the DUT -- turn the DUT off and then back on. This ensures
  a consistent starting point for the qualification tests and allows
  the system to collect cold boot performance metrics. Make sure you
  don't boot from USB.


- Connect the DUT to the network and note its IP address <DUT_IP>. The
  IP address is displayed at the bottom of the network selection
  menu. Unless specified explicitly, the test setup works correctly
  through either wireless or wired network connections.


- Add the DUT root private key to ssh-agent on the Autotest server:

  $ ssh-add $HOME/chromeos-hwqual-TAG/testing_rsa

- If ssh-add fails saying that it cannot connect to your authentication agent,
  retry the command after running:

  $ eval `ssh-agent -s`

- These commands allow the Autotest server to connect and login as root on the
  DUT.


- Make sure you can ssh as root to the DUT from the Autotest
  server. The command below should print 0.

  $ ssh root@<DUT_IP> true; echo $?

================================================================================
Automated and Semi-Automated Test Runs
================================================================================

- Unless otherwise noted, all tests can be performed on an AC-powered DUT.

- Go to the Autotest server directory and clean up previous test results.

  $ cd $HOME/chromeos-hwqual-TAG/autotest/
  $ rm -rf results.*


- Run the fully automated client-side tests:

  $ ./server/autoserv -r results.auto -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.auto


- Plug high-speed high-capacity storage devices in all USB and SD Card
  slots and run the external storage test:

  $ ./server/autoserv -r results.external_devices -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.external_drives


- Run the system suspend/resume stability test:

  $ ./server/autoserv -r results.suspend_resume -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.suspend_resume


- If the DUT has video out ports, run the Video Out semi-automated
  test by following the instructions specified in the control file
  (control.video_out) and then executing:

  $ ./server/autoserv -r results.video_out.${PORT} -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.video_out

- Where PORT is the name of each video port you are testing.  For
  example, if the DUT has one HDMI and one VGA out port, run:

  $ ./server/autoserv -r results.video_out.hdmi1 -m <DUT_IP> \
                -c client/site_tests/suite_HWQual/control.video_out

  $ ./server/autoserv -r results.video_out.vga1 -m <DUT_IP> \
                -c client/site_tests/suite_HWQual/control.video_out


- Run the graphics tearing test:

  $ ./server/autoserv -r results.teartest -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.teartest

- Run audio test, with built-in speakers and microphone

  $ ./server/autoserv -r results.audio -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.audio

- Plug-in headphone and microphone, run audio test again

  $ ./server/autoserv -r results.audio_ext -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.audio

- Run the Keyboard test : 
  (Wait several seconds after running the test. Then strike the "Search" key,
   e.g. the key above Left Shift and below Tab)

  $ ./server/autoserv -r result.keyboard -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.keyboard

- Run the DUT on AC. Probe the AC driver:

  $ ./server/autoserv -r result.probe_ac -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.probe_ac

- Run the DUT on battery. Probe the battery driver:
  (If you just unplugged AC, please wait for a second before running
   the test for kernel updating power status.)

  $ ./server/autoserv -r result.probe_bat -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.probe_bat

- Run the DUT on AC. Plug a power draw USB dongle in each USB port.
  Run the max power draw test:

  $ ./server/autoserv -r results.max_power_draw.ac -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.max_power_draw

- Run the DUT on battery. Plug a power draw USB dongle in each USB
  port. Run the max power draw test:

  $ ./server/autoserv -r results.max_power_draw.batt -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.max_power_draw

- Run the DUT on AC. Run the power settings test:

  $ ./server/autoserv -r results.power_x86_setting.ac -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.power_x86_settings

- Run the DUT on battery. Run the power settings test:

  $ ./server/autoserv -r results.power_x86_setting.batt -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.power_x86_settings

- Make sure the remaining battery charge is less than 5%. Note that the test
  will check and fail quickly if the initial battery charge is more than 5%.
  Run the DUT on AC. Run the battery charge test:

  $ ./server/autoserv -r results.battery_charge_time -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.battery_charge_time

- Make sure that the battery is fully charged. Note that the test will not
  check if the battery is fully charged before running. Run the DUT on
  battery. Run the battery load test by first following the manual
  instructions specified in the control file (control.battery_load)
  and then executing:

  $ ./server/autoserv -r results.battery_load -m <DUT_IP> \
                  -c client/site_tests/suite_HWQual/control.battery_load

================================================================================
Reviewing Automated and Semi-Automated Test Results
================================================================================

- Autotest logs progress and performance data in results.* directories
  specified through the '-r' autoserv option. The easy way to see a summary
  of your test results is to use the 'generate_test_report'
  script installed under $HOME/chromeos-hwqual-TAG/:

  $ ../generate_test_report results.*

  This will display a table with test status and perfomance data for
  all result directories.

- If deeper investigation into a failure is required, you can review
  the debug information stored in results:

  $ ls */*/debug/.

================================================================================
Manual Test Runs
================================================================================

- Perform the manual tests specified in
  $HOME/chromeos-hwqual-TAG/manual/testcases.csv.

- Please note that some tests cannot be tested as they rely on
  features not yet implemented.  They are being included as a preview for
  manual tests that will be required.  Such tests will have
  "NotImplemented" in their "LABELS" column.

- Update this spreadsheet with a column of pass/fail results with any notes
  which may be useful.

================================================================================
Reporting Results
================================================================================

- Upon completing automatic, semi-automatic, and manual test runs, you should
  report your results to a Google technical support contact to verify the
  tests were run correctly and to help diagnose failures.  Verify you have
  updated your manual test spreadsheet as described above and copy it into
  your autotest output directory:

  $ cd $HOME/chromeos-hwqual-TAG/autotest/
  $ cp ../manual/testcases.csv .

- Package all results into a tar file:

  $ tar cjf chromeos-hwqual-results-TAG-DATE.tar.bz2 results.* testcases.csv

- Send the resulting chromeos-hwqual-results file to your technical support
  contact.