Codelab: Finding UI elements using chrome.automation API

A common task in autotests is to make hardware changes and verify that UI gets updated or interact with UI elements and verify that hardware is updated. We can use the chrome.automation API to help with both of these tasks.

Getting familiar with chrome.automation API

Detailed information about chrome.automation API can be found at https://developer.chrome.com/extensions/automation.

In short, the API is a wrapper around Chrome's hierarchy of accessibility nodes that describe Chrome UI elements. The most important attributes of accessibility nodes are role and name. See the section on Accessibility Attributes of the accessiblity overview.

Setup

Follow the steps in Loading autotest extension on device. Loading the AutotestPrivate extension will give you access to chrome.automation API as well.

To find a specific UI element

Load a js console connected to the autotest extension‘s background page. See the previous section for steps on how to connect to the extension’s background page.

NOTE: The following steps are meant to be run interactively in the console and will not work if used in a real test. Section Using chrome.automation in autotests shows how to use the API in a real test.

Let's start by grabbing a reference to the root node of the accessibility tree.

var root;
chrome.automation.getDesktop(r => root = r);

Finding a button in the hierarchy

Let's demonstrate how to simulate a click on the launcher button in the system shelf.

We'll start by listing all buttons visible in the tree.

root.findAll({attributes: {role: "button"}}).map(node => node.name);

After typing that into the console you should get a response such as this:

> (7) ["Back", "Launcher", "Chromium", "Stylus tools", "Status tray, time 4:21
PM, Battery is 22% full.", "Connected to Ethernet", "Battery is 22% full. Time
 left until battery is empty, 1 hour and 39 minutes"]

NOTE: Names will change depending on the locale of the device. We currently don't have a locale independent way of identifying UI nodes.

Just by looking at button names we can easily guess that the button named “Launcher” is the one we're looking for.

Finally, to simulate a click on our button:

var launcher = root.find({attributes: {role: "button", name: "Launcher"}});
launcher.doDefault();

The doDefault method performs an action based on the node's role, which for buttons is a button click.

The find method supports multiple attributes filters. It returns UI elements that satisfy all conditions.

Important roles

The API supports interactions with many types of UI elements.

The following table contains chrome.automation roles for common UI elements:

views class	chorme.automation role
views::Button	button
views::Label	staticText
views::ImageView	image
views::TextField	textField

Finding name and role of a view subclass

View subclasses override the GetAccessibleNodeData method to provide role and name information.

For example, look at views::Button::GetAccessibleNodeData.

Using chrome.automation in autotests

chrome.automation extension can be accessed through the autotest extension.

with Chrome.chrome(autotest_ext=True) as cr:
    ext = cr.autotest_ext
    ext.ExecuteJavaScript("""
        chrome.automation.getDesktop(root => {
            var launcher = root.find({attributes: {role: 'button', name: 'Launcher'}});
            launcher.doDefault();
        });
    """)