ChromeOS ML Model Zoo

This is a collection of TFLite models that can be used to benchmark devices for typical ML use cases within ChromeOS. Where applicable, baseline figures are provided to indicate the minimum performance requirements for these models to meet the user experience goals of those use cases.

These models can be easily deployed to /usr/local/share/ml-test-assets on a DUT via the chromeos-base/ml-test-assets package:

emerge-${BOARD} ml-test-assets && cros deploy <DUT> ml-test-assets

The models can be downloaded directly here

Tools

Latency, Max Memory

Latency and maximum memory usage is measured by the TFLite Benchmark Model Tool.

This is installed by default on all ChromeOS test images.

Example usage:

benchmark_model --graph=${tflite_file} --min_secs=20 <delegate options>

Accuracy

Accuracy is measured by the TFLite Inference Diff Tool.

This is installed by default on all ChromeOS test images.

Example usage:

inference_diff_eval --graph=${tflite_file} <delegate options>

Use Cases

Video Conferencing

• Note 1 : These models are CNN based.
• Note 2 : selfie_segmentation_landscape_256x256 has an F16 and F32 variant, indicated by the filename.

The convolution_benchmark_*_144x256 models are production model graphs with randomized weights, so we don't measure the accuracy of these models.

Image Search

Note: These models are CNN based.

Audio Models

Note: These models are running on CPU in production

Note2: While running benchmark_model with following models, add --run_delay=<secs> to simulate audio server behavior.