ChromeOS ML Model Zoo

This is a collection of TFLite models that can be used to benchmark devices for typical ML use cases within ChromeOS. Where applicable, baseline figures are provided to indicate the minimum performance requirements for these models to meet the user experience goals of those use cases.

Please note that the baseline figures and benchmarks provided here are for rough analysis only, the actual requirements for ChromeOS hardware performance may vary both in the specific benchmarks utilized as well as the target. ChromeOS partners should work with their Google contact on specific requirements.

These models can be easily deployed to /usr/local/share/ml-test-assets on a DUT via the chromeos-base/ml-test-assets package:

emerge-${BOARD} ml-test-assets && cros deploy <DUT> ml-test-assets

The models can be downloaded directly here

Tools

Latency, Max Memory

Latency and maximum memory usage is measured by the TFLite Benchmark Model Tool.

This is installed by default on all ChromeOS test images.

Example usage:

benchmark_model --graph=${tflite_file} --min_secs=20 <delegate options>

Accuracy

Accuracy is measured by the TFLite Inference Diff Tool.

This is installed by default on all ChromeOS test images.

Example usage:

inference_diff_eval --model_file=${tflite_file} <delegate options>

Use Cases

Video Conferencing

• Note 1 : These models are CNN based.
• Note 2 : selfie_ have an F16 and F32 variant, indicated by the filename.*

The convolution_benchmark_* and segmentation_benchmark_512x512* models are production model graphs with randomized weights, so we don't measure the accuracy of these models.

Image Search

Note: These models are CNN based.

Audio Models

Note: These models are running on CPU in production

Note2: While running benchmark_model with following models, add --run_delay=<secs> to simulate audio server behavior.