retry_util: Force curl to use HTTP 1.1

This CL forces curl to use HTTP 1.1 due to intermittent error: "Error in
the HTTP2 framing layer".

BUG=b/171561057
BUG=chromium:1141133
TEST=./run_pytest ./lib/retry_util_unittest.py
TEST=./bin/cros_sdk --create --nouse-image

Change-Id: Icd6b57702a0bbc83845f4552d37d07ac7536f08c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/2491264
Commit-Queue: Conor McNamara <ctmcnamara@chromium.org>
Tested-by: Conor McNamara <ctmcnamara@chromium.org>
Reviewed-by: Andrey Pronin <apronin@chromium.org>
Reviewed-by: Stephane Belmon <sbelmon@google.com>
(cherry picked from commit e1747bcfcc1e939650ebccaa2a2217358a6dd496)
1 file changed
tree: 522afdbaa520f84c7a1744cf5f2f1bd90861f44c
  1. .vscode/
  2. api/
  3. appengine/
  4. bin/
  5. bootstrap/
  6. cbuildbot/
  7. cidb/
  8. cli/
  9. config/
  10. contrib/
  11. cros/
  12. cros_bisect/
  13. infra/
  14. lib/
  15. licensing/
  16. mobmonitor/
  17. scripts/
  18. service/
  19. signing/
  20. ssh_keys/
  21. third_party/
  22. utils/
  23. venv/
  24. .dir-locals.el
  25. .env
  26. .gitignore
  27. .style.yapf
  28. .vpython
  29. __init__.py
  30. AUTHORS
  31. codereview.settings
  32. COMMIT-QUEUE.ini
  33. LICENSE
  34. OWNERS
  35. OWNERS.build
  36. OWNERS.ci
  37. PRESUBMIT.cfg
  38. pylintrc
  39. README.chromium
  40. README.md
  41. run_tests
README.md

Chromite Development: Starter Guide

Objective

This doc tries to give an overview and head start to anyone just starting out on Chromite development.

Background

Before you get started on Chromite, we recommend that you go through ChromeOS developer guides at external (first) and then goto/chromeos-building for internal. The Gerrit starter guide may also be helpful. You should flash a built image on a test device (Ask around for one!).

Chromite was intended to be the unified codebase for anything related to building ChromeOS/ChromiumOS. Currently, it is the codebase responsible for several things including: building the OS from the requisite packages for the necessary board (parallel_emerge), driving the infrastructure build workflow (CBuildBot), hosting a Google App Engine App, and providing utility functions for various scripts scattered around ChromeOS repositories. It is written for the most part in Python with some Bash sprinkled in.

Directory Overview

You can use Code Search to lookup things in Chromite or ChromeOS in general. You can add a ChromeOS filter to only show files from CrOS repositories by going to CS Settings and adding a new Saved query: “package:^chromeos” named “chromeos”.

chromite/api

The Chromite API for the CI system. The API exposes a subset of the chromite functionality that needs to be strictly maintained as much as possible.

chromite/cbuildbot

CBuildBot is the collection of entire code that runs on both the parent and the child build machines. It kicks off the individual stages in a particular build. It is a configurable bot that builds ChromeOS. More details on CBuildBot can be found in this tech talk (slides).

chromite/cbuildbot/builders

This folder contains configurations of the different builders in use. Each has its own set of stages to run usually called under RunStages function. Most builders used regularly are derived from SimpleBuilder class.

chromite/cbuildbot/stages

Each file here has implementations of stages in the build process grouped by similarity. Each stage usually has PerformStage as its primary function.

chromite/lib

Code here is expected to be imported whenever necessary throughout Chromite.

chromite/scripts

Unlike lib, code in scripts will not and should not be imported anywhere. Instead they are executed as required in the build process. Each executable is linked to either wrapper.py or virtualenv_wrapper.py. Some of these links are in chromite/bin. The wrapper figures out the directory of the executable script and the $PYTHONPATH. Finally, it invokes the correct Python installation by moving up the directory structure to find which git repo is making the call.

chromite/service

These files act as the centralized business logic for processes, utilizing lib for the implementation details. Any process that's implemented in chromite should generally have an entry point somewhere in a service such that it can be called from a script, the API, or anywhere else in lib where the process may be useful.

chromite/third_party

This folder contains all the third_party python libraries required by Chromite. You need a very strong reason to add any library to the current list. Please confirm with the owners beforehand.

chromite/utils

This folder contains smaller, generic utility functionality that is not tied to any specific entities in the codebase that would make them more at home in a lib module.

chromite/infra

This folder contains the chromite-specific infra repos.

chromite/*

There are smaller folders with miscellaneous functions like config, licencing, cidb, etc.

Testing your Chromite changes

Before any testing, you should check your code for lint errors with:

$ cros lint <filename>

Unit Tests

Every Python file in Chromite is accompanied by a corresponding filename_unittest.py file. More on unit tests here. Once written, the unit tests can be run using ./run_tests command in the Chromite directory. To test a specific file (say lib/triage_lib.py), use

~/trunk/chromite $ ./run_tests lib/triage_lib_unittest

Run_tests without any argument runs all unit tests in Chromite. These unit tests are run in tryjobs, preCQ and CQ as well.

If you have to create a new Python file in Chromite, you should also create a {filename}_unittest.py in the same directory with all the unit tests. Also make a link called {filename}_unittest to /mnt/host/source/chromite/scripts/wrapper.py. See the other unittest files around if unclear.

Tryjob

You can also fire a build on a server (or even locally) to have an entire build happen similar to how it would in Commit Queue.

$ cros tryjob -g <gerrit-change-id> <trybot-config>
$ cros tryjob -h -> for help on more options

Add --hwtest to add hardware testing to your tryjob. You can use the link provided by the command to check the status of your tryjob. Alternatively, you can go to the CI UI tryjobs page and filter results by your email.

Pre-CQ

Once you mark your CL as Commit-Queue +1 on Chromium Gerrit, the PreCQ will pick up your change and fire few preset config runs as a precursor to CQ. Currently, it doesn’t include any hardware or VM testing (for now!).

Commit Queue

This is the final step in getting your change pushed. CQ is the most comprehensive of all tests. There are a multitude of CL's being validated in the same CQ. Once a CL is verified by CQ, it is merged into the codebase.

How does ChromeOS build work?

Refer to these talk slides on ChromeOS Build Overview.