Welcome to the Build API.
The Build API is a CLI-only, proto based API to execute build steps. It was created to provide a stable interface for the CI builders. The proto files (in chromite/infra/proto) define the services/RPCs provided by the API. The modules in controller/ are the entry points for the RPCs defined in the proto files. The Build API is invoked via the build_api
script, which takes 4 arguments; the name of the endpoint being called (e.g. chromite.api.SdkService/Create), and the input, output, and optional config protos, which are provided as paths to files containing the respective JSON or protobuf-binary encoded messages.
To manually call an endpoint, e.g. for testing, the gen_call_scripts process is recommended, it makes calling a specific endpoint much simpler. Please also contribute new example input files when you add new endpoints!
The overall process is simple whether you want to do it manually or in a script, though. You'll need to build out an instance of the request message and write it to a file, and then just call the build_api
script.
The only tricky part is getting the compiled protobuf. If you‘re working in recipes or in chromite the problem has already been addressed. Otherwise, you’ll need to figure out a process that works for you depending on your language and purpose. For immediate, local work, compiling the proto with protoc should be relatively straightforward, but for production services consulting the CrOS CI team may be worthwhile.
This section contains information for developers contributing to the Build API.
The Build API has some special, automatic functionality that triggers for a few specific proto messages/fields. These are meant to facilitate some specific aspects of the Build API and reduce boilerplate code.
The service and method options extensions in chromite/api/build_api.proto
define some key information about the implementation of the endpoint.
The first are the module
and implementation_name
fields, in the service and method options respectively, that tell the Build API how to call the endpoint. The service‘s module
option is required, and defines the name of the controller module where the service’s RPCs are implemented. The method‘s implementation_name
field is optional, and defines the name of the function in the service’s controller module for the RPC. When the implementation_name
is not given, the API expects the function to have the same name as the RPC in the proto.
The two options extensions also define the service_chroot_assert
and method_chroot_assert
fields, respectively. These two fields allow defining whether the endpoint must run INSIDE
or OUTSIDE
the chroot, or if either is fine when not set. The service‘s option is the default for all of its RPCs, and the RPC’s method option overrides it when set.
INSIDE
chroot endpointsAt first the INSIDE
behavior can be somewhat confusing, but is designed to reduce boilerplate by automating chroot interactions. When writing an endpoint that needs to run inside the chroot, setting the chroot assertion to INSIDE
is not required, but is strongly recommended for the sake of simplifying the implementation and standardizing the chroot interactions.
All endpoints that do use the INSIDE
functionality must have a chromiumos.Chroot
field, but are otherwise free to use whatever it needs. The Build API parses the chromiumos.Chroot
field, removes it from the input, then executes the endpoint inside the chroot. The endpoint's implementation can be written as if it is always inside the chroot, because while Build API invocations are always made from outside the chroot, it ensures the implementation does not even get imported until after it has entered the chroot.
INSIDE
Chroot Path
UtilitiesThe automatic chroot handling means inserting and extracting artifacts is not possible manually. This gap has been filled by the Path
, ResultPath
, and SyncedDir
messages, defined in chromiumos/common.proto
, that allows the implementations to always behave as if they are working with local files without considering chroot pathing implications.
The Path
message tells the Build API a path, and whether the path is inside or outside of the chroot. A Path
message in a request can be used to inject a file or folder into the chroot for the endpoint to use. Before entering the chroot, the Build API copies the path into a temporary directory inside the chroot, and changes the path in the request sent to the inside-chroot invocation of the endpoint to point to that inside chroot path. For example, if you need /working/directory/image.bin
for an endpoint inside the chroot, the Build API will create a /path/to/chroot/tmp/rand-tmp-dir/
, copy in image.bin
, and then the implementation running inside the chroot will be given /tmp/rand-tmp-dir/image.bin
.
The ResultPath
message provides a similar functionality for extracting paths from the chroot. TheResultPath
message itself must be defined in the request, and is analogous to passing a function an output directory. To use, simply set the paths of the response Path
messages to the files or directories that need to be extracted from the chroot. After the endpoint execution completes, all Path
messages in the response are copied into the given result path, and the paths in the response are updated to reflect their final location. Worth noting, the implementation does not require gathering artifacts to a specific location inside the chroot, the Build API will handle gathering the files into the ResultPath outside the chroot.
The SyncedDir
message in a request provides a blanket, bidirectional sync of a directory. Any files present in the specified directory are copied into a temp directory in the chroot before it is executed, then after it finishes the source directory is emptied, and all files in the chroot directory are copied out to the source directory. This message can be useful for situations where the directory structure or contents is not necessarily important, for example, setting a process' log directory to the SyncedDir
path allows extracting all the log files.
Make sure you've consulted the Build and CI teams when considering making breaking changes that affect the Build API.
This directory is a separate repo that contains all of the raw .proto files. You will find message, service, and method definitions, and their configurations. You will need to commit and upload the proto changes separately from the chromite changes.
When making changes to the proto, you must:
chromite/infra/proto/generate.sh
.chromite/api/compile_build_api_proto
.Cq-Depend
.At time of writing, the PCQ does not support Cq-Depend:
between the infra/proto and chromite repos, so it must be handled manually.
This repo was and will be pinned to a specific revision in the manifest files when we get closer to completing work on the Build API. For the speed we're moving at now, though, having that revision pinned and updating the manifest has caused far more problems than its solving.
When we do go back to the pinned revision:
The generated protobuf messages.
Do not edit files in this package directly!
The proto can be compiled using the compile_build_api_proto
script in the api directory. The protoc version is locked and fetched from CIPD to ensure compatibility with the client library in third_party/
.
This directory contains the entry point for all of the implemented services. The protobuf service module option (defined in build_api.proto) references the module in this package. The functions in this package should all operate as one would expect a controller to operate in an MVC application - translating the request into the internal representation that's passed along to the relevant service(s), then translates their output to a specified response format.
This directory contains scripts that may not be 100% supported yet. See contrib/README.md
for information about the scripts.