This README is a continuation of Hello World Tutorial Part 4, creating a “Hello world!” endpoint from scratch. This tutorial assumes you have already executed all the steps in Parts 1-4, and have all the code in place.
In this tutorial, we'll be looking at how to implement API endpoints that execute inside the SDK (a.k.a. the chroot), and the utilities the Build API provides to do so.
If your endpoint depends on executing code inside the SDK, then you will need this functionality. Generally, we expect most endpoints to run inside the chroot, since the build itself should be taking place in the chroot. Existing endpoints in this category include things like building/installing packages (SysrootService/InstallPackages
, a.k.a. build_packages
), and running the ebuild tests (TestService/BuildTargetUnitTest
, a.k.a. cros_run_unit_tests
).
If the functionality you‘re adding an endpoint for can be entirely executed from outside the chroot, then you won’t need this functionality. These types of endpoints are most often setup or teardown operations, since the build itself should be inside the SDK. Existing endpoints in this category include scripts that actually execute outside of the chroot, such as making the chroot itself (SdkService/Create
, a.k.a. cros_sdk
), operations we run as soon as possible to reduce time spent on irrelevant builds (e.g. PackagesService/Uprev
), and cleanup steps (e.g. SdkService/Clean
).
If you‘re ever unsure where yours should run, we’re happy to help!
The first step is just getting inside. In Part 1 we talked about the service and method options for configuring the endpoints modules and functions that get executed when calling the endpoints. These options also contain fields for configuring whether the endpoint gets run inside or outside of the chroot. The service_options
has the service_chroot_assert
field, and the method_options
the method_chroot_assert
field. These fields are enums that direct the API how to execute the endpoint. The functionality requires no additional imports, the fields just need to be added and set.
The service option allows setting the default for all endpoints in the service, but when both are set, the method option will take precedence over the service option. A value MUST be set if the endpoint does not execute properly both inside and outside of the SDK. If the endpoint is capable of executing in either environment, then the field does not need to be set.
The other important note is that the API needs to know which chroot to enter. Every endpoint that enters or uses the chroot MUST include a Chroot message field in the request. See the updates below for the new import and an example of the new field in the request. However, when you‘re running the endpoints locally, it will mostly likely work without populating the chroot field. When not populated, the Build API will use the default chroot location. If you’re a person for whom the default will not work locally, you probably already know because you had to manually set it up that way. If you care to check it out, the Chroot message definition is in infra/proto_branched/src/chromiumos/common.proto
.
With all that in mind, let's update our endpoint to execute inside the SDK.
infra/proto_branched/src/chromite/api/hello.proto
:
// Proto config. syntax = "proto3"; package chromite.api; option go_package = "go.chromium.org/chromiumos/infra/proto/go/chromite/api"; import "chromite/api/build_api.proto"; // NEW! import "chromiumos/common.proto"; // HelloService/Hello request and response messages. message HelloRequest { string target = 1; // NEW! chromiumos.Chroot chroot = 2; } message HelloResponse { string hello_message = 1; } service HelloService { option (service_options) = { module: "hello", // NEW! service_chroot_assert: INSIDE, }; rpc Hello(HelloRequest) returns (HelloResponse); }
And now let's regenerate the protobuf bindings.
$> cd ~/chromiumos/infra/proto_branched $> ./generate.sh $> cd ~/chromiumos/chromite/api $> ./compile_build_api_proto
That's it! Our endpoint will now always execute inside the chroot.
Let‘s change our endpoint again. Instead of passing the target as a string, let’s pass a file that contains multiple targets, one per line. In addition to printing out the messages, let's also write them out, one message per file, each named after their target.
The Build API has specific messages that are used to support injecting and extracting files and folders from the SDK. We'll use some of these here to implement our new functionality.
infra/proto_branched/src/chromite/api/hello.proto
:
// Proto config. syntax = "proto3"; package chromite.api; option go_package = "go.chromium.org/chromiumos/infra/proto/go/chromite/api"; import "chromite/api/build_api.proto"; import "chromiumos/common.proto"; // HelloService/Hello request and response messages. message HelloRequest { // NEW! reserved "target"; // NEW! reserved 1; chromiumos.Chroot chroot = 2; // NEW! chromiumos.Path targets_file = 3; // NEW! chromiumos.ResultPath output_dir = 4; } message HelloResponse { // NEW! reserved "messages"; // NEW! reserved 1; // NEW! repeated chromiumos.Path message_files = 2; } service HelloService { option (service_options) = { module: "hello", service_chroot_assert: INSIDE, }; rpc Hello(HelloRequest) returns (HelloResponse); }
First thing to note is we deprecated the target and message fields from the request and response, respectively, by reserving the field name and number. In the request, we also added chromiumos.Path
and chromiumos.ResultPath
fields, and in the response, a repeated chromiumos.Path
field.
The Path
message family are used by the Build API to automatically inject artifacts (i.e. files and folders) into and extract artifacts from the SDK. The functionality can handle both files and folders in both directions. There is also a bidirectional sync with the SyncedDir
. For more details, see the chroot reference.
We will be using the targets_file
field in the request to inject a file containing targets, replacing the target field we previously had. The endpoint will then print the messages as it did before, but also write each message to a separate file, and return those paths in the message_files
field, each named for their target. Those files will all be collected in the path we specify in the output_dir
field.
With our proto updated, let's change our lib to write out the message to a file and return the file path rather than the message itself.
chromite/lib/hello_lib.py
import os from chromite.lib import osutils def hello(target, output_dir): msg = f'Hello, {target}!' print(msg) file_path = os.path.join(output_dir, target) osutils.WriteFile(file_path, msg) return file_path
Finally, let's update the controller to use the new fields in the request, and fix the hello_lib call now that it has a new signature.
from chromite.api import faux from chromite.api import validate from chromite.api.gen.chromiumos import common_pb2 from chromite.lib import hello_lib from chromite.lib import osutils def _hello_success(_request, response, _config): file_proto = response.message_files.add() file_proto.path = '/tmp/target' file_proto.location = common_pb2.Path.OUTSIDE @faux.success(_hello_success) @faux.empty_error @validate.exists('targets_file.path') @validate.validation_complete def Hello(request, response, _config): # Read targets, one per line. targets = osutils.ReadFile(request.targets_file.path).splitlines() # Don't delete since we need it to exist afterwords. with osutils.TempDir(delete=False) as tmp_dir: for target in targets: # Print the greeting and get the file it created. target_file = hello_lib.hello(target=target, output_dir=tmp_dir) # Add it to the response. file_proto = response.message_files.add() file_proto.path = target_file file_proto.location = common_pb2.Path.INSIDE
First, we changed the @validate
decorator to exists
, which verifies a file exists. We then read the targets files, and call the hello_lib for each target, adding the results to the response as we go. Finally, the faux success function needs to reflect our new response.
First, we'll regenerate the compiled proto to pick up our changes.
$> cd ~/chromiumos/infra/proto_branched $> ./generate.sh --allow-breaking $> cd ~/chromiumos/chromite/api $> ./compile_build_api_proto
Note that this time we invoked ./generate.sh
with flag --allow-breaking
. This is necessary because we modified existing fields in our proto, namely target
and messages
in HelloRequest
and HelloResponse
, respectively. By default, ./generate.sh
checks for breaking protobuffer changes, and fails if an existing field is deleted. Flag --allow-breaking
overrides this behavior.
Next we need to create the targets file for the request, and our output directory.
$> printf "world\nmoon\neverybody" > /tmp/targets-file $> mkdir /tmp/hello-results
Now we can set up our new input file, pointing to those locations. The 2 used for the location fields in our input file below is just the enum value for Path.Location.OUTSIDE
.
~/chromiumos/chromite/api/contrib/call_scripts/hello__hello_input.json
{ "targets_file": {"path": "/tmp/targets-file", "location": 2}, "output_dir": {"path": {"path": "/tmp/hello-results", "location": 2}} }
We're all set, now we just run it.
$> cd ~/chromiumos/chromite/api/contrib/call_scripts/ $> ./hello__hello Running chromite.api.HelloService/Hello 15:17:13: DEBUG: Services registered successfully. 15:17:13: INFO: Re-executing the endpoint inside the chroot. 15:17:13: DEBUG: Copying /tmp/targets-file to /usr/local/google/home/saklein/chromiumos/chroot/tmp/tmptdazkiqh/targets-file 15:17:13: INFO: Writing input message to: /usr/local/google/home/saklein/chromiumos/chroot/tmp/tmpx9cumt3j/request 15:17:13: INFO: Writing config message to: /usr/local/google/home/saklein/chromiumos/chroot/tmp/tmpx9cumt3j/config 15:17:13: INFO: run: cros_sdk --chroot /usr/local/google/home/saklein/chromiumos/chroot -- build_api chromite.api.HelloService/Hello --input-json /tmp/tmpx9cumt3j/request --output-binary /tmp/tmpx9cumt3j/response --config-json /tmp/tmpx9cumt3j/config --debug 15:17:15: DEBUG: Services registered successfully. 15:17:15: DEBUG: Validating targets_file.path exists. Hello, world! Hello, moon! Hello, everybody! 15:17:15: INFO: Endpoint execution completed, return code: 0 15:17:15: DEBUG: Copying /usr/local/google/home/saklein/chromiumos/chroot/tmp/tmpxjuwd_lj/world to /tmp/hello-results/world 15:17:15: DEBUG: Copying /usr/local/google/home/saklein/chromiumos/chroot/tmp/tmpxjuwd_lj/moon to /tmp/hello-results/moon 15:17:15: DEBUG: Copying /usr/local/google/home/saklein/chromiumos/chroot/tmp/tmpxjuwd_lj/everybody to /tmp/hello-results/everybody Completed chromite.api.HelloService/Hello Success! Return Code: 0 Result: { "messageFiles": [ { "location": 2, "path": "/tmp/hello-results/world" }, { "location": 2, "path": "/tmp/hello-results/moon" }, { "location": 2, "path": "/tmp/hello-results/everybody" } ] }
We can also check our output files to make sure they also have our messages.
$> tail /tmp/hello-results/world /tmp/hello-results/moon /tmp/hello-results/everybody ==> /tmp/hello-results/world <== Hello, world! ==> /tmp/hello-results/moon <== Hello, moon! ==> /tmp/hello-results/everybody <== Hello, everybody!
Success!