src/cmd/cos_dkms/README.md

# cos-dkms

`cos-dkms` is a tool designed for Google Container-Optimized OS (COS) to
dynamically build, install, and manage kernel modules. It is intended to be a
drop-in replacement for the upstream Dynamic Kernel Module Support (DKMS) tool,
but tailored for the unique environment of COS and GKE.

Key features include:

*   **Cross-compilation**: Compile kernel modules against specific COS kernel
    versions.
*   **Toolchain Orchestration**: Automatically download and use the exact
    compiler toolchain and kernel headers matching the target COS image.
*   **GCS Caching**: Cache built modules and sources in a Google Cloud Storage
    (GCS) bucket to speed up deployments and avoid recompilation.
*   **Module Signing**: Sign compiled modules using local keys or Google Cloud
    KMS to satisfy COS kernel module signature verification.

Code-level documentation is available
[here](https://pkg.go.dev/cos.googlesource.com/cos/tools.git/src/pkg/dkms).

Documentation for the high-level cos-dkms runner script is available
[here](https://cos.googlesource.com/cos/tools/+/refs/heads/master/src/cmd/cos_dkms_runner).

--------------------------------------------------------------------------------

## Table of Contents

1.  [Running cos-dkms](#running-cos-dkms)
2.  [Actions](#actions)
3.  [Configuration (dkms.conf)](#configuration)
    *   [Configuration Examples](#configuration-examples)
4.  [Module Signing](#module-signing)
5.  [Example Workflows](#workflows)
    *   [Workflow 1: Installing Precompiled Modules](#workflow-precompiled-modules)
    *   [Workflow 2: Installing Modules Compiled by Google for COS](#workflow-google-compiled-modules)
    *   [Workflow 3: Installing Modules with modprobe](#workflow-modprobe)
    *   [Workflow 4: Building and Distributing Modules](#workflow-building-distributing)

--------------------------------------------------------------------------------

## Running cos-dkms {#running-cos-dkms}

Because COS is a minimal operating system without a pre-installed compiler
toolchain, `cos-dkms` is intended to be run inside a Docker container. Google
provides a pre-built container image: `gcr.io/cos-cloud/cos-dkms:latest`.

When running this container, the following directories should be mounted:

*   The module source directory.
*   The output directory for installed modules.
*   `/etc/lsb-release` (to allow `cos-dkms` to automatically detect the host's
    COS build ID and board).
*   Optionally, a cache directory for kernel headers and toolchains to speed up
    subsequent runs.

### Basic Docker Template

```bash
docker run --rm -it \
  -v "/path/to/your/sources:/usr/src/my-package-1.0" \
  -v "/path/to/install/output:/var/lib/dkms-install" \
  -v /etc/lsb-release:/etc/lsb-release \
  gcr.io/cos-cloud/cos-dkms:latest \
  [action] [options] my-package/1.0
```

If you are using cos-dkms to load modules, then you will need to run the
container with `--privileged`. For modules which rely on networking, passing
`--net=host` may also be required.

--------------------------------------------------------------------------------

## Actions {#actions}

`cos-dkms` supports a set of actions to manage the lifecycle of a kernel module.
In most cases, you will only need the high-level `install` and `remove`
commands, which automatically orchestrate the lower-level actions.

```
       +-----------+   calls    +---------+   calls    +---------+
       |  Install  |  ------->  |  Build  |  ------->  |   Add   |
       +-----------+            +---------+            +---------+
           | undone by             | undone by           | undone by
           v                       v                     v
       +-----------+            +---------+            +---------+
       | Uninstall |  <-------  | Unbuild |  <-------  |  Remove |
       +-----------+    calls   +---------+    calls   +---------+
```

### High-Level Actions

#### `install`

Installs all modules of a package into the install tree.

*   **Behavior**: If the package is not yet added, it runs `add`. If it is not
    yet built, it runs `build`. It then copies the compiled `.ko` files to the
    install tree.
*   **Version Safety**: By default, it checks if the version being installed is
    older than an already installed version. If so, it skips installation. Use
    `--force-version-override` to bypass.
*   **GCS Integration**: If `--gcs-bucket` is provided, then it will attempt to
    download sources and modules from the given bucket. If `--upload` is passed,
    then it will attempt to upload the sources and modules to the bucket after
    they are added and built, respectively.
*   **Key Flags**:
*   *   `-i`, `--installtree`: The directory where the modules will be installed.
        Defaults to /lib/modules/${kernel-version}.
    *   `-b`, `--kernelmodulestree`: The directory containing the in-tree kernel
        modules. Only used with --insert-on-install. Defaults to ${installtree}.
        The modules in this tree can be loaded as dependencies when inserting
        modules installed by cos-dkms. If a module is present in both the kernel
        modules tree and the install tree, then only the one in the install tree
        will be loaded.
        The reason to separate this value from the install tree is that the install
        tree must be writable, while the kernel's module directory may not be
        writable. This is the case in COS instances, where the /lib/modules
        directory is read-only. See the examples below for correct usage of
        --installtree and --kernelmodulestree in COS.
    *   `-I`, `--insert-on-install`: Inserts the modules directly into the
        running kernel (using `insmod`-like behavior) with specified
        `--module-arg` parameters. This takes into account the order of the
        modules listed in the dkms.conf file in order to install earlier modules
        first, when possible. This allows the installation of module lists with
        implicit dependencies, such as when one module does not explicitly
        import symbols from another but does require that the other module be
        loaded first.
    *   `--module-arg`: Argument to pass to a specific kernel module when loading
        it with `--insert-on-install`. Multiple args for multiple modules can be
        specified at the same time by passing a comma separated list or passing
        --module-arg multiple times. The format is
        --module-arg=module1.param1=value1,module2.param1=value1, etc.
    *   `-M`, `--modprobe-on-install`: Automatically runs `depmod` and
        `modprobe` to load the modules into the running kernel after
        installation. This is useful for more complex installations that require
        modprobe configurations in /etc/modprobe.d.
        Note that this doesn't use --kernelmodulestree because depmod and modprobe
        require all modules to be under a subdirectory of
        /lib/modules/<kernel version>, so specifying a separate
        --kernelmodulestree and --installtree would not work as expected.
    *   `--no-depmod`: Skip running `depmod` before `modprobe` (only used with
        `-M`).

#### `remove`

Completely removes a package.

*   **Behavior**: Automatically runs `uninstall` (removes from install tree) and
    `unbuild` (cleans build artifacts), then removes the package sources from
    the DKMS source tree.
*   **GCS Integration**: If `--gcs-bucket` is provided and `--upload` is passed,
    it also removes the package from the bucket.

--------------------------------------------------------------------------------

### Low-Level Actions

#### `add`

Adds a package to the local DKMS source tree.

*   **Behavior**: Looks for source files in the source tree (default
    `/usr/src`). If not found, it tries to copy them from a local directory
    named `<package>-<version>`.
*   **GCS Integration**: If `--gcs-bucket` is provided and no local sources are
    found, it checks the bucket for sources. If `--upload` is specified, it
    uploads the sources to the bucket.

#### `build`

Compiles the modules in the package.

*   **Behavior**: Copies sources to a build directory, applies patches (if
    specified in `dkms.conf`), and runs the `MAKE` command.
*   **GCS Integration**: If `--gcs-bucket` is provided, it attempts to download
    precompiled modules first. It only builds locally if they are missing. If
    `--upload` is specified, it uploads the compiled modules to the bucket after
    they are built.
*   **Key Flags**:
    *   `--install-build-dependencies`: Automatically downloads the correct
        kernel headers and compiler toolchain for the target COS version.
    *   `--make-variables=cos-default`: Appends default toolchain variables
        (like `CC`, `LD`, etc.) matching the COS kernel build environment.
    *   `-j`, `--jobs`: Number of parallel make jobs (default 1).
    *   `--upload`: Uploads the compiled modules to the GCS bucket after a
        successful build.

#### `status`

Displays the current status of a package.

*   **Output Statuses**:
    *   `Added`: Sources are in the DKMS source tree.
    *   `Built`: Sources are added and modules are compiled.
    *   `Installed`: Modules are compiled and copied to the install tree.
    *   `Broken`: Package is in an inconsistent state (e.g., missing sources,
        invalid config).
*   **GCS Integration**: Reflects the status in the GCS bucket if `--gcs-bucket`
    is provided.

#### `unbuild`

Cleans build artifacts.

*   **Behavior**: Removes compiled modules from the build directory.
    Automatically runs `uninstall` first if the package is installed.
*   **GCS Integration**: If `--gcs-bucket` is provided and `--upload` is passed,
    it also removes the compiled modules from the bucket.

#### `uninstall`

Removes compiled modules from the install tree.

--------------------------------------------------------------------------------

### Global Options

These flags can be passed to most commands:

*   `-p`, `--package` / `-m`, `--module`: Name of the package.
*   `--package-version` / `--module-version`: Version of the package.
*   `-k`, `--kernel-version`: Target kernel version (defaults to running kernel,
    or detected from headers if set to `cos-default`).
*   `-a`, `--arch`: Target architecture (default: running host arch).
*   `--build-id`: Target COS build ID (e.g., `18244.151.84`).
*   `--board`: Target COS board (e.g., `lakitu`).
*   `--gcs-bucket`: GCS bucket path (e.g., `gs://my-bucket/dkms`) to use as a
    cache.
*   `-f`, `--force`: Force the action even if `cos-dkms` thinks it is already
    done.
*   `-s`, `--sourcetree`: The directory where the module source tree is located.
    Defaults to /usr/src.

Additionally, cos-dkms uses [glog](https://pkg.go.dev/github.com/golang/glog) for
logging, so arguments to configure glog logging (logtostderr, stderrthreshold,
etc.) are accepted.

--------------------------------------------------------------------------------

## Configuration (dkms.conf) {#configuration}

The `dkms.conf` file is a bash script that defines how `cos-dkms` should handle
the package. It supports Dell DKMS-compatible directives.

### Environment Variables Available in `dkms.conf`

When `dkms.conf` is sourced, the following variables are pre-populated and can
be referenced:

*   `$module`: Package name.
*   `$module_version`: Package version.
*   `$kernelver`: Target kernel version.
*   `$arch`: Target architecture.
*   `$build_id` / `$board`: COS-specific build metadata.
*   `$dkms_tree`: Path to the DKMS tree.
*   `$source_tree`: Path to the source tree.
*   `$kernel_source_dir`: Path to kernel headers/sources.
*   `$build_dir`: Path to the temporary build directory.

### Directives

Each of these is a top-level variable in the dkms.conf file which controls the
behavior of cos-dkms in some way. Each directive has a default value which will
be overwritten by any user-specified values in the dkms.conf. The notation
`VARIABLE[#]` indicates that `VARIABLE` is a bash array which can contain
multiple values. If only one variable in the array is used, then a simple
assignment works, e.g., `BUILT_MODULE_NAME=my-module` has the same effect as
`BUILT_MODULE_NAME=(my-module)`.

#### Package Metadata

*   `PACKAGE_NAME`: Name of the package (defaults to `--package`).
*   `PACKAGE_VERSION`: Version of the package (defaults to `--package-version`).

#### Build Commands

*   `MAKE[#]`: The command used to compile the modules. The first entry in this
    array defaults to `make -C ${kernel_source_dir} M=${build_dir}`. You can
    specify multiple `MAKE` commands indexed by number, paired with
    `MAKE_MATCH[#]` to select a command based on the kernel version.
*   `MAKE_MATCH[#]`: A regex matched against `$kernelver`. The first `MAKE[#]`
    whose `MAKE_MATCH[#]` matches the target kernel version will be used.

#### Module Definitions

For packages containing multiple modules, use 0-indexed arrays:

*   `BUILT_MODULE_NAME[#]`: The filename of the compiled module (without `.ko`).
    The first entry in this array defaults to the package name.
*   `BUILT_MODULE_LOCATION[#]`: Directory where the module is built, relative to
    `$build_dir`. Default is empty (root of build dir).
*   `DEST_MODULE_NAME[#]`: Destination filename for the module. Defaults to the
    corresponding `BUILT_MODULE_NAME[#]`.
*   `DEST_MODULE_LOCATION[#]`: Destination directory in the install tree,
    relative to `$installtree`. **Must start with `/kernel`**. Defaults to
    `/kernel/updates`.
*   `STRIP[#]`: Whether to strip debug symbols for the module (`yes` or `no`).
    Defaults to `yes`.

#### Dependencies and Constraints

*   `BUILD_DEPENDS[#]`: List of other DKMS packages that must be built before
    this one. Currently informational.
*   `BUILD_EXCLUSIVE_KERNEL`: Regex. If the target `$kernelver` does not match,
    the build will be skipped.
*   `BUILD_EXCLUSIVE_ARCH`: Regex. If the target `$arch` does not match, the
    build will be skipped.
*   `AUTOINSTALL`: If set to `yes`, indicates the module should be rebuilt on
    kernel upgrades. Currently informational.

#### Patches

*   `PATCH[#]`: Filename of a patch (in `p1` format) located in the `patches/`
    subdirectory of the package, to be applied before building.
*   `PATCH_MATCH[#]`: Regex matched against `$kernelver`. The patch will only be
    applied if it matches.

#### Hooks

Scripts (located in the package source) to run at various lifecycle stages:

*   `POST_ADD`
*   `PRE_BUILD` / `POST_BUILD`
*   `PRE_INSTALL` / `POST_INSTALL`
*   `POST_REMOVE`

--------------------------------------------------------------------------------

### Configuration Examples {#configuration-examples}

#### Simple Case (Single Module)

If your package contains a single module whose name matches the package name,
and it builds with a standard `make` command, **you do not need a `dkms.conf`
file at all**. The default values are sufficient.

#### Typical Case (Multiple Modules & Hooks)

A typical `dkms.conf` for a package with multiple modules and custom build
steps:

```bash
PACKAGE_NAME="my-driver"
PACKAGE_VERSION="1.0.0"

# First module
BUILT_MODULE_NAME[0]="my-core-driver"
DEST_MODULE_LOCATION[0]="/kernel/drivers/misc"

# Second module (depends on core)
BUILT_MODULE_NAME[1]="my-extension-driver"
BUILT_MODULE_LOCATION[1]="extensions/"
DEST_MODULE_LOCATION[1]="/kernel/drivers/misc"

# Custom build command
MAKE="make -C ${kernel_source_dir} M=${build_dir} EXTRA_CFLAGS=-DDEBUG"

# Hooks
PRE_BUILD="setup_env.sh"
POST_INSTALL="echo 'Driver installed successfully!'"
```

#### Real-world Case (Lustre Client Drivers)

This example shows how to configure a complex package that requires source
modification and autotools configuration, and has many modules with strict
load-order requirements. Note that it takes advantage of the implicit pairing of
the module name/location lists and the module destination defaults to keep the
file simple.

```bash
# Install build-time dependencies inside the builder container
POST_ADD="apt update && DEBIAN_FRONTEND=noninteractive apt install -y python3-dev libtool git flex bison build-essential libmpich-dev clang python3-setuptools llvm binutils lld libkeyutils-dev libmount-dev pkg-config libnl-3-dev libnl-genl-3-dev libyaml-dev libjson-c-dev"

# Configure the build directory
# Remove gcc-specific flags since COS uses clang.
clean_sources="sed -i -e 's/-Wno-stringop-overflow//g' -e 's/-Wno-stringop-truncation//g' -e 's/-Wno-format-truncation//g' config/lustre-toolchain.m4"
configure="LLVM=1 ./configure --disable-server --enable-client --disable-strict-errors --with-linux=/lib/modules/${kernelver}/build/"
PRE_BUILD="cd ${build_dir} && ${clean_sources} && bash autogen.sh && ${configure}"

# List of modules to build and their locations
BUILT_MODULE_NAME=(
    ksocklnd ko2iblnd fid fld libcfs lmv lnet lov lustre
    mdc mgc obdclass obdecho osc ptlrpc lnet_selftest
)
BUILT_MODULE_LOCATION=(
    lnet/klnds/socklnd lnet/klnds/o2iblnd lustre/fid
    lustre/fld libcfs/libcfs lustre/lmv lnet/lnet
    lustre/lov lustre/llite lustre/mdc lustre/mgc
    lustre/obdclass lustre/obdecho lustre/osc
    lustre/ptlrpc lnet/selftest
)
```

--------------------------------------------------------------------------------

## Module Signing {#module-signing}

COS enforces kernel module signature verification. Modules that are not signed
by a trusted key will fail to load (`Required key not available`).

`cos-dkms` can automatically sign compiled modules during the `build` or
`install` phase.

By default, the `sha256` algorithm is used for signing. You can customize this
by passing `--hash-algorithm` (e.g., `sha384`, `sha512`).

### Signing Options

-   **Local Private Key (Default)**: `cos-dkms` will look for a private key and
    certificate.
    *   Default paths: `/var/lib/dkms/mok.key` (private key) and
        `/var/lib/dkms/mok.cert` (certificate).
    *   You can override these using the environment variables
        `MODULES_SIGN_KEY` and `MODULES_SIGN_CERT`.
-   **Cloud KMS**: You can also use a key stored in Google Cloud KMS.
    *   Pass the KMS key resource name via the `--cloud-kms-key` flag:
        `--cloud-kms-key=projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key/cryptoKeyVersions/1`
    *   You must also provide the public certificate path via
        `MODULES_SIGN_CERT` or have it at `/var/lib/dkms/mok.cert`.

--------------------------------------------------------------------------------

## Example Workflows {#workflows}

### Workflow 1: Installing Precompiled Modules {#workflow-precompiled-modules}

This workflow is intended to be used in production VMs to install modules. It
downloads precompiled modules from a GCS bucket cache and installs them. **No
compiler toolchain, kernel headers, or build dependencies are needed on the
VM.**

```
+-------------------+      Check GCS Cache      +-------------------+
|    Target VM      |  ---------------------->  |    GCS Bucket     |
| (No Compiler req) |  <----------------------  | (Precompiled .ko) |
+-------------------+      Download & Load      +-------------------+
```

#### Steps:

1.  On the COS VM, run the `cos-dkms` container.
2.  Provide the `--gcs-bucket` where the precompiled modules are stored.
3.  Use the `--insert-on-install` flag to automatically load them.

#### Example Command:

```bash
docker run --rm -it \
  --privileged --net=host \
  -v /etc/lsb-release:/etc/lsb-release \
  -v /lib/modules:/host_modules \
  gcr.io/cos-cloud/cos-dkms:latest \
  install my-driver/1.0.0 \
  --gcs-bucket=gs://my-company-cos-drivers \
  --kernelmodulestree=/host_modules \
  --insert-on-install
```

Using the high-level
[cos-dkms runner script](https://cos.googlesource.com/cos/tools/+/refs/heads/master/src/cmd/cos_dkms_runner),
this command can be simplified to:

```bash
sudo cos-dkms install my-driver/1.0.0 \
  --gcs-bucket=gs://my-company-cos-drivers \
  --insert-on-install
```

*What happens under the hood:*

1.  `cos-dkms` detects the running kernel version, board, and build ID.
2.  It checks `gs://my-company-cos-drivers/` for the precompiled `my-driver`
    modules matching those exact parameters.
3.  It downloads the precompiled `.ko` files.
4.  It copies them to `/lib/modules/$(uname -r)/kernel/updates` in the container.
5.  It dynamically computes the module insertion order from the dkms.conf file
    and the modules' dependencies (including modules from the read-only
    kernelmodulestree mapped from the host), then uses `insmod` to load each
    module in the correct order.

--------------------------------------------------------------------------------

### Workflow 2: Installing Modules Compiled by Google for COS {#workflow-google-compiled-modules}

This is a special case of workflow 1 to install modules which are compiled as
part of the official COS image build and signed by Google. For these modules, a
value of `--gcs-bucket=cos-default` can be passed, and cos-dkms will infer the
correct COS artifacts bucket to download modules from.

#### Example Command (installing Lustre drivers):

```bash
docker run --rm -it \
  --privileged --net=host \
  -v /etc/lsb-release:/etc/lsb-release \
  -v /lib/modules:/host_modules \
  gcr.io/cos-cloud/cos-dkms:latest \
  install lustre-client-drivers/2.14.0_p224 \
  --gcs-bucket=cos-default \
  --kernelmodulestree=/host_modules \
  --insert-on-install
```

Using the high-level
[cos-dkms runner script](https://cos.googlesource.com/cos/tools/+/refs/heads/master/src/cmd/cos_dkms_runner),
this command can be simplified to:

```bash
sudo cos-dkms install lustre-client-drivers/2.14.0_p224 \
  --gcs-bucket=cos-default \
  --insert-on-install
```

*What happens under the hood:*

1.  `cos-dkms` detects the running kernel version, board, and build ID.
2.  It checks the `gs://cos-tools/` bucket for the precompiled modules (in this
    case, Lustre) matching those exact parameters.
3.  It downloads the precompiled `.ko` files.
4.  It copies them to `/lib/modules/$(uname -r)/kernel/updates` in the container.
5.  It dynamically computes the module insertion order from the dkms.conf file
    and the modules' dependencies (including modules from the read-only
    kernelmodulestree mapped from the host), then uses `insmod` to load each
    module in the correct order.

--------------------------------------------------------------------------------

### Workflow 3: Installing Modules with modprobe {#workflow-modprobe}

Some modules require more complex configuration to install properly. In these cases,
using modprobe is more robust and configurable than the --insert-on-install option
that cos-dkms provides. However, it requires a little more setup on COS than using
--insert-on-install because the /lib/modules directory in COS is read-only.

At a high level, the workflow for using --modprobe-on-install on COS is to add
any necessary modprobe configurations in the host, copy the host's kernel modules
to a writable directory, and mount that directory into the cos-dkms container.

#### Example Command (installing Lustre drivers with modprobe):

```bash
# Write modprobe.d confs as normal; this one deals with the fact that
# lustre can't automatically load ksocklnd from the container context.
echo 'softdep lustre pre: ksocklnd post:' > /tmp/lustre.conf

# Copy the kernel's modules to a rw directory.
cp -ra /lib/modules/ /tmp/host_modules

# Run cos-dkms, pretending that the container's /lib/modules directory is the real
# one from this host. Doing it this way means that depmod/modprobe work as normal.
docker run --rm -it --privileged --net=host \
  -v /etc/lsb-release:/etc/lsb-release \
  -v /tmp/host_modules:/lib/modules \
  -v /tmp/lustre.conf:/etc/modprobe.d/lustre.conf \
  gcr.io/cos-cloud/cos-dkms:latest \
  install lustre-client-drivers --latest --modprobe-on-install \
  --gcs-bucket=cos-default \
  --logtostderr -v2
```

*What happens under the hood:*

1.  `cos-dkms` detects the running kernel version, board, and build ID.
2.  It checks `gs://my-company-cos-drivers/` for the precompiled `my-driver`
    modules matching those exact parameters.
3.  It downloads the precompiled `.ko` files.
4.  It copies them to `/lib/modules/$(uname -r)/kernel/updates`.
5.  It runs depmod to build the module dependency graph, taking into account
    the newly compiled modules and the host kernel's modules since they are
    both in the "expected" /lib/modules/<kernel version> directory.
6.  It runs modprobe to install the Lustre kernel drivers, taking into account
    the modprobe configurations in the containers /etc/modprobe.conf, in this
    case, the lustre.conf that we wrote in /tmp on the host.

--------------------------------------------------------------------------------

### Workflow 4: Building and Distributing Modules {#workflow-building-distributing}

This workflow can be used by developers or in CI/CD pipelines to compile module
sources for a specific COS version, sign them, and upload them to a GCS bucket
cache for use in Workflow 1.

```
+------------------+     Download Headers      +--------------------+
| Builder Machine  |  <----------------------  |  COS Repositories  |
| (Runs cos-dkms)  |   & Compiler Toolchain    +--------------------+
+------------------+
         |
         | Build & Sign Modules
         v
+------------------+      Upload Artifacts     +--------------------+
|  Compiled .ko    |  ---------------------->  |     GCS Bucket     |
+------------------+                           |      (Cache)       |
                                               +--------------------+
```

#### Steps:

1.  Prepare your module source directory containing the code and a `dkms.conf`.
2.  Run `cos-dkms build` with:
    *   Target `lsb-release`: An LSB release from a target COS image. The
        variables in this file will be used to infer the correct toolchain,
        kernel version, and arch to allow cross-compiling the modules for a
        particular COS version.
    *   `--install-build-dependencies`: To fetch the matching COS kernel headers
        and toolchain.
    *   `--make-variables=cos-default`: To use the official COS compiler flags.
    *   `--gcs-bucket` and `--upload`: To save the results to the cache.
    *   Signature keys (local or KMS) to sign the modules.

#### Example Command:

```bash
docker run --rm -it \
  -v /$PWD/targets/lsb-release:/etc/lsb-release \
  -v "$PWD/mok.key:/var/lib/dkms/mok.key" \
  -v "$PWD/mok.cert:/var/lib/dkms/mok.cert" \
  gcr.io/cos-cloud/cos-dkms:latest \
  build my-module/v1.2.3 \
  --install-build-dependencies \
  --make-variables=cos-default \
  --gcs-bucket=gs://my-company/my-cos-drivers \
  --upload \
  -v2 --logtostderr
```

After running this command, a user can install the modules on a COS VM with a
symmetric command:

```bash
docker run --rm -it \
  --privileged --net=host \
  -v /etc/lsb-release:/etc/lsb-release \
  -v /lib/modules:/host_modules
  gcr.io/cos-cloud/cos-dkms:latest \
  install my-module/v1.2.3 \
  --gcs-bucket=gs://my-company/my-cos-drivers \
  --kernelmodulestree=/host_modules \
  -v2 --logtostderr
```

*What happens under the hood:*

1.  `cos-dkms` downloads the exact `clang` toolchain and kernel headers for the
    COS version that matches the variables specified in the target LSB release.
2.  It copies the module sources from the provided GCS bucket to a build
    directory and applies any patches.
3.  It builds the module using the `MAKE` command from the dkms.conf.
4.  It signs all the compiled `.ko` files using the provided `mok.key` and
    `mok.cert`.
5.  It uploads both the source package and the compiled, signed `.ko` files to
    `gs://my-company/my-cos-drivers/` under a path specific to the kernel
    version, board, and build ID.