cos-dkms is a tool designed for Google Container-Optimized OS (COS) to dynamically build, install, and manage kernel modules. It is intended to be a drop-in replacement for the upstream Dynamic Kernel Module Support (DKMS) tool, but tailored for the unique environment of COS and GKE.
Key features include:
Code-level documentation is available here.
Documentation for the high-level cos-dkms runner script is available here.
Because COS is a minimal operating system without a pre-installed compiler toolchain, cos-dkms is intended to be run inside a Docker container. Google provides a pre-built container image: gcr.io/cos-cloud/cos-dkms:latest.
When running this container, the following directories should be mounted:
/etc/lsb-release (to allow cos-dkms to automatically detect the host's COS build ID and board).docker run --rm -it \ -v "/path/to/your/sources:/usr/src/my-package-1.0" \ -v "/path/to/install/output:/var/lib/dkms-install" \ -v /etc/lsb-release:/etc/lsb-release \ gcr.io/cos-cloud/cos-dkms:latest \ [action] [options] my-package/1.0
If you are using cos-dkms to load modules, then you will need to run the container with --privileged. For modules which rely on networking, passing --net=host may also be required.
cos-dkms supports a set of actions to manage the lifecycle of a kernel module. In most cases, you will only need the high-level install and remove commands, which automatically orchestrate the lower-level actions.
+-----------+ calls +---------+ calls +---------+
| Install | -------> | Build | -------> | Add |
+-----------+ +---------+ +---------+
| undone by | undone by | undone by
v v v
+-----------+ +---------+ +---------+
| Uninstall | <------- | Unbuild | <------- | Remove |
+-----------+ calls +---------+ calls +---------+
installInstalls all modules of a package into the install tree.
add. If it is not yet built, it runs build. It then copies the compiled .ko files to the install tree.--force-version-override to bypass.--gcs-bucket is provided, then it will attempt to download sources and modules from the given bucket. If --upload is passed, then it will attempt to upload the sources and modules to the bucket after they are added and built, respectively.-i, --installtree: The directory where the modules will be installed. Defaults to /lib/modules/${kernel-version}.-b, --kernelmodulestree: The directory containing the in-tree kernel modules. Only used with --insert-on-install. Defaults to ${installtree}. The modules in this tree can be loaded as dependencies when inserting modules installed by cos-dkms. If a module is present in both the kernel modules tree and the install tree, then only the one in the install tree will be loaded. The reason to separate this value from the install tree is that the install tree must be writable, while the kernel's module directory may not be writable. This is the case in COS instances, where the /lib/modules directory is read-only. See the examples below for correct usage of --installtree and --kernelmodulestree in COS.-I, --insert-on-install: Inserts the modules directly into the running kernel (using insmod-like behavior) with specified --module-arg parameters. This takes into account the order of the modules listed in the dkms.conf file in order to install earlier modules first, when possible. This allows the installation of module lists with implicit dependencies, such as when one module does not explicitly import symbols from another but does require that the other module be loaded first.--module-arg: Argument to pass to a specific kernel module when loading it with --insert-on-install. Multiple args for multiple modules can be specified at the same time by passing a comma separated list or passing --module-arg multiple times. The format is --module-arg=module1.param1=value1,module2.param1=value1, etc.-M, --modprobe-on-install: Automatically runs depmod and modprobe to load the modules into the running kernel after installation. This is useful for more complex installations that require modprobe configurations in /etc/modprobe.d. Note that this doesn't use --kernelmodulestree because depmod and modprobe require all modules to be under a subdirectory of /lib/modules/, so specifying a separate --kernelmodulestree and --installtree would not work as expected.--no-depmod: Skip running depmod before modprobe (only used with -M).removeCompletely removes a package.
uninstall (removes from install tree) and unbuild (cleans build artifacts), then removes the package sources from the DKMS source tree.--gcs-bucket is provided and --upload is passed, it also removes the package from the bucket.addAdds a package to the local DKMS source tree.
/usr/src). If not found, it tries to copy them from a local directory named <package>-<version>.--gcs-bucket is provided and no local sources are found, it checks the bucket for sources. If --upload is specified, it uploads the sources to the bucket.buildCompiles the modules in the package.
dkms.conf), and runs the MAKE command.--gcs-bucket is provided, it attempts to download precompiled modules first. It only builds locally if they are missing. If --upload is specified, it uploads the compiled modules to the bucket after they are built.--install-build-dependencies: Automatically downloads the correct kernel headers and compiler toolchain for the target COS version.--make-variables=cos-default: Appends default toolchain variables (like CC, LD, etc.) matching the COS kernel build environment.-j, --jobs: Number of parallel make jobs (default 1).--upload: Uploads the compiled modules to the GCS bucket after a successful build.statusDisplays the current status of a package.
Added: Sources are in the DKMS source tree.Built: Sources are added and modules are compiled.Installed: Modules are compiled and copied to the install tree.Broken: Package is in an inconsistent state (e.g., missing sources, invalid config).--gcs-bucket is provided.unbuildCleans build artifacts.
uninstall first if the package is installed.--gcs-bucket is provided and --upload is passed, it also removes the compiled modules from the bucket.uninstallRemoves compiled modules from the install tree.
These flags can be passed to most commands:
-p, --package / -m, --module: Name of the package.--package-version / --module-version: Version of the package.-k, --kernel-version: Target kernel version (defaults to running kernel, or detected from headers if set to cos-default).-a, --arch: Target architecture (default: running host arch).--build-id: Target COS build ID (e.g., 18244.151.84).--board: Target COS board (e.g., lakitu).--gcs-bucket: GCS bucket path (e.g., gs://my-bucket/dkms) to use as a cache.-f, --force: Force the action even if cos-dkms thinks it is already done.-s, --sourcetree: The directory where the module source tree is located. Defaults to /usr/src.Additionally, cos-dkms uses glog for logging, so arguments to configure glog logging (logtostderr, stderrthreshold, etc.) are accepted.
The dkms.conf file is a bash script that defines how cos-dkms should handle the package. It supports Dell DKMS-compatible directives.
dkms.confWhen dkms.conf is sourced, the following variables are pre-populated and can be referenced:
$module: Package name.$module_version: Package version.$kernelver: Target kernel version.$arch: Target architecture.$build_id / $board: COS-specific build metadata.$dkms_tree: Path to the DKMS tree.$source_tree: Path to the source tree.$kernel_source_dir: Path to kernel headers/sources.$build_dir: Path to the temporary build directory.Each of these is a top-level variable in the dkms.conf file which controls the behavior of cos-dkms in some way. Each directive has a default value which will be overwritten by any user-specified values in the dkms.conf. The notation VARIABLE[#] indicates that VARIABLE is a bash array which can contain multiple values. If only one variable in the array is used, then a simple assignment works, e.g., BUILT_MODULE_NAME=my-module has the same effect as BUILT_MODULE_NAME=(my-module).
PACKAGE_NAME: Name of the package (defaults to --package).PACKAGE_VERSION: Version of the package (defaults to --package-version).MAKE[#]: The command used to compile the modules. The first entry in this array defaults to make -C ${kernel_source_dir} M=${build_dir}. You can specify multiple MAKE commands indexed by number, paired with MAKE_MATCH[#] to select a command based on the kernel version.MAKE_MATCH[#]: A regex matched against $kernelver. The first MAKE[#] whose MAKE_MATCH[#] matches the target kernel version will be used.For packages containing multiple modules, use 0-indexed arrays:
BUILT_MODULE_NAME[#]: The filename of the compiled module (without .ko). The first entry in this array defaults to the package name.BUILT_MODULE_LOCATION[#]: Directory where the module is built, relative to $build_dir. Default is empty (root of build dir).DEST_MODULE_NAME[#]: Destination filename for the module. Defaults to the corresponding BUILT_MODULE_NAME[#].DEST_MODULE_LOCATION[#]: Destination directory in the install tree, relative to $installtree. Must start with /kernel. Defaults to /kernel/updates.STRIP[#]: Whether to strip debug symbols for the module (yes or no). Defaults to yes.BUILD_DEPENDS[#]: List of other DKMS packages that must be built before this one. Currently informational.BUILD_EXCLUSIVE_KERNEL: Regex. If the target $kernelver does not match, the build will be skipped.BUILD_EXCLUSIVE_ARCH: Regex. If the target $arch does not match, the build will be skipped.AUTOINSTALL: If set to yes, indicates the module should be rebuilt on kernel upgrades. Currently informational.PATCH[#]: Filename of a patch (in p1 format) located in the patches/ subdirectory of the package, to be applied before building.PATCH_MATCH[#]: Regex matched against $kernelver. The patch will only be applied if it matches.Scripts (located in the package source) to run at various lifecycle stages:
POST_ADDPRE_BUILD / POST_BUILDPRE_INSTALL / POST_INSTALLPOST_REMOVEIf your package contains a single module whose name matches the package name, and it builds with a standard make command, you do not need a dkms.conf file at all. The default values are sufficient.
A typical dkms.conf for a package with multiple modules and custom build steps:
PACKAGE_NAME="my-driver" PACKAGE_VERSION="1.0.0" # First module BUILT_MODULE_NAME[0]="my-core-driver" DEST_MODULE_LOCATION[0]="/kernel/drivers/misc" # Second module (depends on core) BUILT_MODULE_NAME[1]="my-extension-driver" BUILT_MODULE_LOCATION[1]="extensions/" DEST_MODULE_LOCATION[1]="/kernel/drivers/misc" # Custom build command MAKE="make -C ${kernel_source_dir} M=${build_dir} EXTRA_CFLAGS=-DDEBUG" # Hooks PRE_BUILD="setup_env.sh" POST_INSTALL="echo 'Driver installed successfully!'"
This example shows how to configure a complex package that requires source modification and autotools configuration, and has many modules with strict load-order requirements. Note that it takes advantage of the implicit pairing of the module name/location lists and the module destination defaults to keep the file simple.
# Install build-time dependencies inside the builder container POST_ADD="apt update && DEBIAN_FRONTEND=noninteractive apt install -y python3-dev libtool git flex bison build-essential libmpich-dev clang python3-setuptools llvm binutils lld libkeyutils-dev libmount-dev pkg-config libnl-3-dev libnl-genl-3-dev libyaml-dev libjson-c-dev" # Configure the build directory # Remove gcc-specific flags since COS uses clang. clean_sources="sed -i -e 's/-Wno-stringop-overflow//g' -e 's/-Wno-stringop-truncation//g' -e 's/-Wno-format-truncation//g' config/lustre-toolchain.m4" configure="LLVM=1 ./configure --disable-server --enable-client --disable-strict-errors --with-linux=/lib/modules/${kernelver}/build/" PRE_BUILD="cd ${build_dir} && ${clean_sources} && bash autogen.sh && ${configure}" # List of modules to build and their locations BUILT_MODULE_NAME=( ksocklnd ko2iblnd fid fld libcfs lmv lnet lov lustre mdc mgc obdclass obdecho osc ptlrpc lnet_selftest ) BUILT_MODULE_LOCATION=( lnet/klnds/socklnd lnet/klnds/o2iblnd lustre/fid lustre/fld libcfs/libcfs lustre/lmv lnet/lnet lustre/lov lustre/llite lustre/mdc lustre/mgc lustre/obdclass lustre/obdecho lustre/osc lustre/ptlrpc lnet/selftest )
COS enforces kernel module signature verification. Modules that are not signed by a trusted key will fail to load (Required key not available).
cos-dkms can automatically sign compiled modules during the build or install phase.
By default, the sha256 algorithm is used for signing. You can customize this by passing --hash-algorithm (e.g., sha384, sha512).
cos-dkms will look for a private key and certificate./var/lib/dkms/mok.key (private key) and /var/lib/dkms/mok.cert (certificate).MODULES_SIGN_KEY and MODULES_SIGN_CERT.--cloud-kms-key flag: --cloud-kms-key=projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key/cryptoKeyVersions/1MODULES_SIGN_CERT or have it at /var/lib/dkms/mok.cert.This workflow is intended to be used in production VMs to install modules. It downloads precompiled modules from a GCS bucket cache and installs them. No compiler toolchain, kernel headers, or build dependencies are needed on the VM.
+-------------------+ Check GCS Cache +-------------------+ | Target VM | ----------------------> | GCS Bucket | | (No Compiler req) | <---------------------- | (Precompiled .ko) | +-------------------+ Download & Load +-------------------+
cos-dkms container.--gcs-bucket where the precompiled modules are stored.--insert-on-install flag to automatically load them.docker run --rm -it \ --privileged --net=host \ -v /etc/lsb-release:/etc/lsb-release \ -v /lib/modules:/host_modules \ gcr.io/cos-cloud/cos-dkms:latest \ install my-driver/1.0.0 \ --gcs-bucket=gs://my-company-cos-drivers \ --kernelmodulestree=/host_modules \ --insert-on-install
Using the high-level cos-dkms runner script, this command can be simplified to:
sudo cos-dkms install my-driver/1.0.0 \ --gcs-bucket=gs://my-company-cos-drivers \ --insert-on-install
What happens under the hood:
cos-dkms detects the running kernel version, board, and build ID.gs://my-company-cos-drivers/ for the precompiled my-driver modules matching those exact parameters..ko files./lib/modules/$(uname -r)/kernel/updates in the container.insmod to load each module in the correct order.This is a special case of workflow 1 to install modules which are compiled as part of the official COS image build and signed by Google. For these modules, a value of --gcs-bucket=cos-default can be passed, and cos-dkms will infer the correct COS artifacts bucket to download modules from.
docker run --rm -it \ --privileged --net=host \ -v /etc/lsb-release:/etc/lsb-release \ -v /lib/modules:/host_modules \ gcr.io/cos-cloud/cos-dkms:latest \ install lustre-client-drivers/2.14.0_p224 \ --gcs-bucket=cos-default \ --kernelmodulestree=/host_modules \ --insert-on-install
Using the high-level cos-dkms runner script, this command can be simplified to:
sudo cos-dkms install lustre-client-drivers/2.14.0_p224 \ --gcs-bucket=cos-default \ --insert-on-install
What happens under the hood:
cos-dkms detects the running kernel version, board, and build ID.gs://cos-tools/ bucket for the precompiled modules (in this case, Lustre) matching those exact parameters..ko files./lib/modules/$(uname -r)/kernel/updates in the container.insmod to load each module in the correct order.Some modules require more complex configuration to install properly. In these cases, using modprobe is more robust and configurable than the --insert-on-install option that cos-dkms provides. However, it requires a little more setup on COS than using --insert-on-install because the /lib/modules directory in COS is read-only.
At a high level, the workflow for using --modprobe-on-install on COS is to add any necessary modprobe configurations in the host, copy the host's kernel modules to a writable directory, and mount that directory into the cos-dkms container.
# Write modprobe.d confs as normal; this one deals with the fact that # lustre can't automatically load ksocklnd from the container context. echo 'softdep lustre pre: ksocklnd post:' > /tmp/lustre.conf # Copy the kernel's modules to a rw directory. cp -ra /lib/modules/ /tmp/host_modules # Run cos-dkms, pretending that the container's /lib/modules directory is the real # one from this host. Doing it this way means that depmod/modprobe work as normal. docker run --rm -it --privileged --net=host \ -v /etc/lsb-release:/etc/lsb-release \ -v /tmp/host_modules:/lib/modules \ -v /tmp/lustre.conf:/etc/modprobe.d/lustre.conf \ gcr.io/cos-cloud/cos-dkms:latest \ install lustre-client-drivers --latest --modprobe-on-install \ --gcs-bucket=cos-default \ --logtostderr -v2
What happens under the hood:
cos-dkms detects the running kernel version, board, and build ID.gs://my-company-cos-drivers/ for the precompiled my-driver modules matching those exact parameters..ko files./lib/modules/$(uname -r)/kernel/updates.This workflow can be used by developers or in CI/CD pipelines to compile module sources for a specific COS version, sign them, and upload them to a GCS bucket cache for use in Workflow 1.
+------------------+ Download Headers +--------------------+
| Builder Machine | <---------------------- | COS Repositories |
| (Runs cos-dkms) | & Compiler Toolchain +--------------------+
+------------------+
|
| Build & Sign Modules
v
+------------------+ Upload Artifacts +--------------------+
| Compiled .ko | ----------------------> | GCS Bucket |
+------------------+ | (Cache) |
+--------------------+
dkms.conf.cos-dkms build with:lsb-release: An LSB release from a target COS image. The variables in this file will be used to infer the correct toolchain, kernel version, and arch to allow cross-compiling the modules for a particular COS version.--install-build-dependencies: To fetch the matching COS kernel headers and toolchain.--make-variables=cos-default: To use the official COS compiler flags.--gcs-bucket and --upload: To save the results to the cache.docker run --rm -it \ -v /$PWD/targets/lsb-release:/etc/lsb-release \ -v "$PWD/mok.key:/var/lib/dkms/mok.key" \ -v "$PWD/mok.cert:/var/lib/dkms/mok.cert" \ gcr.io/cos-cloud/cos-dkms:latest \ build my-module/v1.2.3 \ --install-build-dependencies \ --make-variables=cos-default \ --gcs-bucket=gs://my-company/my-cos-drivers \ --upload \ -v2 --logtostderr
After running this command, a user can install the modules on a COS VM with a symmetric command:
docker run --rm -it \ --privileged --net=host \ -v /etc/lsb-release:/etc/lsb-release \ -v /lib/modules:/host_modules gcr.io/cos-cloud/cos-dkms:latest \ install my-module/v1.2.3 \ --gcs-bucket=gs://my-company/my-cos-drivers \ --kernelmodulestree=/host_modules \ -v2 --logtostderr
What happens under the hood:
cos-dkms downloads the exact clang toolchain and kernel headers for the COS version that matches the variables specified in the target LSB release.MAKE command from the dkms.conf..ko files using the provided mok.key and mok.cert..ko files to gs://my-company/my-cos-drivers/ under a path specific to the kernel version, board, and build ID.