Add license text scanning to install hook for SBOM

gen-package-licenses.sh pre-processes package
licenses so we can depend on license.yaml generated
by the script for license names.

Any license not listed by SPDX spec needs to have
license text scanned. Though license.yaml contains
some scanned license text, it doesn't always have
what we need so we have to scan the source code
ourselves.

This CL adds functions to SBOM generation
for correcting license names for SPDX spec,
and scanning license text. It also updates the code
for fetching go dependencies and make use of package
source code fetched by gen-package-license.sh.

The license file of package lsof was added to the source
code on Jan 13 2023, so we need to treat it as a special
case until we update it to a newer version.

netcat is very special. See its license text in
license_data.py.

BUG=b/254334533
TEST=presubmit
RELEASE_NOTE=None

Change-Id: I6f860766b43f2738bc933b82416086bc3ab9277f
Reviewed-on: https://cos-review.googlesource.com/c/third_party/platform/crosutils/+/41682
Reviewed-by: Robert Kolchmeyer <rkolchmeyer@google.com>
Tested-by: Cusky Presubmit Bot <presubmit@cos-infra-prod.iam.gserviceaccount.com>
Reviewed-by: Vaibhav Rustagi <vaibhavrustagi@google.com>
5 files changed