| commit | bfae4bfc108dd981bf3887b45cee10395253fe1d | [log] [tgz] |
|---|---|---|
| author | Michael Mortensen <mmortensen@google.com> | Fri Oct 16 16:25:09 2020 -0600 |
| committer | Commit Bot <commit-bot@chromium.org> | Tue Oct 20 01:22:35 2020 +0000 |
| tree | e84870e0b5bf017fc2a7e76746af98fb512ea824 | |
| parent | fcec7c21e226eaf20bd283926a2289e74131b5d4 [diff] |
Add wheels for google-cloud-logging dependencies. BUG=chromium:1138969 TEST=`create_venv3 chromite/venv/requirements.txt` These changes down-version google_api_core to 1.19.0 and protobuf to 3.11.3, to match current ebuild constraints. I downloaded google_auth 1.14 to avoid downloading the very latest because google_api_core 1.19.0 only needs google_auth 1.14.0. Note that we already have protobuf 3.11.3 in the tree, so no packages were needed for this. I verified this with running pip download --only-binary=:all: -r ../chromite/venv/requirements.txt -d pip_packages/ --find-links pip_packages/ and it only installed one additional package (grpcio-1.33) but said that was to satisfy grpcio>=1.0rc1, but we are currently on 1.0.0. pip download commands used: pip download --python-version 36 --abi cp36m "google_api_core==1.19.0" --only-binary=:all: --dest=pip_packages pip download --python-version 36 --abi cp36m "protobuf==3.11.3" --only-binary=:all: --dest=pip_packages pip download --python-version 36 --abi cp36m "google_auth==1.14.0" --only-binary=:all: --dest=pip_packages Additional files pulled in from new pinned versions in requirements.txt, which will be a follow-on CL. Change-Id: If68bc9cf74f8b5ee0e3c0e8a023b095fab81959b Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/infra_virtualenv/+/2481614 Reviewed-by: Chris McDonald <cjmcdonald@chromium.org> Tested-by: Michael Mortensen <mmortensen@google.com> Commit-Queue: Michael Mortensen <mmortensen@google.com>
This repository provides a common Python virtualenv interface that Chromium OS infrastructure code can depend on.
A repository adding virtualenv should mimic this repository, which itself uses virtualenv for running unit tests.
Key files:
bin/python_venv starts an instance of Python that uses the virtualenv.bin/turtle is an example script for running a Python module using bin/python_venv.venv/requirements.txt lists the packages to install inside the virtualenv.venv is added to PYTHONPATH. For example, venv/cros_venv can be imported inside the virtualenv using import cros_venv.To make sure all your requirements are pinned, run:
bin/python_venv -m pip freeze
Copy the output into requirements.txt
Refer to Pip's documentation for the requirements.txt format.
Packages to be installed inside a virtualenv must first be added to pip_packages.
To add packages, run:
bin/python_venv -m pip wheel -w pip_packages foo==1.2.3
Refer to Pip's documentation for details on the arguments for pip.
If the resultant wheel contains linux and not manylinux, then it is NOT portable. You will need to build a portable wheel; see next section.
Commit the wheel and make a CL.
You need to use the standard manylinux Docker image to build the portable wheel.
Install Docker:
https://g3doc.corp.google.com/cloud/containers/g3doc/glinux-docker/install.md?cl=head
Add the manylinux Docker image:
docker pull quay.io/pypa/manylinux2010_x86_64@sha256:6ccd12df1a01003214df0211f16d2de70d17eba35f8e7b0fff2d66c18cb0434e
Get the image ID:
docker image ls quay.io/pypa/manylinux2010_x86_64
Run the Docker image:
docker run -i -t -v "$(pwd)/pip_packages:/pip_packages" --rm $IMAGE /bin/bash
Build a portable wheel:
cd /opt/python/cp27-cp27mu bin/pip wheel -w /pip_packages foo==1.2.3 auditwheel repair --plat manylinux2010_x86_64 -w /pip_packages /pip_packages/foo-1.2.3-cp27-cp27mu-linux_x86_64.whl
Keep the manylinux wheel and remove the linux wheel.
Since these wheels are built as root inside the container, you may need to chown the files on the host to prevent permission issues later:
sudo chown $(id -un):$(id -gn) pip_packages/*
Add the packages to requirements.txt. If the packages are not in pip_packages yet, add the packages to pip_packages.
“First party modules” refers to Chromium OS code (anything checked out by repo).
NOTE: Do not use this for third party dependencies (stuff not owned by Chromium OS)! This should only be used to set up imports for stuff we own. For example, importing python-MySQL SHOULD NOT use this, but importing chromite MAY use this.
There are two ways to do this:
venv.sys.path in __init__.py.Adding a symlink to venv is simple and should be self-explanatory. However, keep in mind that repo checkouts may not always have the same structure, and certain environments such as production servers may check out repositories in completely different locations. This method is not powerful enough to account for these environments.
Modifying sys.path is a lot more powerful. The way to do this is to add a small bit of code to the __init__.py of the package that needs the import.
Example (do not copy and paste blindly):
import os
import sys
# The path of the package
PKGDIR = __path__[0]
# Paths to check
_PATH1 = os.path.join(PKGDIR, '../foo')
_PATH2 = '/opt/foo'
if os.path.exists(_PATH1):
sys.path.append(_PATH1)
elif os.path.exists(_PATH2):
sys.path.append(_PATH2)
else:
raise ImportError('foo not found')
You must also add the contents of the other project's requirements.txt to your project. We do not attempt to resolve dependencies recursively as that is very difficult.
The bin/create_venv script prepares a virtualenv using a requirements.txt file.
bin/create_venv requirements.txt
The script will print the path to the virtualenv to stdout. Note that the output ends with a newline; Bash handles this, but Python does not.
To run the virtualenv Python, call bin/python under the virtualenv directory.
Together, this might look up:
venv=$(bin/create_venv requirements.txt)
${venv}/bin/python
NOTE: It is not generally safe to run the other scripts in the virtualenv's bin directory due to hard-coded paths. Instead of running bin/pip for example, use bin/python -m pip.