Make Linux binary wheels portable BUG=chromium:787545 TEST=bin/run_tests TEST=rm -rf ~/.cache/cros_venv TEST=in chromeos-admin, bin/test_venv_packages TEST=in autotest, bin/test_lucifer TEST=in chromtie, bin/sysmon --help Change-Id: Ib050d178dfe1ffef2ba2a32ef11ac3b988e9c14d Reviewed-on: https://chromium-review.googlesource.com/776208 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org>
diff --git a/README.md b/README.md index bd9e17e..0b382ac 100644 --- a/README.md +++ b/README.md
@@ -15,14 +15,21 @@ * `bin/python_venv` starts an instance of Python that uses the virtualenv. * `bin/turtle` is an example script for running a Python module using - `bin/python_venv` + `bin/python_venv`. * `venv/requirements.txt` lists the packages to install inside the - virtualenv. Refer to Pip's - [documentation](https://pip.pypa.io/en/stable/) for the - `requirements.txt` format. + virtualenv. * `venv` is added to `PYTHONPATH`. For example, `venv/cros_venv` can be imported inside the virtualenv using `import cros_venv`. +To make sure all your requirements are pinned, run: + + bin/python_venv -m pip freeze + +Copy the output into `requirements.txt` + +Refer to Pip's [documentation](https://pip.pypa.io/en/stable/) for the +`requirements.txt` format. + ## Adding packages to be available for use Packages to be installed inside a virtualenv must first be added to @@ -30,18 +37,54 @@ To add packages, run: - $ bin/python_venv -m pip wheel -w pip_packages <packages to install> - + bin/python_venv -m pip wheel -w pip_packages foo==1.2.3 + Refer to Pip's documentation for details on the arguments for `pip`. -Commit the added package and make a CL. +If the resultant wheel contains `linux` and not `manylinux`, then it +is NOT portable. You will need to build a portable wheel; see next +section. + +Commit the wheel and make a CL. + +## Building portable wheel with C extensions + +You need to use the standard manylinux Docker image to build the +portable wheel. + +Install Docker. + +Add the manylinux Docker image: + + docker pull quay.io/pypa/manylinux1_x86_64@sha256:18be396acea7199c38f943be794d24405c0d84fcfdef15e2599084ec7810bba9 + +Get the image ID: + + docker image ls + +Run the Docker image: + + docker run -i -t -v "$(pwd)/pip_packages:/pip_packages" --rm $IMAGE /bin/bash + +Build a portable wheel: + + /opt/python/cp27-cp27mu pip wheel -w /pip_packages foo==1.2.3 + auditwheel repair -w /pip_packages /pip_packages/foo-1.2.3-cp27-cp27mu-linux_x86_64.whl + +Keep the `manylinux` wheel and remove the `linux` wheel. + +Since these wheels are built as root inside the container, you may +need to chown the files on the host to prevent permission issues +later: + + sudo chown $(id -un):$(id -gn) pip_packages/* ## Adding third party packages to a virtualenv Add the packages to `requirements.txt`. If the packages are not in `pip_packages` yet, add the packages to `pip_packages`. -## Adding first party modules to a virtualenv +## Adding first party modules to a virtualenv's import path "First party modules" refers to Chromium OS code (anything checked out by `repo`). @@ -76,7 +119,7 @@ # Paths to check _PATH1 = os.path.join(PKGDIR, '../foo') _PATH2 = '/opt/foo' - + if os.path.exists(_PATH1): sys.path.append(_PATH1) elif os.path.exists(_PATH2): @@ -93,7 +136,7 @@ The `bin/create_venv` script prepares a virtualenv using a `requirements.txt` file. - $ bin/create_venv requirements.txt + bin/create_venv requirements.txt The script will print the path to the virtualenv to stdout. Note that the output ends with a newline; Bash handles this, but Python does @@ -104,8 +147,8 @@ Together, this might look up: - $ venv=$(bin/create_venv requirements.txt) - $ ${venv}/bin/python + venv=$(bin/create_venv requirements.txt) + ${venv}/bin/python NOTE: It is not generally safe to run the other scripts in the virtualenv's `bin` directory due to hard-coded paths. Instead of
diff --git a/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-linux_x86_64.whl b/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-linux_x86_64.whl deleted file mode 100644 index 6bcf4a7..0000000 --- a/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-linux_x86_64.whl +++ /dev/null Binary files differ
diff --git a/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-manylinux1_x86_64.whl new file mode 100644 index 0000000..6ba154e --- /dev/null +++ b/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl b/pip_packages/PyYAML-3.12-cp27-cp27mu-manylinux1_x86_64.whl similarity index 77% rename from pip_packages/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl rename to pip_packages/PyYAML-3.12-cp27-cp27mu-manylinux1_x86_64.whl index 2abaf75..e494eb1 100644 --- a/pip_packages/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl +++ b/pip_packages/PyYAML-3.12-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-linux_x86_64.whl b/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-manylinux1_x86_64.whl similarity index 87% rename from pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-linux_x86_64.whl rename to pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-manylinux1_x86_64.whl index aacd359..d0d0f2c 100644 --- a/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-linux_x86_64.whl +++ b/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/coverage-4.4.1-cp27-none-linux_x86_64.whl b/pip_packages/coverage-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl similarity index 73% rename from pip_packages/coverage-4.4.1-cp27-none-linux_x86_64.whl rename to pip_packages/coverage-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl index b159977..48529e0 100644 --- a/pip_packages/coverage-4.4.1-cp27-none-linux_x86_64.whl +++ b/pip_packages/coverage-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/lazy_object_proxy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/lazy_object_proxy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl new file mode 100644 index 0000000..3f37053 --- /dev/null +++ b/pip_packages/lazy_object_proxy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/lazy_object_proxy-1.2.2-cp27-none-linux_x86_64.whl b/pip_packages/lazy_object_proxy-1.2.2-cp27-none-linux_x86_64.whl deleted file mode 100644 index 8973090..0000000 --- a/pip_packages/lazy_object_proxy-1.2.2-cp27-none-linux_x86_64.whl +++ /dev/null Binary files differ
diff --git a/pip_packages/psutil-4.3.1-cp27-cp27mu-linux_x86_64.whl b/pip_packages/psutil-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl similarity index 72% rename from pip_packages/psutil-4.3.1-cp27-cp27mu-linux_x86_64.whl rename to pip_packages/psutil-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl index a4189cb..14df460 100644 --- a/pip_packages/psutil-4.3.1-cp27-cp27mu-linux_x86_64.whl +++ b/pip_packages/psutil-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/pycrypto-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/pycrypto-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl new file mode 100644 index 0000000..a0a39af --- /dev/null +++ b/pip_packages/pycrypto-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/pycrypto-2.6.1-cp27-none-linux_x86_64.whl b/pip_packages/pycrypto-2.6.1-cp27-none-linux_x86_64.whl deleted file mode 100644 index 2a9a7d7..0000000 --- a/pip_packages/pycrypto-2.6.1-cp27-none-linux_x86_64.whl +++ /dev/null Binary files differ
diff --git a/pip_packages/scandir-1.5-cp27-cp27mu-linux_x86_64.whl b/pip_packages/scandir-1.5-cp27-cp27mu-linux_x86_64.whl deleted file mode 100644 index 195d4ec..0000000 --- a/pip_packages/scandir-1.5-cp27-cp27mu-linux_x86_64.whl +++ /dev/null Binary files differ
diff --git a/pip_packages/scandir-1.5-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/scandir-1.5-cp27-cp27mu-manylinux1_x86_64.whl new file mode 100644 index 0000000..bcf012a --- /dev/null +++ b/pip_packages/scandir-1.5-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/scandir-1.5-cp27-none-linux_x86_64.whl b/pip_packages/scandir-1.5-cp27-none-linux_x86_64.whl deleted file mode 100644 index 3a2df92..0000000 --- a/pip_packages/scandir-1.5-cp27-none-linux_x86_64.whl +++ /dev/null Binary files differ
diff --git a/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-linux_x86_64.whl b/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-linux_x86_64.whl deleted file mode 100644 index 2fa0e6d..0000000 --- a/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-linux_x86_64.whl +++ /dev/null Binary files differ
diff --git a/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-manylinux1_x86_64.whl new file mode 100644 index 0000000..7058ba1 --- /dev/null +++ b/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/wrapt-1.10.8-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/wrapt-1.10.8-cp27-cp27mu-manylinux1_x86_64.whl new file mode 100644 index 0000000..46aa2bc --- /dev/null +++ b/pip_packages/wrapt-1.10.8-cp27-cp27mu-manylinux1_x86_64.whl Binary files differ
diff --git a/pip_packages/wrapt-1.10.8-cp27-none-linux_x86_64.whl b/pip_packages/wrapt-1.10.8-cp27-none-linux_x86_64.whl deleted file mode 100644 index e8f888b..0000000 --- a/pip_packages/wrapt-1.10.8-cp27-none-linux_x86_64.whl +++ /dev/null Binary files differ