Make Linux binary wheels portable
BUG=chromium:787545
TEST=bin/run_tests
TEST=rm -rf ~/.cache/cros_venv
TEST=in chromeos-admin, bin/test_venv_packages
TEST=in autotest, bin/test_lucifer
TEST=in chromtie, bin/sysmon --help
Change-Id: Ib050d178dfe1ffef2ba2a32ef11ac3b988e9c14d
Reviewed-on: https://chromium-review.googlesource.com/776208
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
diff --git a/README.md b/README.md
index bd9e17e..0b382ac 100644
--- a/README.md
+++ b/README.md
@@ -15,14 +15,21 @@
* `bin/python_venv` starts an instance of Python that uses the
virtualenv.
* `bin/turtle` is an example script for running a Python module using
- `bin/python_venv`
+ `bin/python_venv`.
* `venv/requirements.txt` lists the packages to install inside the
- virtualenv. Refer to Pip's
- [documentation](https://pip.pypa.io/en/stable/) for the
- `requirements.txt` format.
+ virtualenv.
* `venv` is added to `PYTHONPATH`. For example, `venv/cros_venv` can
be imported inside the virtualenv using `import cros_venv`.
+To make sure all your requirements are pinned, run:
+
+ bin/python_venv -m pip freeze
+
+Copy the output into `requirements.txt`
+
+Refer to Pip's [documentation](https://pip.pypa.io/en/stable/) for the
+`requirements.txt` format.
+
## Adding packages to be available for use
Packages to be installed inside a virtualenv must first be added to
@@ -30,18 +37,54 @@
To add packages, run:
- $ bin/python_venv -m pip wheel -w pip_packages <packages to install>
-
+ bin/python_venv -m pip wheel -w pip_packages foo==1.2.3
+
Refer to Pip's documentation for details on the arguments for `pip`.
-Commit the added package and make a CL.
+If the resultant wheel contains `linux` and not `manylinux`, then it
+is NOT portable. You will need to build a portable wheel; see next
+section.
+
+Commit the wheel and make a CL.
+
+## Building portable wheel with C extensions
+
+You need to use the standard manylinux Docker image to build the
+portable wheel.
+
+Install Docker.
+
+Add the manylinux Docker image:
+
+ docker pull quay.io/pypa/manylinux1_x86_64@sha256:18be396acea7199c38f943be794d24405c0d84fcfdef15e2599084ec7810bba9
+
+Get the image ID:
+
+ docker image ls
+
+Run the Docker image:
+
+ docker run -i -t -v "$(pwd)/pip_packages:/pip_packages" --rm $IMAGE /bin/bash
+
+Build a portable wheel:
+
+ /opt/python/cp27-cp27mu pip wheel -w /pip_packages foo==1.2.3
+ auditwheel repair -w /pip_packages /pip_packages/foo-1.2.3-cp27-cp27mu-linux_x86_64.whl
+
+Keep the `manylinux` wheel and remove the `linux` wheel.
+
+Since these wheels are built as root inside the container, you may
+need to chown the files on the host to prevent permission issues
+later:
+
+ sudo chown $(id -un):$(id -gn) pip_packages/*
## Adding third party packages to a virtualenv
Add the packages to `requirements.txt`. If the packages are not in
`pip_packages` yet, add the packages to `pip_packages`.
-## Adding first party modules to a virtualenv
+## Adding first party modules to a virtualenv's import path
"First party modules" refers to Chromium OS code (anything checked out
by `repo`).
@@ -76,7 +119,7 @@
# Paths to check
_PATH1 = os.path.join(PKGDIR, '../foo')
_PATH2 = '/opt/foo'
-
+
if os.path.exists(_PATH1):
sys.path.append(_PATH1)
elif os.path.exists(_PATH2):
@@ -93,7 +136,7 @@
The `bin/create_venv` script prepares a virtualenv using a
`requirements.txt` file.
- $ bin/create_venv requirements.txt
+ bin/create_venv requirements.txt
The script will print the path to the virtualenv to stdout. Note that
the output ends with a newline; Bash handles this, but Python does
@@ -104,8 +147,8 @@
Together, this might look up:
- $ venv=$(bin/create_venv requirements.txt)
- $ ${venv}/bin/python
+ venv=$(bin/create_venv requirements.txt)
+ ${venv}/bin/python
NOTE: It is not generally safe to run the other scripts in the
virtualenv's `bin` directory due to hard-coded paths. Instead of
diff --git a/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-linux_x86_64.whl b/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
deleted file mode 100644
index 6bcf4a7..0000000
--- a/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
+++ /dev/null
Binary files differ
diff --git a/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-manylinux1_x86_64.whl
new file mode 100644
index 0000000..6ba154e
--- /dev/null
+++ b/pip_packages/MySQL_python-1.2.5-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl b/pip_packages/PyYAML-3.12-cp27-cp27mu-manylinux1_x86_64.whl
similarity index 77%
rename from pip_packages/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl
rename to pip_packages/PyYAML-3.12-cp27-cp27mu-manylinux1_x86_64.whl
index 2abaf75..e494eb1 100644
--- a/pip_packages/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl
+++ b/pip_packages/PyYAML-3.12-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-linux_x86_64.whl b/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-manylinux1_x86_64.whl
similarity index 87%
rename from pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-linux_x86_64.whl
rename to pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-manylinux1_x86_64.whl
index aacd359..d0d0f2c 100644
--- a/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-linux_x86_64.whl
+++ b/pip_packages/SQLAlchemy-1.0.15-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/coverage-4.4.1-cp27-none-linux_x86_64.whl b/pip_packages/coverage-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl
similarity index 73%
rename from pip_packages/coverage-4.4.1-cp27-none-linux_x86_64.whl
rename to pip_packages/coverage-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl
index b159977..48529e0 100644
--- a/pip_packages/coverage-4.4.1-cp27-none-linux_x86_64.whl
+++ b/pip_packages/coverage-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/lazy_object_proxy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/lazy_object_proxy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl
new file mode 100644
index 0000000..3f37053
--- /dev/null
+++ b/pip_packages/lazy_object_proxy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/lazy_object_proxy-1.2.2-cp27-none-linux_x86_64.whl b/pip_packages/lazy_object_proxy-1.2.2-cp27-none-linux_x86_64.whl
deleted file mode 100644
index 8973090..0000000
--- a/pip_packages/lazy_object_proxy-1.2.2-cp27-none-linux_x86_64.whl
+++ /dev/null
Binary files differ
diff --git a/pip_packages/psutil-4.3.1-cp27-cp27mu-linux_x86_64.whl b/pip_packages/psutil-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl
similarity index 72%
rename from pip_packages/psutil-4.3.1-cp27-cp27mu-linux_x86_64.whl
rename to pip_packages/psutil-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl
index a4189cb..14df460 100644
--- a/pip_packages/psutil-4.3.1-cp27-cp27mu-linux_x86_64.whl
+++ b/pip_packages/psutil-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/pycrypto-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/pycrypto-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
new file mode 100644
index 0000000..a0a39af
--- /dev/null
+++ b/pip_packages/pycrypto-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/pycrypto-2.6.1-cp27-none-linux_x86_64.whl b/pip_packages/pycrypto-2.6.1-cp27-none-linux_x86_64.whl
deleted file mode 100644
index 2a9a7d7..0000000
--- a/pip_packages/pycrypto-2.6.1-cp27-none-linux_x86_64.whl
+++ /dev/null
Binary files differ
diff --git a/pip_packages/scandir-1.5-cp27-cp27mu-linux_x86_64.whl b/pip_packages/scandir-1.5-cp27-cp27mu-linux_x86_64.whl
deleted file mode 100644
index 195d4ec..0000000
--- a/pip_packages/scandir-1.5-cp27-cp27mu-linux_x86_64.whl
+++ /dev/null
Binary files differ
diff --git a/pip_packages/scandir-1.5-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/scandir-1.5-cp27-cp27mu-manylinux1_x86_64.whl
new file mode 100644
index 0000000..bcf012a
--- /dev/null
+++ b/pip_packages/scandir-1.5-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/scandir-1.5-cp27-none-linux_x86_64.whl b/pip_packages/scandir-1.5-cp27-none-linux_x86_64.whl
deleted file mode 100644
index 3a2df92..0000000
--- a/pip_packages/scandir-1.5-cp27-none-linux_x86_64.whl
+++ /dev/null
Binary files differ
diff --git a/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-linux_x86_64.whl b/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-linux_x86_64.whl
deleted file mode 100644
index 2fa0e6d..0000000
--- a/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-linux_x86_64.whl
+++ /dev/null
Binary files differ
diff --git a/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-manylinux1_x86_64.whl
new file mode 100644
index 0000000..7058ba1
--- /dev/null
+++ b/pip_packages/subprocess32-3.5.0rc1-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/wrapt-1.10.8-cp27-cp27mu-manylinux1_x86_64.whl b/pip_packages/wrapt-1.10.8-cp27-cp27mu-manylinux1_x86_64.whl
new file mode 100644
index 0000000..46aa2bc
--- /dev/null
+++ b/pip_packages/wrapt-1.10.8-cp27-cp27mu-manylinux1_x86_64.whl
Binary files differ
diff --git a/pip_packages/wrapt-1.10.8-cp27-none-linux_x86_64.whl b/pip_packages/wrapt-1.10.8-cp27-none-linux_x86_64.whl
deleted file mode 100644
index e8f888b..0000000
--- a/pip_packages/wrapt-1.10.8-cp27-none-linux_x86_64.whl
+++ /dev/null
Binary files differ
