Clean up and update README
BUG=chromium:708735
TEST=None
Change-Id: I0b8655ba220f6ccc46cf2afed199195f0286fc5c
Reviewed-on: https://chromium-review.googlesource.com/470846
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
diff --git a/README.md b/README.md
index efd928b..21252ed 100644
--- a/README.md
+++ b/README.md
@@ -2,12 +2,23 @@
=======================
This repository provides a common Python virtualenv interface that
-infra code (such as chromite) can depend on. At this point, it is
-experimental and not yet used in production.
+infra code (such as chromite) can depend on.
-Virtualenv users should create a `requirements.txt` file listing the
-packages that they need and use the wrapper scripts (described below)
-to create the virtualenv and run commands within it.
+Usage
+-----
+
+Virtualenv users should mimic this repository, which itself uses
+virtualenv for running unit tests.
+
+1. Create a `venv` directory. All packages and modules in this
+ directory will be importable inside the virtualenv.
+2. Create a `requirements.txt` file inside `venv` to list external
+ packages to install.
+3. Copy `bin/find_virtualenv.sh` and `bin/turtle` which serve as
+ templates.
+
+Adding pre-built packages
+-------------------------
To add packages to this repository, run:
@@ -15,46 +26,37 @@
Commit the changes and make a CL.
-For example for chromite, from within `chromite/virtualenv`, run:
+Low level API
+-------------
- $ pip wheel -w pip_packages -r requirements.txt
-
-Wrapper scripts
----------------
-
-`create_venv` creates or updates a virtualenv using a
+The `create_venv` script prepares a virtualenv using a
`requirements.txt` file.
- $ create_venv .venv requirements.txt
+ $ create_venv requirements.txt
-To run the virtualenv python, use:
+The script will print the path to the virtualenv to stdout. Note that
+the output ends with a newline; Bash handles this, but Python does
+not.
- $ .venv/bin/python
+To run the virtualenv Python, call `bin/python` under the virtualenv
+directory.
-NOTE: it is not generally safe to run the other scripts in `.venv/bin`
-due to the hard-coded paths in the virtualenv. Instead of running
-`.venv/bin/pip` for example, use `.venv/bin/python -m pip`.
+Together, this might look up:
-Here’s a complete example:
+ $ venv=$(create_venv requirements.txt)
+ $ ${venv}/bin/python
- $ echo mock==2.0.0 > requirements.txt
- $ ./create_venv .venv requirements.txt
- $ .venv/bin/python
- Python 2.7.6 (default, Jun 22 2015, 17:58:13)
- [GCC 4.8.2] on linux2
- Type "help", "copyright", "credits" or "license" for more information.
- >>> import sys
- >>> sys.prefix # This points to the virtualenv now
- '/usr/local/google/home/ayatane/src/chromiumos/infra_virtualenv/.venv'
- >>> import mock
+NOTE: it is not generally safe to run the other scripts in the
+virtualenv's `bin` directory due to hard-coded paths. Instead of
+running `bin/pip` for example, use `bin/python -m pip`.
Adding arbitrary directories to import path
-------------------------------------------
NOTE: Do not use this for third party dependencies (stuff not owned by
ChromiumOS)! This should only be used to set up imports for stuff we
-own. For example, importing python-MySQL should NOT use this, but
-importing chromite from Autotest may use this.
+own. For example, importing python-MySQL SHOULD NOT use this, but
+importing chromite from Autotest MAY use this.
This should be handled by the minimum amount of code in the package's
`__init__.py` file.
@@ -69,51 +71,14 @@
_chromite_parent = 'site-packages'
sys.path.append(_chromite_parent)
-A solid understanding of the [Python import system](https://docs.python.org/3/reference/import.html)
-is recommended (link is for Python 3, but is informative).
+A solid understanding of the Python [import system][1] is recommended
+(the link is for Python 3, but it is informative).
-In brief, `__init__.py` is executed whenever the package is imported.
-The package is imported before any submodule or subpackage is
-imported. The package is only imported once per Python process;
-future imports get the "cached" "singleton" package object. Thus,
-`__init__.py` will modify `sys.path` exactly once and is guaranteed to
-be run before anything in that package is used.
+[1]: https://docs.python.org/3/reference/import.html
-Background for __init__.py recommended usage
---------------------------------------------
-
-(Updated on 2017-02-21)
-
-Previously, we set up the import path for first party modules by
-patching sys.path in very creative ways. This tends to cause
-problems.
-
-In the original virtualenv design, first party packages would be
-handled by using `pip -e` to install the packages inside the
-virtualenv in editable mode. However, the implementation of this is
-wonky, buggy, and generally considered a second citizen to installing
-packages "properly". The blocking issue is that pip -e must copy the
-entire source tree internally, all for writing a metadata file and
-what amounts to a symlink. This takes a considerable amount of time
-for large packages such as chromite (the .git directory is copied
-also). This is not trivially fixed upstream, and the editable
-installs feature is not considered a top priority.
-
-Thus, the reason for adding our own import path patching is to work
-around pip while solving our existing woes.
-
-A test concept of the feature was implemented using .pth files, which
-are simple files that contain paths to add to Python’s import path.
-
-The problem with this implementation is that our code is run in a lot
-of really weird configurations. Having a single .pth file may not be
-good enough. While relative paths are supported (and sane), some of
-the places our code is run do not use the same file system hierarchy.
-Also, there is no simple way to handle recursive requirements.
-
-Thus, going forward, the standard way to support first party imports
-is a small bit of `sys.path` patching code in the respective package’s
-`__init__.py` file. This enable the use of Python's full power for
-handling weird environments as needed; a little logic goes a long way.
-A lot of other things also just work due to `__init__.py` file
-semantics: for example, recursive requirements.
+In brief, `__init__.py` is executed whenever a package is imported. A
+package is imported before any submodule or subpackage is imported. A
+package is only imported once per Python process; future imports are
+looked up in `sys.modules`. Thus, `__init__.py` will modify
+`sys.path` exactly once and is guaranteed to be run before anything in
+that package is imported.
