Migrate create_venv to Python
This commit came about for the purpose of adding a method for
importing first party (ChromeOS owned) Python modules inside
virtualenvs. Because this feature would have added another Python
script that create_venv would need to spawn and also because
create_venv is stretching the limits for a comfortably maintainable
shell script, the best course of action would be to port all of
create_venv to Python.
As a side effect of being written in Python, unit tests are now easier
to write, and the new cros_venv package has a number of unit tests.
Coupled with the expensive VM tests hosted elsewhere, this should
provide good coverage for virtualenv code, which will be crucial for
all of its users.
This Python rewrite is basically 100% backward compatible with the
previous version, with a few disclaimers:
- The locking mechanism changed from the flock utility to a Python
implementation of a symlink-based lock.
The flock utility was not used because it would be very awkward to
call from Python.
The new locking mechanism uses symlink locks instead of an flock(2)
based mechanism because flock(2) has quite a few gotchas, is
slightly trickier to implement, and extremely complicated to unit
test (due to one of said gotchas). As a bonus, the new symlink
implementation includes the process id of the owning process, which
may be useful.
- Recursive expansion of requirement files has been removed. This
feature is not used yet, and after careful consideration I decided
that the feature was not worth it.
The cost of removing the feature (the benefit of the feature) is
that first party packages that depend on other first party
packages (for example, Autotest depending on chromite) will need to
have their requirements file kept up to date with their dependencys
requirements file.
The cost of the feature is the maintenance of some dozen lines of
ad-hoc text processing code and a negligible performance cost on
every run.
However, the main motivation is that pips handling of recursive
relative paths (and thus, our handling of recursive relative paths)
is insane. Relative paths are resolved relative to the current
directory, regardless of the location of any recursive requirements
files. This is completely unmanageable and it is best to throw out
the entire feature than to rely on this insanity.
Having to keep requirement files in sync across repositories is
annoying, but quite simple and any errors would be easy to debug.
The subtleties of some twice recursive relative path or some other
insane situation that breaks due to some rename thats otherwise
completely unrelated would be FUN.
The original motivation for the migration to Python was to add a
method for importing first party (ChromeOS owned) Python modules
inside virtualenvs. Such a feature was not added. If such a feature
is still needed, it can be recovered from a patchset.
Currently, we set up the import path for first party modules by
patching sys.path in very creative ways. This tends to cause
problems.
In the original virtualenv design, first party packages would be
handled by using pip -e to install the packages inside the virtualenv
in editable mode. However, the implementation of this is wonky,
buggy, and generally considered a second citizen to installing
packages "properly". The blocking issue is that pip -e must copy the
entire source tree internally, all for writing a metadata file and
what amounts to a symlink. This takes a considerable amount of time
for large packages such as chromite (the .git directory is copied
also). This is not trivially fixed upstream, and the editable
installs feature is not considered a top priority.
Thus, the reason for adding our own import path patching is to work
around pip while solving our existing woes.
A test concept of the feature was implemented using .pth files, which
are simple files that contain paths to add to Pythons import path.
The problem with this implementation is that our code is run in a lot
of really weird configurations. Having a single .pth file may not be
good enough. While relative paths are supported (and sane), some of
the places our code is run do not use the same file system hierarchy.
Also, there is no simple way to handle recursive requirements.
Thus, going forward, the standard way to support first party imports
is a small bit of sys.path patching code in the respective packages
__init__.py file. This enable the use of Python for handling weird
environments as needed; a little logic goes a long way. A lot of
other things also just work due to __init__.py file semantics. The
details are in the README added by this commit.
However, the migration to Python is still, I think, a good thing,
so I am keeping this commit, removing the .pth handling parts.
BUG=None
TEST=Run unit tests and virtualenv VM tests
Change-Id: Ibd817b889acb74d62d7ce7ebc9c017341e2d3580
Reviewed-on: https://chromium-review.googlesource.com/444924
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..0d20b64
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+*.pyc
diff --git a/PRESUBMIT.cfg b/PRESUBMIT.cfg
new file mode 100644
index 0000000..97fce11
--- /dev/null
+++ b/PRESUBMIT.cfg
@@ -0,0 +1,2 @@
+[Hook Scripts]
+hook0: ./bin/run_tests
diff --git a/README.md b/README.md
index 620899b..efd928b 100644
--- a/README.md
+++ b/README.md
@@ -47,3 +47,73 @@
>>> sys.prefix # This points to the virtualenv now
'/usr/local/google/home/ayatane/src/chromiumos/infra_virtualenv/.venv'
>>> import mock
+
+Adding arbitrary directories to import path
+-------------------------------------------
+
+NOTE: Do not use this for third party dependencies (stuff not owned by
+ChromiumOS)! This should only be used to set up imports for stuff we
+own. For example, importing python-MySQL should NOT use this, but
+importing chromite from Autotest may use this.
+
+This should be handled by the minimum amount of code in the package's
+`__init__.py` file.
+
+Example:
+
+ """Autotest package."""
+
+ import sys
+
+ # Use the minimum amount of logic to find the path to add
+ _chromite_parent = 'site-packages'
+ sys.path.append(_chromite_parent)
+
+A solid understanding of the [Python import system](https://docs.python.org/3/reference/import.html)
+is recommended (link is for Python 3, but is informative).
+
+In brief, `__init__.py` is executed whenever the package is imported.
+The package is imported before any submodule or subpackage is
+imported. The package is only imported once per Python process;
+future imports get the "cached" "singleton" package object. Thus,
+`__init__.py` will modify `sys.path` exactly once and is guaranteed to
+be run before anything in that package is used.
+
+Background for __init__.py recommended usage
+--------------------------------------------
+
+(Updated on 2017-02-21)
+
+Previously, we set up the import path for first party modules by
+patching sys.path in very creative ways. This tends to cause
+problems.
+
+In the original virtualenv design, first party packages would be
+handled by using `pip -e` to install the packages inside the
+virtualenv in editable mode. However, the implementation of this is
+wonky, buggy, and generally considered a second citizen to installing
+packages "properly". The blocking issue is that pip -e must copy the
+entire source tree internally, all for writing a metadata file and
+what amounts to a symlink. This takes a considerable amount of time
+for large packages such as chromite (the .git directory is copied
+also). This is not trivially fixed upstream, and the editable
+installs feature is not considered a top priority.
+
+Thus, the reason for adding our own import path patching is to work
+around pip while solving our existing woes.
+
+A test concept of the feature was implemented using .pth files, which
+are simple files that contain paths to add to Python’s import path.
+
+The problem with this implementation is that our code is run in a lot
+of really weird configurations. Having a single .pth file may not be
+good enough. While relative paths are supported (and sane), some of
+the places our code is run do not use the same file system hierarchy.
+Also, there is no simple way to handle recursive requirements.
+
+Thus, going forward, the standard way to support first party imports
+is a small bit of `sys.path` patching code in the respective package’s
+`__init__.py` file. This enable the use of Python's full power for
+handling weird environments as needed; a little logic goes a long way.
+A lot of other things also just work due to `__init__.py` file
+semantics: for example, recursive requirements.
diff --git a/bin/run_tests b/bin/run_tests
new file mode 100755
index 0000000..0f75f90
--- /dev/null
+++ b/bin/run_tests
@@ -0,0 +1,8 @@
+#!/bin/sh
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+set -eu
+cd "$(dirname "$0")/.."
+find . -name "*.pyc" -print0 | xargs -0 rm
+python2 -m unittest discover cros_venv
diff --git a/create_venv b/create_venv
index 1be6047..d146005 100755
--- a/create_venv
+++ b/create_venv
@@ -1,103 +1,14 @@
#!/bin/bash
-# Copyright 2016 The Chromium OS Authors. All rights reserved.
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# Create or update a virtualenv.
#
-# $ create_venv path/to/venv path/to/requirements.txt
+# $ create_venv path/to/venv path/to/requirements.txt [path/to/custom.pth]
+#
+# See the cros_venv Python package for details, or pass the --help
+# option.
set -eu
-
basedir=$(readlink -f "$(dirname "${BASH_SOURCE[0]}" )")
-pkgdir="$basedir/pip_packages"
-# base_deps are pip requirements automatically included in every
-# virtualenv so we have control over which versions of bootstrap
-# packages are installed.
-base_deps=( setuptools==28.2.0 pip==8.1.2 )
-
-main() {
- if [[ $# -ne 2 ]]; then
- print_help >&2
- exit 1
- fi
-
- local venv_dir=$1
- local requirements=$2
- local lock="$venv_dir/.create_venv.lock"
-
- mkdir -p "$venv_dir"
- (
- # Try to acquire a lock for 3 minutes
- if ! flock -w 180 9; then
- echo 'Failed to acquire lock on virtualenv' >&2
- exit 1
- fi
- if up_to_date "$venv_dir" "$requirements"; then
- echo "Existing virtualenv $venv_dir is already up to date"
- else
- echo "Creating or updating virtualenv in $venv_dir"
- init_venv "$venv_dir"
- install_packages_in_venv "$venv_dir" "$requirements"
- mark_up_to_date "$venv_dir" "$requirements"
- fi
- ) 9>"$lock"
-}
-
-print_help() {
- echo "Usage: $0 path/to/venv path/to/requirements.txt
-
-Create or update a Python virtualenv."
-}
-
-up_to_date() {
- local venv_dir=$1
- local requirements=$2
- local installed="$venv_dir/.installed.txt"
- print_reqs "$requirements" | cmp -s - "$installed"
-}
-
-mark_up_to_date() {
- local venv_dir=$1
- local requirements=$2
- local installed="$venv_dir/.installed.txt"
- print_reqs "$requirements" > "$installed"
-}
-
-print_reqs() {
- local requirements=$1
-
- for dep in "${base_deps[@]}"; do
- echo "$dep"
- done
- "$basedir/expand_reqs.py" <"$requirements" 2>/dev/null
-}
-
-init_venv() {
- local venv_dir=$1
- # TODO(ayatane): Ubuntu Precise ships with virtualenv 1.7, which
- # requires specifying --setuptools, else distribute is used (which
- # is deprecated). virtualenv after 1.10 supports setuptools by
- # default. virtualenv >1.10 accepts --setuptools but does not
- # document it. Once we no longer have any hosts on virtualenv 1.7,
- # --setuptools can be removed.
- virtualenv "$venv_dir" --extra-search-dir="$pkgdir" --setuptools
-}
-
-install_packages_in_venv() {
- local venv_dir=$1
- local requirements=$2
- local dep
- for dep in "${base_deps[@]}"; do
- venv_pip_install "$venv_dir" "$dep"
- done
- venv_pip_install "$venv_dir" -r "$requirements"
-}
-
-venv_pip_install() {
- local venv_dir=$1
- shift 1
- "$venv_dir/bin/python" \
- -m pip install --no-index -f "file://$pkgdir" "$@"
-}
-
-main "$@"
+PYTHONPATH="$basedir" /usr/bin/python -m cros_venv "$@"
diff --git a/cros_venv/__init__.py b/cros_venv/__init__.py
new file mode 100644
index 0000000..72ddb66
--- /dev/null
+++ b/cros_venv/__init__.py
@@ -0,0 +1,21 @@
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""This package contains modules for creating and updating virtualenvs.
+
+To run:
+
+ $ python2 -m venv
+
+See __main__.py for details.
+
+To run the tests:
+
+ $ python2 -m unittest discover venv
+"""
+
+import os
+
+PACKAGE_DIR = os.path.abspath(__path__[0])
+WORKTREE_DIR = os.path.dirname(PACKAGE_DIR)
diff --git a/cros_venv/__main__.py b/cros_venv/__main__.py
new file mode 100644
index 0000000..512c6c3
--- /dev/null
+++ b/cros_venv/__main__.py
@@ -0,0 +1,45 @@
+#/usr/bin/python2
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Create or update a virtualenv.
+
+See __init__.py for how to run this module.
+"""
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import argparse
+import logging
+
+from cros_venv import venvlib
+
+
+def main():
+ """See module docstring."""
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('venv_dir')
+ parser.add_argument('reqs_file')
+ parser.add_argument('--verbose', action='store_true')
+ args = parser.parse_args()
+
+ configure_logging(args.verbose)
+
+ venv = venvlib.Venv(
+ venv_dir=args.venv_dir,
+ reqs_file=args.reqs_file)
+ venv.ensure()
+
+
+def configure_logging(verbose):
+ if verbose:
+ logging.basicConfig(level='DEBUG')
+ else:
+ logging.getLogger().addHandler(logging.NullHandler())
+
+
+if __name__ == '__main__':
+ main()
diff --git a/cros_venv/flock.py b/cros_venv/flock.py
new file mode 100644
index 0000000..0543285
--- /dev/null
+++ b/cros_venv/flock.py
@@ -0,0 +1,133 @@
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Lock files"""
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import errno
+import fcntl
+import logging
+import os
+import random
+import time
+
+
+logger = logging.getLogger(__name__)
+
+
+class _ProcDir(object):
+ """Class wrapping /proc related functions."""
+
+ def __init__(self, procdir):
+ self._procdir = procdir
+
+ def getpath(self, pid):
+ """Return /proc dir path for pid."""
+ return os.path.join(self._procdir, str(pid))
+
+ def getpid(self, path):
+ """Return int pid for /proc path."""
+ proc_prefix_len = len(self._procdir) + 1 # Handle extra ending slash
+ pid_string = path[proc_prefix_len:]
+ return int(pid_string)
+
+_procdir = _ProcDir('/proc')
+
+
+def _is_pid_running(pid):
+ """Return True if pid is running."""
+ try:
+ os.kill(pid, 0)
+ except OSError as e:
+ return e.errno != errno.ESRCH
+ else:
+ return True
+
+
+class FileLock(object):
+ """Context manager for an exclusive file lock."""
+
+ def __init__(self, lockfile, timeout=180):
+ """Initialize instance.
+
+ Args:
+ lockfile: Path to lockfile
+ timeout: Timeout for grabbing lock, in seconds.
+ """
+ self._lockfile = lockfile
+ self._timeout = timeout
+
+ @property
+ def _proc_path(self):
+ return _procdir.getpath(os.getpid())
+
+ def __enter__(self):
+ self._acquire_lock()
+ return self
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ self._release_lock()
+
+ def _acquire_lock(self):
+ """Try to acquire lock within timeout."""
+ timeout = time.time() + self._timeout
+ self._clean_stale_lock()
+ while True:
+ try:
+ os.symlink(self._proc_path, self._lockfile)
+ except OSError as e:
+ logger.warning('Error acquiring lock: %s', e)
+ else:
+ return
+ if time.time() > timeout:
+ break
+ time.sleep(random.random())
+ raise Exception('Trying to acquire lock %r timed out.'
+ % (self._lockfile,))
+
+ def _release_lock(self):
+ """Release own lock."""
+ if os.readlink(self._lockfile) != self._proc_path:
+ logger.warning('Lockfile %s does not belong to this process',
+ self._lockfile)
+ else:
+ os.unlink(self._lockfile)
+
+ def _clean_stale_lock(self):
+ """Clean up stale locks."""
+ # TODO(ayatane): Old flock(2) lock file handling can be removed
+ # after the code has existed for Long Enough that there aren't
+ # reasonably any instances of it left on developer or prod
+ # machines.
+ self._clean_flock_lock()
+ self._clean_orphaned_lock()
+
+ def _clean_flock_lock(self):
+ """Clean up old flock(2)-based lock."""
+ if self._is_lock_flock():
+ logger.warning('Removing old style lock file')
+ os.unlink(self._lockfile)
+
+ def _clean_orphaned_lock(self):
+ """Clean up orphaned lock."""
+ if self._is_lock_orphaned():
+ logger.warning('Removing orphaned lock')
+ os.unlink(self._lockfile)
+
+ def _is_lock_flock(self):
+ """Return True if old flock(2) lock exists."""
+ return (os.path.isfile(self._lockfile)
+ and not os.path.islink(self._lockfile))
+
+ def _is_lock_orphaned(self):
+ """Return True if lock is orphaned."""
+ return (os.path.islink(self._lockfile)
+ and not _is_pid_running(self._get_lock_pid()))
+
+ def _get_lock_pid(self):
+ """Return pid of current lock file as int."""
+ return _procdir.getpid(os.readlink(self._lockfile))
diff --git a/cros_venv/test_flock.py b/cros_venv/test_flock.py
new file mode 100644
index 0000000..ad4e87e
--- /dev/null
+++ b/cros_venv/test_flock.py
@@ -0,0 +1,183 @@
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import errno
+import fcntl
+import logging
+import mock
+import os
+import unittest
+
+from cros_venv import testcases
+from cros_venv import flock
+
+
+_nullhandler = logging.NullHandler()
+
+
+def setUpModule():
+ flock.logger.addHandler(_nullhandler)
+
+
+def tearDownModule():
+ flock.logger.removeHandler(_nullhandler)
+
+
+class ProcDirTestCase(unittest.TestCase):
+ """Tests for _ProcDir."""
+
+ def test_getpath(self):
+ """Test getpath()."""
+ got = flock._procdir.getpath(123)
+ self.assertEqual(got, '/proc/123')
+
+ def test_getpid(self):
+ """Test getpid()."""
+ got = flock._procdir.getpid('/proc/123')
+ self.assertEqual(got, 123)
+
+
+class IsPidRunningTestCase(unittest.TestCase):
+ """Tests for _is_pid_running()."""
+
+ @mock.patch('os.kill')
+ def test_no_such_process(self, kill):
+ """Test _is_pid_running() when no such process."""
+ kill.side_effect = OSError(errno.ESRCH, '')
+ self.assertFalse(flock._is_pid_running(1))
+
+ @mock.patch('os.kill')
+ def test_operation_not_permitted(self, kill):
+ """Test _is_pid_running() when operation not permitted."""
+ kill.side_effect = OSError(errno.EPERM, '')
+ self.assertTrue(flock._is_pid_running(1))
+
+ @mock.patch('os.kill')
+ def test_okay(self, _kill):
+ """Test _is_pid_running() when operation not permitted."""
+ self.assertTrue(flock._is_pid_running(1))
+
+
+class FileLockTestCase(testcases.TmpdirTestCase):
+ """Tests for flock."""
+
+ def test_symlink_fail(self):
+ """Test that symlink raises OSError when it already exists."""
+ os.symlink('tmp', 'foo')
+ with self.assertRaises(OSError):
+ os.symlink('tmp', 'foo')
+
+ def test_FileLock(self):
+ """Test that FileLock acquires and releases lock."""
+ self.assertFalse(os.path.lexists('lockfile'))
+ with flock.FileLock('lockfile', timeout=0):
+ self.assertTrue(os.path.lexists('lockfile'))
+ self.assertFalse(os.path.lexists('lockfile'))
+
+ def test_FileLock_proc_target(self):
+ """Test that FileLock lock points to process /proc."""
+ with flock.FileLock('lockfile', timeout=0):
+ self.assertEqual(os.readlink('lockfile'),
+ '/proc/' + str(os.getpid()))
+
+ def test_FileLock_timeout(self):
+ """Test that FileLock raises exception on timeout."""
+ os.symlink('tmp', 'lockfile')
+ with self.assertRaises(Exception):
+ with flock.FileLock('lockfile', timeout=0):
+ pass
+
+ def test__release_lock(self):
+ """Test _release_lock()."""
+ file_lock = flock.FileLock('lockfile')
+ os.symlink(file_lock._proc_path, 'lockfile')
+ file_lock._release_lock()
+ self.assertFalse(os.path.lexists('lockfile'))
+
+ def test__release_lock_other_process(self):
+ """Test _release_lock() ignores other process's lock."""
+ file_lock = flock.FileLock('lockfile')
+ os.symlink('/proc/1', 'lockfile')
+ file_lock._release_lock()
+ self.assertTrue(os.path.lexists('lockfile'))
+
+ def test__get_lock_pid(self):
+ """Test _get_lock_pid."""
+ file_lock = flock.FileLock('lockfile')
+ os.symlink('/proc/1', 'lockfile')
+ self.assertEqual(file_lock._get_lock_pid(), 1)
+
+ def test__is_lock_orphaned_own_lock(self):
+ """Test _is_lock_orphaned() with own lock."""
+ file_lock = flock.FileLock('lockfile')
+ os.symlink(file_lock._proc_path, 'lockfile')
+ self.assertFalse(file_lock._is_lock_orphaned())
+
+ def test__is_lock_orphaned_missing(self):
+ """Test _is_lock_orphaned() with missing lock."""
+ file_lock = flock.FileLock('lockfile')
+ self.assertFalse(file_lock._is_lock_orphaned())
+
+ @mock.patch.object(flock, '_is_pid_running')
+ def test__is_lock_orphaned(self, is_running):
+ """Test _is_lock_orphaned()."""
+ is_running.return_value = False
+ file_lock = flock.FileLock('lockfile')
+ os.symlink(file_lock._proc_path, 'lockfile')
+ self.assertTrue(file_lock._is_lock_orphaned())
+
+ def test__is_lock_flock(self):
+ """Test _is_lock_flock()."""
+ file_lock = flock.FileLock('lockfile')
+ with open('lockfile', 'w') as _f:
+ pass
+ self.assertTrue(file_lock._is_lock_flock())
+
+ def test__is_lock_flock_symlink(self):
+ """Test _is_lock_flock() with symlink lock."""
+ file_lock = flock.FileLock('lockfile')
+ os.symlink(file_lock._proc_path, 'lockfile')
+ self.assertFalse(file_lock._is_lock_flock())
+
+ def test__is_lock_flock_missing(self):
+ """Test _is_lock_flock() with missing lock."""
+ file_lock = flock.FileLock('lockfile')
+ self.assertFalse(file_lock._is_lock_flock())
+
+ @mock.patch.object(flock, '_is_pid_running')
+ def test__clean_orphaned_lock(self, is_running):
+ """Test _clean_orphaned_lock()."""
+ is_running.return_value = False
+ file_lock = flock.FileLock('lockfile')
+ os.symlink('/proc/1', 'lockfile')
+ file_lock._clean_orphaned_lock()
+ self.assertFalse(os.path.lexists('lockfile'))
+
+ @mock.patch.object(flock, '_is_pid_running')
+ def test__clean_orphaned_lock_owned(self, is_running):
+ """Test _clean_orphaned_lock() ignores owned lock."""
+ is_running.return_value = True
+ file_lock = flock.FileLock('lockfile')
+ os.symlink('/proc/1', 'lockfile')
+ file_lock._clean_orphaned_lock()
+ self.assertTrue(os.path.lexists('lockfile'))
+
+ def test__clean_flock_lock(self):
+ """Test _clean_flock_lock()."""
+ file_lock = flock.FileLock('lockfile')
+ with open('lockfile', 'w') as _f:
+ pass
+ file_lock._clean_flock_lock()
+ self.assertFalse(os.path.exists('lockfile'))
+
+ def test__clean_flock_lock_symlink(self):
+ """Test _clean_flock_lock() skips symlink lock."""
+ file_lock = flock.FileLock('lockfile')
+ os.symlink('/proc/1', 'lockfile')
+ file_lock._clean_flock_lock()
+ self.assertTrue(os.path.exists('lockfile'))
diff --git a/cros_venv/test_venvlib.py b/cros_venv/test_venvlib.py
new file mode 100644
index 0000000..57cd127
--- /dev/null
+++ b/cros_venv/test_venvlib.py
@@ -0,0 +1,101 @@
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import os
+import unittest
+
+from cros_venv import testcases
+from cros_venv import venvlib
+
+
+class VenvTestCase(unittest.TestCase):
+
+ """TestCase for venvlib."""
+
+ def test__PACKAGE_DIR(self):
+ """Test that _PACKAGE_DIR exists."""
+ self.assertTrue(os.path.isdir(venvlib._PACKAGE_DIR))
+
+ def test__iter_equal(self):
+ """Test when _iter_equal() is True."""
+ self.assertTrue(venvlib._iter_equal(
+ iter(['foo', 'bar']),
+ iter(['foo', 'bar']),
+ ))
+
+ def test__iter_equal_wrong_length(self):
+ """Test when _iter_equal() is False due to length."""
+ self.assertFalse(venvlib._iter_equal(
+ iter(['foo', 'bar', 'baz']),
+ iter(['foo', 'bar']),
+ ))
+
+ def test__iter_equal_fill_with_None(self):
+ """Test that _iter_equal() is True due to filling with None."""
+ self.assertTrue(venvlib._iter_equal(
+ iter(['foo', 'bar', None]),
+ iter(['foo', 'bar']),
+ ))
+
+ def test__iter_equal_wrong_value(self):
+ """Test when _iter_equal() is False due to value."""
+ self.assertFalse(venvlib._iter_equal(
+ iter(['foo', 'baz']),
+ iter(['foo', 'bar']),
+ ))
+
+
+class VenvTmpDirTestCase(testcases.TmpdirTestCase):
+
+ """TestCase for venvlib that uses a temp directory."""
+
+ def test__makedirs_exist_ok(self):
+ """Test _makedirs_exist_ok() ignores existing dir."""
+ os.mkdir('foo')
+ try:
+ venvlib._makedirs_exist_ok('foo')
+ except OSError:
+ self.fail('OSError raised')
+
+ def test__reqs_up_to_date(self):
+ """Test _reqs_up_to_date()."""
+ venv = venvlib.Venv('.', 'reqs.txt')
+ os.makedirs('lib/python2.7/site-packages')
+ with open('reqs.txt', 'w') as f:
+ f.write('mir.qualia==1.2.3\n')
+ with open('.installed.txt', 'w') as f:
+ f.write('mir.qualia==1.2.3\n')
+ self.assertTrue(venv._reqs_up_to_date())
+
+ def test__reqs_up_to_date_missing(self):
+ """Test _reqs_up_to_date() with missing installed reqs."""
+ venv = venvlib.Venv('.', 'reqs.txt')
+ self.assertFalse(venv._reqs_up_to_date())
+
+ def test__reqs_up_to_date_different(self):
+ """Test _reqs_up_to_date() with different installed reqs."""
+ venv = venvlib.Venv('.', 'reqs.txt')
+ os.makedirs('lib/python2.7/site-packages')
+ with open('reqs.txt', 'w') as f:
+ f.write('mir.qualia==1.2.4\n')
+ with open('.installed.txt', 'w') as f:
+ f.write('mir.qualia==1.2.3\n')
+ self.assertFalse(venv._reqs_up_to_date())
+
+ def test__venv_initialized(self):
+ """Test _venv_initialized()."""
+ venv = venvlib.Venv('.', 'reqs.txt')
+ os.makedirs('bin')
+ with open('bin/python', 'wb') as f:
+ f.write(b'python binary')
+ self.assertTrue(venv._venv_initialized())
+
+ def test__venv_initialized_missing(self):
+ """Test _venv_initialized() with missing bin/python."""
+ venv = venvlib.Venv('.', 'reqs.txt')
+ self.assertFalse(venv._venv_initialized())
diff --git a/cros_venv/testcases.py b/cros_venv/testcases.py
new file mode 100644
index 0000000..c64c054
--- /dev/null
+++ b/cros_venv/testcases.py
@@ -0,0 +1,27 @@
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""TestCase classes"""
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import shutil
+import tempfile
+import os
+import unittest
+
+
+class TmpdirTestCase(unittest.TestCase):
+ """TestCase subclass providing a tmpdir fixture."""
+
+ def setUp(self):
+ self.tmpdir = tempfile.mkdtemp()
+ self._old_cwd = os.getcwd()
+ os.chdir(self.tmpdir)
+
+ def tearDown(self):
+ os.chdir(self._old_cwd)
+ shutil.rmtree(self.tmpdir)
diff --git a/cros_venv/venvlib.py b/cros_venv/venvlib.py
new file mode 100644
index 0000000..ff2220f
--- /dev/null
+++ b/cros_venv/venvlib.py
@@ -0,0 +1,115 @@
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Virtualenv management"""
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import itertools
+import os
+import shutil
+import subprocess
+
+import cros_venv
+from cros_venv import flock
+
+_PACKAGE_DIR = os.path.join(cros_venv.WORKTREE_DIR, 'pip_packages')
+
+# BASE_DEPENDENCIES are pip requirements automatically included in every
+# virtualenv so we have control over which versions of bootstrap
+# packages are installed.
+_BASE_DEPENDENCIES = ('setuptools==28.2.0', 'pip==8.1.2')
+
+
+class Venv(object):
+ """Wraps all operations on a virtualenv directory."""
+
+ def __init__(self, venv_dir, reqs_file):
+ self._venv_dir = venv_dir
+ self._reqs_file = reqs_file
+
+ @property
+ def _venv_python(self):
+ return os.path.join(self._venv_dir, 'bin', 'python')
+
+ @property
+ def _lock_file(self):
+ return os.path.join(self._venv_dir, '.create_venv.lock')
+
+ @property
+ def _installed_reqs_file(self):
+ return os.path.join(self._venv_dir, '.installed.txt')
+
+ @property
+ def _site_packages(self):
+ return os.path.join(self._venv_dir, 'lib', 'python2.7', 'site-packages')
+
+ def ensure(self):
+ """Create or update virtualenv."""
+ _makedirs_exist_ok(self._venv_dir)
+ with flock.FileLock(self._lock_file):
+ if not self._venv_initialized():
+ self._init_venv()
+ if not self._reqs_up_to_date():
+ self._install_reqs()
+
+ def _venv_initialized(self):
+ """Check if virtualenv is initialized."""
+ return os.path.exists(self._venv_python)
+
+ def _init_venv(self):
+ """Initialize virtualenv."""
+ subprocess.check_call([
+ 'virtualenv', self._venv_dir,
+ '--extra-search-dir', _PACKAGE_DIR,
+ # TODO(ayatane): Ubuntu Precise ships with virtualenv 1.7,
+ # which requires specifying --setuptools, else distribute is
+ # used (distribute is deprecated). virtualenv after 1.10
+ # uses setuptools by default. virtualenv >1.10 accepts the
+ # --setuptools option but does not document it. Once we no
+ # longer have any hosts on virtualenv 1.7, the --setuptools
+ # option can be removed.
+ '--setuptools'])
+
+ def _reqs_up_to_date(self):
+ """Return whether the virtualenv reqs file is up to date."""
+ if not os.path.exists(self._installed_reqs_file):
+ return False
+ with open(self._installed_reqs_file) as installed, \
+ open(self._reqs_file) as specified:
+ return _iter_equal(installed, specified)
+
+ def _install_reqs(self):
+ """Install indicated packages into virtualenv.
+
+ The corresponding requirements file is also copied after
+ installation.
+ """
+ subprocess.check_call(
+ [self._venv_python, '-m', 'pip', 'install',
+ '--no-index', '-f', 'file://' + _PACKAGE_DIR]
+ + list(_BASE_DEPENDENCIES)
+ + ['-r', self._reqs_file]
+ )
+ shutil.copyfile(self._reqs_file, self._installed_reqs_file)
+
+
+def _iter_equal(first, second):
+ """Return whether two iterables are equal.
+
+ If one iterable is shorter, it will be filled with None and compared
+ with the other.
+ """
+ return all(x == y for x, y in itertools.izip_longest(first, second))
+
+
+def _makedirs_exist_ok(path):
+ """Make directories recursively, ignoring if directory already exists."""
+ try:
+ os.makedirs(path)
+ except OSError:
+ if not os.path.isdir(path):
+ raise
diff --git a/expand_reqs.py b/expand_reqs.py
deleted file mode 100755
index a38043f..0000000
--- a/expand_reqs.py
+++ /dev/null
@@ -1,109 +0,0 @@
-#!/usr/bin/env python
-# Copyright 2016 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-"""Expand recursive requirements in pip requirements.txt files.
-
-$ ./expand_reqs.py <requirements.txt
-
-Run tests using: python -m unittest expand_reqs
-"""
-
-import os
-import shutil
-import sys
-import tempfile
-import unittest
-
-
-def main():
- for line in expand_reqs(sys.stdin):
- print line
-
-
-def expand_reqs(infile):
- """Yield each line in infile, recursing into included requirements files."""
- for line in infile:
- line = line.strip()
- yield line
- requirements = _get_requirements_file(line)
- if requirements and os.path.exists(requirements):
- with open(requirements) as subfile:
- for subline in expand_reqs(subfile):
- yield subline
- yield '# END %s' % line
-
-
-def _get_requirements_file(line):
- """Return the requirements file specified by the input line.
-
- A requirements line looks like: -r requirements.txt
-
- Args:
- line: String input
-
- Returns:
- Requirements file path as string if present, else an empty string.
- """
- parts = line.split()
- if len(parts) == 2 and parts[0] == '-r':
- return parts[1]
- else:
- return ''
-
-
-class TmpdirTestCase(unittest.TestCase):
- """TestCase subclass providing a tmpdir fixture."""
-
- def setUp(self):
- self.tmpdir = tempfile.mkdtemp()
- self._old_cwd = os.getcwd()
- os.chdir(self.tmpdir)
-
- def tearDown(self):
- os.chdir(self._old_cwd)
- shutil.rmtree(self.tmpdir)
-
-
-class ExpandReqsTestCase(TmpdirTestCase):
- """Tests for expand_reqs()."""
-
- def setUp(self):
- super(ExpandReqsTestCase, self).setUp()
- self.flat = 'flat.txt'
- self.recursive = 'recursive.txt'
- with open(self.flat, 'w') as f:
- f.write('foo\nbar')
- with open(self.recursive, 'w') as f:
- f.write('spam\neggs\n-r flat.txt')
-
- def test_no_recurse(self):
- with open(self.flat) as f:
- self.assertEqual(
- list(expand_reqs(f)),
- ['foo', 'bar'])
-
- def test_recurse(self):
- with open(self.recursive) as f:
- self.assertEqual(
- list(expand_reqs(f)),
- ['spam', 'eggs', '-r flat.txt',
- 'foo', 'bar', '# END -r flat.txt'])
-
-
-class GetRequirementsFileTestCase(unittest.TestCase):
- """Tests for _get_requirements_file()."""
-
- def test_no_match(self):
- self.assertEqual(_get_requirements_file('foo'), '')
-
- def test_match(self):
- self.assertEqual(_get_requirements_file('-r foo.txt'), 'foo.txt')
-
- def test_match_with_whitespace(self):
- self.assertEqual(_get_requirements_file(' -r foo.txt \n'), 'foo.txt')
-
-
-if __name__ == '__main__':
- main()
