Windows binary tool bootstrap

This directory has the ‘magic’ for the depot_tools windows binary update mechanisms.

A previous Python may actually be in use when it is run, preventing us from replacing it outright without breaking running code. To ommodate this, and Python cleanup, we handle Python in two stages:

1. Use CIPD to install both Git and Python at once.
2. Use “win_tools.py” as a post-processor to install generated files and fix-ups.

Software bootstrapped

• Python 2 and 3 (https://www.python.org/)
• Git for Windows (https://git-for-windows.github.io/)

Mechanism

Any time a user runs gclient on windows, it will invoke the depot_tools autoupdate script depot_tools.bat. This, in turn, will run win_tools.bat, which does the bulk of the work.

win_tools.bat will successively look to see if the local version of the binary package is present, and if so, if it's the expected version. If either of those cases is not true, it will download and unpack the respective binary.

Installation of Git and Python is done by the win_tools.bat script, which uses CIPD (via the cipd bootstrap) to acquire and install each package into the root of the depot_tools repository. Afterwards, the win_tools.py Python script is invoked to install stubs, wrappers, and support scripts into depot_tools for end-users.

Manifest

The Git and Python versions are specified in manifest.txt.

There is an associated file, manifest_bleeding_edge.txt, that can be used to canary new versions on select bots. Any bots with a .bleeding_edge file in their depot_tools root will automatically use the bleeding edge manifest. This allows opt-in systems to test against new versions of Python or Git. Once those versions have been verified correct, manifest.txt can be updated to the same specification, which will cause the remainder of systems to update.

Bundles

Git and Python bundle construction is documented in infra packaging.

Note that in order for the update to take effect, gclient currently needs to run twice. The first time it will update the depot_tools repo, and the second time it will see the new git version and update to it. This is a bug that should be fixed, in case you're reading this and this paragraph infuriates you more than the rest of this README.

Testing

After any modification to this script set, a test sequence should be run on a Windows bot.

The post-processing will regenerate “python.bat” and “python3.bat” to point to the current Python instance. Any previous Python installations will stick around, but new invocations will use the new instance. Old installations will die off either due to processes terminating or systems restarting. When this happens, they will be cleaned up by the post-processing script.

Testing

For each of the following test scenarios, run these commands and verify that they are working:

:: Assert that `gclient` invocation will update (and do the update).
gclient version

:: Assert that Python fundamentally works.
python -c "import Queue; print dir(Queue)"

:: Assert that Python 3 fundamentally works.
python3 -c "import queue; print(dir(queue))"

:: Assert that Python scripts work from `cmd.exe`.
git map-branches

:: Assert that `git bash` works.
git bash

## (Within `git bash`) assert that Python fundamentally works.
python -c "import Queue; print dir(Queue)"
## (Within `git bash`) assert that Python 3 fundamentally works.
python3 -c "import queue; print(dir(queue))"
## (Within `git bash`) assert that Python scripts work.
git map-branches

Run this sequence through the following upgrade/downgrade procedures:

• Cold default installation.
  • Clean depot_tools via: git clean -x -f -d .
  • Run through test steps.
  • Test upgrade to bleeding edge (if it differs).
    • Run python.bat in another shell, keep it open
    • Run python3.bat in another shell, keep it open
    • Add .bleeding_edge to depot_tools root.
    • Run through test steps.
    • In the old python.bat shell, run import Queue, confirm that it works.
    • In the old python3.bat shell, run import queue, confirm that it works.
    • Close the Python shell, run gclient version, ensure that old directory is cleaned.
• Cold bleeding edge installation.
  • Clean depot_tools via: git clean -x -f -d .
  • Add .bleeding_edge to depot_tools root.
  • Run through test steps.
  • Test downgrade to default (if it differs).
    • Run python.bat in another shell, keep it open
    • Run python3.bat in another shell, keep it open
    • Delete .bleeding_edge from depot_tools root.
    • Run through test steps.
    • In the old python.bat shell, run import Queue, confirm that it works.
    • In the old python3.bat shell, run import queue, confirm that it works.
    • Close the Python shell, run gclient version, ensure that old directory is cleaned.
• Warm bleeding edge upgrade.
  • Clean depot_tools via: git clean -x -f -d .
  • Run gclient version to load defaults.
  • Run python.bat in another shell, keep it open
  • Run python3.bat in another shell, keep it open
  • Add .bleeding_edge to depot_tools root.
  • Run through test steps.
  • In the old python.bat shell, run import Queue, confirm that it works.
  • In the old python3.bat shell, run import queue, confirm that it works.
  • Close the Python shell, run gclient version, ensure that old directory is cleaned.
• Upgradable and Revertible.
  • Checkout current HEAD.
  • Run gclient version to load HEAD toolchain (initial).
  • Apply the candidate patch.
  • Run through test steps (upgrade).
  • Checkout current HEAD again.
  • Run gclient version to load HEAD toolchain (revert).
  • Run through test steps.

This will take some time, but will ensure that all affected bots and users should not encounter any problems due to the change. As systems and users are migrated off of this implicit bootstrap, the testing procedure will become less critical.