mac-arm64: Provide an arm64-native ninja-mac for builds in ¼ of the time

ninja-mac is expanded to become a universal (“fat”) executable. The
existing x86_64 slice was taken from the x86_64 ninja-mac as it
previously existed in this location, and a new arm64 slice was added.

This is an aid to native development on arm64. In macOS, subprocesses
inherit their parent’s “binpref,” meaning that children of a translated
process will, by default, also be translated, if there’s a choice
between native and translated. Where ninja-mac is x86_64-only and runs
translated on arm64, all tools it invokes will also run translated, even
if native versions are available. This has an impact on any part of the
toolchain for which universal versions are available: in Chrome’s case,
tools such as ld64 are affected. In Crashpad’s build, which uses the
compiler provided by Xcode, it even affects the compiler, leading to
significant differences in build time on arm64.

mark@arm-and-hammer zsh% file $(which ninja-mac)
…/ninja-mac: Mach-O 64-bit executable x86_64
mark@arm-and-hammer zsh% time ninja -C out/release
ninja: Entering directory `out/release'
[1/1] Regenerating ninja files
[511/511] LINK ./crashpad_minidump_test
ninja -C out/release  1384.03s user 111.05s system 393% cpu 6:19.53 total

mark@arm-and-hammer zsh% file $(which ninja-mac)
…/ninja-mac: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
…/ninja-mac (for architecture x86_64):	Mach-O 64-bit executable x86_64
…/ninja-mac (for architecture arm64):	Mach-O 64-bit executable arm64
mark@arm-and-hammer zsh% time ninja -C out/release
ninja: Entering directory `out/release'
[1/1] Regenerating ninja files
[511/511] LINK ./crashpad_minidump_test
ninja -C out/release  606.80s user 34.71s system 701% cpu 1:31.41 total

This differs from the previous attempt, e64cd3aeff71
(https://chromium-review.googlesource.com/c/2288417) in that the arm64
version was built with Xcode 12.2b1 (12B5018i). The linker in this
version of Xcode produces linker-signed output for arm64 by default. As
of macOS 11.0db6 (20A5364e), all arm64 code must be signed. More
information at
https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11-universal-apps-beta-release-notes#Code-Signing.
In the previous attempt, an older Xcode was used and the output was not
signed at all. Because it did not run on systems upgraded to 11.0db6,
this version of ninja-mac was backed out in a070dd18d79d
(https://chromium-review.googlesource.com/c/2393775).

To create this ninja-mac:

% git clone https://github.com/ninja-build/ninja.git
% cd ninja
% git checkout v1.8.2
% mkdir ../build
% cd ../build
% LDFLAGS=-Wl,-s ../ninja/configure.py --bootstrap
% lipo -create -output ninja-mac $(which ninja-mac) ninja

An explanation of the LDFLAGS: e64cd3aeff71 used “strip” on the output,
matching the stripped x86_64 slice, but “strip” is not currently
compatible with linker-signed executables. This is documented at
https://developer.apple.com/documentation/xcode-release-notes/xcode-12_2-beta-release-notes#Apple-Clang-Compiler.
I was unable to use the recommended workaround in this case. Rather than
leave the output unstripped, I asked ld64 to strip symbols using its -s
option, which it believes is obsolete, but in fact provides the intended
behavior.

The x86_64 slice is unsigned, as it was previously. The new arm64 slice
is linker-signed with an ad-hoc signature. The x86_64 slice was built
with the macOS 10.12 SDK and runs on Mac OS X 10.6 or later. The arm64
slice was built with and has a minimum runtime requirement of macOS
11.0. In practice, the difference doesn’t matter, as the 11.0 SDK is
required to build for arm64, and no macOS version earlier than 11.0 will
run on arm64.

Bug: chromium:1103315
Change-Id: I9f201d0d6f476e187acc0008681c3fd2c1a72aa0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2427051
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Mark Mentovai <mark@chromium.org>
1 file changed
tree: bd01a1bf926cc97bffcbe938b57bd2b22c7fefcf
  1. bootstrap/
  2. fetch_configs/
  3. git-templates/
  4. infra/
  5. man/
  6. python-bin/
  7. recipes/
  8. testing_support/
  9. tests/
  10. third_party/
  11. win_toolchain/
  12. zsh-goodies/
  13. .cipd_impl.ps1
  14. .gitattributes
  15. .gitignore
  16. .style.yapf
  17. .vpython
  18. .vpython3
  19. auth.py
  20. autoninja
  21. autoninja.bat
  22. autoninja.py
  23. bb
  24. bb.bat
  25. bootstrap_python3
  26. breakpad.py
  27. cipd
  28. cipd.bat
  29. cipd_bin_setup.bat
  30. cipd_bin_setup.sh
  31. cipd_client_version
  32. cipd_client_version.digests
  33. cipd_manifest.txt
  34. cipd_manifest.versions
  35. cit
  36. cit.bat
  37. cit.py
  38. clang-format
  39. clang-format.bat
  40. clang_format.py
  41. clang_format_merge_driver
  42. clang_format_merge_driver.bat
  43. clang_format_merge_driver.py
  44. codereview.settings
  45. compile_single_file
  46. compile_single_file.bat
  47. compile_single_file.py
  48. cpplint.bat
  49. cpplint.py
  50. cpplint_chromium.py
  51. cros
  52. CROS_OWNERS
  53. crosjobs
  54. detect_host_arch.py
  55. dirmd
  56. dirmd.bat
  57. download_from_google_storage
  58. download_from_google_storage.bat
  59. download_from_google_storage.py
  60. ensure_bootstrap
  61. fetch
  62. fetch.bat
  63. fetch.py
  64. fix_encoding.py
  65. gclient
  66. gclient-new-workdir.py
  67. gclient.bat
  68. gclient.py
  69. gclient_completion.sh
  70. gclient_eval.py
  71. gclient_paths.py
  72. gclient_scm.py
  73. gclient_utils.py
  74. gerrit_client.py
  75. gerrit_util.py
  76. git-cache
  77. git-cl
  78. git-crrev-parse
  79. git-drover
  80. git-find-releases
  81. git-footers
  82. git-freeze
  83. git-gs
  84. git-hyper-blame
  85. git-map
  86. git-map-branches
  87. git-mark-merge-base
  88. git-nav-downstream
  89. git-nav-upstream
  90. git-new-branch
  91. git-number
  92. git-rebase-update
  93. git-rename-branch
  94. git-reparent-branch
  95. git-retry
  96. git-runhooks
  97. git-squash-branch
  98. git-thaw
  99. git-upstream-diff
  100. git_cache.py
  101. git_cl.py
  102. git_cl_completion.sh
  103. git_common.py
  104. git_dates.py
  105. git_drover.py
  106. git_find_releases.py
  107. git_footers.py
  108. git_freezer.py
  109. git_hyper_blame.py
  110. git_map.py
  111. git_map_branches.py
  112. git_mark_merge_base.py
  113. git_nav_downstream.py
  114. git_new_branch.py
  115. git_number.py
  116. git_rebase_update.py
  117. git_rename_branch.py
  118. git_reparent_branch.py
  119. git_retry.py
  120. git_squash_branch.py
  121. git_upstream_diff.py
  122. gn
  123. gn.bat
  124. gn.py
  125. goma_auth
  126. goma_auth.bat
  127. goma_ctl
  128. goma_ctl.bat
  129. GOMA_OWNERS
  130. gsutil.py
  131. gsutil.py.bat
  132. gsutil.vpython
  133. led
  134. led.bat
  135. LICENSE
  136. lockfile.py
  137. luci-auth
  138. luci-auth.bat
  139. LUCI_OWNERS
  140. lucicfg
  141. lucicfg.bat
  142. mac_toolchain
  143. metrics.py
  144. metrics.README.md
  145. metrics_utils.py
  146. my_activity.py
  147. ninja
  148. ninja-linux32
  149. ninja-linux64
  150. ninja-mac
  151. ninja.exe
  152. ninjalog.README.md
  153. ninjalog_uploader.py
  154. ninjalog_uploader_wrapper.py
  155. OWNERS
  156. owners.py
  157. owners_finder.py
  158. post_build_ninja_summary.py
  159. PRESUBMIT.py
  160. presubmit_canned_checks.py
  161. presubmit_support.py
  162. profile.xml
  163. prpc
  164. prpc.bat
  165. pylint
  166. pylint-1.5
  167. pylint-1.6
  168. pylint-1.7
  169. pylint-1.8
  170. pylint-1.9
  171. pylint.bat
  172. pylint_main.py
  173. pylintrc
  174. python_runner.sh
  175. rdb
  176. rdb.bat
  177. rdb_wrapper.py
  178. README.gclient.md
  179. README.git-cl.md
  180. README.md
  181. repo
  182. roll-dep
  183. roll-dep.bat
  184. roll_dep.py
  185. scm.py
  186. setup_color.py
  187. split_cl.py
  188. subcommand.py
  189. subprocess2.py
  190. update_depot_tools
  191. update_depot_tools.bat
  192. update_depot_tools_toggle.py
  193. upload_metrics.py
  194. upload_to_google_storage.py
  195. vpython
  196. vpython.bat
  197. vpython3
  198. vpython3.bat
  199. WATCHLISTS
  200. watchlists.py
  201. weekly
  202. win32imports.py
  203. wtf
  204. yapf
  205. yapf.bat
README.md

depot_tools

Tools for working with Chromium development. It requires python 2.7 or 3.8 for python 3 support.

Tools

The most important tools are:

  • fetch: A gclient wrapper to checkout a project. Use fetch --help for more details.
  • gclient: A meta-checkout tool. Think repo or git submodules, except that it support OS-specific rules, e.g. do not checkout Windows only dependencies when checking out for Android. Use gclient help for more details and README.gclient.md.
  • git cl: A code review tool to interact with Rietveld or Gerrit. Use git cl help for more details and README.git-cl.md.
  • roll-dep: A gclient dependency management tool to submit a dep roll, updating a dependency to a newer revision.

There are a lot of git utilities included.

Updating

depot_tools updates itself automatically when running gclient tool. To disable auto update, set the environment variable DEPOT_TOOLS_UPDATE=0.

To update package manually, run update_depot_tools.bat on Windows, or ./update_depot_tools on Linux or Mac.

On Windows only, running gclient will install git and python.

Contributing

To contribute change for review:

git new-branch <somename>
# Hack
git add .
git commit -a -m "Fixes goat teleporting"
# find reviewers
git cl owners
git log -- <yourfiles>

# Request a review.
git cl upload -r reviewer1@chromium.org,reviewer2@chromium.org --send-mail

# Edit change description if needed.
git cl desc

# If change is approved, flag it to be committed.
git cl set-commit

# If change needs more work.
git rebase-update
...
git cl upload -t "Fixes goat teleporter destination to be Australia"

See also open bugs, open reviews, forum or report problems.

cpplint.py

Until 2018, our cpplint.py was a copy of the upstream version at https://github.com/google/styleguide/tree/gh-pages/cpplint. Unfortunately, that repository is not maintained any more. If you want to update cpplint.py in depot_tools, just upload a patch to do so. We will figure out a long-term strategy via issue https://crbug.com/916550.

Note that the cpplint.py here is also used by the Tricium analyzer, so if the cpplint.py here changes, we should also update the copy used there.