build: use electron-frameworks sccache (#14171)

* build: update-external-binaries fetches sccache

* build: add util.add_exec_bit in scripts/

* build: use util.add_exec_bit in create-dist

* build: use util.add_exec_bit in update-external-binaries

this is needed to work around a bug in python's zipfile module that doesn't preserve the exec bit

https://bugs.python.org/issue18262

* fix: linting errors

* build: vsts, circleci use patched sccache

* build: always look for the x64 sccache

as it's the only arch we have it on

* fix: windows-specific errors in updaste-external-binaries

* fix: tyop

* fix: set SCCACHE_BUCKET, SCCACHE_TWO_TIER on circleci

* fix: syntax error in circleci yaml

* fix: keep churning

* chore: add tracer to file downloader

* docs: add sccache instructions for GN builds

* build: pull down the darwin sccache on mas builds

* build: use gn sync verbosely on circleci and vsts

* docs: copyediting

* build: remove unnecessary cache-dir arg

* docs: fix shell quoting in gn build instructions

* fix: invoke gclient without -verbose in circleci

* refactor: remove debug tracer

* fix: invoke gclient without -verbose in appveyor

* fix: invoke gclient without -verbose in vsts

* fix: pull add_exec_bit from correct source

* fix: remove 'SCCACHE_TWO_TIER' from CI scripts

* refactor: remove SCCACHE_BUCKET from ci scripts

this environment variable will be set via the CI UI instead

* refactor: clarify log message

* fix: set SCCACHE_PATH correctly for Windows CI

.circleci/config.yml

@@ -169,7 +169,6 @@ gn-build-steps: &gn-build-steps
           gclient config \
             --name "src/electron" \
             --unmanaged \
-            --cache-dir "${GIT_CACHE_PATH}" \
             $GCLIENT_EXTRA_ARGS \
             "$CIRCLE_REPOSITORY_URL"
 
@@ -182,9 +181,9 @@ gn-build-steps: &gn-build-steps
         name: GN gen
         command: |
           cd src
-          SCCACHE_PATH="$PWD/libchromiumcontent/tools/sccache/aad2120/linux/sccache"
+          SCCACHE_PATH="$PWD/electron/external_binaries/sccache"
-          echo 'export SCCACHE_WRAPPER="'"$SCCACHE_PATH"'"' >> $BASH_ENV
           echo 'export CHROMIUM_BUILDTOOLS_PATH="'"$PWD"'/buildtools"' >> $BASH_ENV
+          echo 'export SCCACHE_WRAPPER="'"$SCCACHE_PATH"'"'            >> $BASH_ENV
           source $BASH_ENV
           gn gen out/Default --args='import("'$GN_CONFIG'") cc_wrapper="'"$SCCACHE_PATH"'"'" $GN_EXTRA_ARGS"
     - run:

appveyor-gn.yml

@@ -21,12 +21,11 @@ build_script:
   - md src
   - ps: Move-Item $env:APPVEYOR_BUILD_FOLDER -Destination src\electron
   - ps: $env:CHROMIUM_BUILDTOOLS_PATH="$pwd\src\buildtools"
-  - ps: $env:SCCACHE_PATH="$pwd\src\libchromiumcontent\tools\sccache\aad2120\windows\sccache.exe"
+  - ps: $env:SCCACHE_PATH="$pwd\src\electron\external_binaries\sccache\sccache.exe"
   - >-
       gclient config
       --name "src\electron"
       --unmanaged
-      --cache-dir "${GIT_CACHE_PATH}"
       "https://github.com/electron/electron"
   - gclient sync --with_branch_heads --with_tags
   - cd src

docs/development/build-instructions-gn.md

@@ -5,7 +5,7 @@ build.
 
 > **NOTE**: The GN build system is in _experimental_ status.
 
-## Prerequisites
+## Platform prerequisites
 
 Check the build prerequisites for your platform before proceeding
 
@@ -13,7 +13,7 @@ Check the build prerequisites for your platform before proceeding
   * [Linux](build-instructions-linux.md#prerequisites)
   * [Windows](build-instructions-windows.md#prerequisites)
 
-## Install `depot_tools`
+## GN prerequisites
 
 You'll need to install [`depot_tools`][depot-tools], the toolset
 used for fetching Chromium and its dependencies.
@@ -27,20 +27,33 @@ try to download a Google-internal version that only Googlers have access to).
 
 [depot-tools]: http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up
 
-## Getting the Code
+## Cached builds (optional step)
 
-### Using a Git cache (optional step)
+### GIT_CACHE_PATH
 
-`gclient` fetches about 16G worth of repository data. If you plan on building
+If you plan on building Electron more than once, adding a git cache will
-more than once, consider using its cache feature to make future calls faster:
+speed up subsequent calls to `gclient`. To do this, set a `GIT_CACHE_PATH`
+environment variable:
 
 ```sh
-$ export GIT_CACHE_PATH="$HOME/.git_cache"
+$ export GIT_CACHE_PATH="${HOME}/.git_cache"
-$ mkdir -p "$GIT_CACHE_PATH"
+$ mkdir -p "${GIT_CACHE_PATH}"
-# This will take about 16G.
+# This will use about 16G.
 ```
 
-### Getting the code with gclient
+### sccache
+
+Thousands of files must be compiled to build Chromium and Electron.
+You can avoid much of the wait by reusing Electron CI's build output via
+[sccache](https://github.com/mozilla/sccache). This requires some
+optional steps (listed below) and these two environment variables:
+
+```sh
+export SCCACHE_BUCKET="electronjs-sccache"
+export SCCACHE_TWO_TIER=true
+```
+
+## Getting the code
 
 ```sh
 $ mkdir electron-gn && cd electron-gn
@@ -57,7 +70,9 @@ $ gclient sync --with_branch_heads --with_tags
 ```sh
 $ cd src
 $ export CHROMIUM_BUILDTOOLS_PATH=`pwd`/buildtools
-$ gn gen out/Default --args='import("//electron/build/args/debug.gn")'
+# this next line is needed only if building with sccache
+$ export GN_EXTRA_ARGS="${GN_EXTRA_ARGS} cc_wrapper=\"${PWD}/electron/external_binaries/sccache\""
+$ gn gen out/Default --args="import(\"//electron/build/args/debug.gn\") $GN_EXTRA_ARGS"
 ```
 
 This will generate a build directory `out/Default` under `src/` with
@@ -73,14 +88,14 @@ out/Default --list`.
 Electron:**
 
 ```sh
-$ gn gen out/Default --args='import("//electron/build/args/debug.gn")'
+$ gn gen out/Default --args='import("//electron/build/args/debug.gn") $GN_EXTRA_ARGS'
 ```
 
 **For generating Release (aka "non-component" or "static") build config of
 Electron:**
 
 ```sh
-$ gn gen out/Default --args='import("//electron/build/args/release.gn")'
+$ gn gen out/Default --args="import(\"//electron/build/args/release.gn\") $GN_EXTRA_ARGS"
 ```
 
 **To build, run `ninja` with the `electron:electron_app` target:**

script/create-dist.py

@@ -11,10 +11,13 @@ import stat
 if sys.platform == "win32":
   import _winreg
 
-from lib.config import BASE_URL, PLATFORM, enable_verbose_mode, \
+from lib.config import BASE_URL, PLATFORM, build_env, \
-                       get_target_arch, get_zip_name, build_env
+                       enable_verbose_mode, get_target_arch, get_zip_name
-from lib.util import scoped_cwd, rm_rf, get_electron_version, make_zip, \
+
-                     execute, electron_gyp, electron_features, parse_version
+from lib.util import add_exec_bit, electron_features, electron_gyp, \
+                     execute, get_electron_version, make_zip, \
+                     parse_version, rm_rf, scoped_cwd
+
 from lib.env_util import get_vs_location
 
 
@@ -144,7 +147,7 @@ def copy_chrome_binary(binary):
 
   # Copy file and keep the executable bit.
   shutil.copyfile(src, dest)
-  os.chmod(dest, os.stat(dest).st_mode | stat.S_IEXEC)
+  add_exec_bit(dest)
 
 def copy_vcruntime_binaries():
   arch = get_target_arch()
@@ -299,7 +302,7 @@ def create_chrome_binary_zip(binary, version):
     dest = os.path.join(DIST_DIR, binary)
     # Copy file and keep the executable bit.
     shutil.copyfile(src, dest)
-    os.chmod(dest, os.stat(dest).st_mode | stat.S_IEXEC)
+    add_exec_bit(dest)
 
     dist_name = get_zip_name(binary, version)
     zip_file = os.path.join(SOURCE_ROOT, 'dist', dist_name)

script/lib/util.py

@@ -8,6 +8,7 @@ import platform
 import re
 import shutil
 import ssl
+import stat
 import subprocess
 import sys
 import tarfile
@@ -83,6 +84,7 @@ def download(text, url, path):
     if hasattr(ssl, '_create_unverified_context'):
       ssl._create_default_https_context = ssl._create_unverified_context
 
+    print "Downloading %s to %s" % (url, path)
     web_file = urllib2.urlopen(url)
     file_size = int(web_file.info().getheaders("Content-Length")[0])
     downloaded_size = 0
@@ -295,6 +297,9 @@ def update_node_modules(dirname, env=None):
     else:
       execute_stdout(args, env)
 
+def add_exec_bit(filename):
+  os.chmod(filename, os.stat(filename).st_mode | stat.S_IEXEC)
+
 def clean_parse_version(v):
   return parse_version(v.split("-")[0])        
 

script/update-external-binaries.py

@@ -4,11 +4,11 @@ import errno
 import sys
 import os
 
-from lib.config import get_target_arch
+from lib.config import PLATFORM, get_target_arch
-from lib.util import safe_mkdir, rm_rf, extract_zip, tempdir, download
+from lib.util import add_exec_bit, download, extract_zip, rm_rf, \
+                     safe_mkdir, tempdir
 
-
+VERSION = 'v1.4.0'
-VERSION = 'v1.3.0'
 SOURCE_ROOT = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
 FRAMEWORKS_URL = 'http://github.com/electron/electron-frameworks/releases' \
                  '/download/' + VERSION
@@ -31,6 +31,13 @@ def main():
   elif sys.platform in ['cygwin', 'win32']:
     download_and_unzip('directxsdk-' + get_target_arch())
 
+  # get sccache & set exec bit. https://bugs.python.org/issue15795
+  download_and_unzip('sccache-{0}-x64'.format(PLATFORM))
+  appname = 'sccache'
+  if sys.platform == 'win32':
+    appname += '.exe'
+  add_exec_bit(os.path.join('external_binaries', appname))
+
   with open(version_file, 'w') as f:
     f.write(VERSION)
 

vsts-gn.yml

@@ -16,7 +16,6 @@ phases:
       gclient config \
         --name "src/electron" \
         --unmanaged \
-        --cache-dir "${GIT_CACHE_PATH}" \
         "https://github.com/electron/electron"
       mkdir src
       git clone https://github.com/electron/electron src/electron
@@ -32,7 +31,7 @@ phases:
   - bash: |
       cd src
       export CHROMIUM_BUILDTOOLS_PATH=`pwd`/buildtools
-      export SCCACHE_WRAPPER="`pwd`/libchromiumcontent/tools/sccache/aad2120/mac/sccache"
+      export SCCACHE_WRAPPER="`pwd`/electron/external_binaries/sccache"
       export SCCACHE_HELPER="`pwd`/libchromiumcontent/script/sccache"
       "$SCCACHE_HELPER" --start-server --azure_container "$(SCCACHE_AZURE_BLOB_CONTAINER)" --azure_connection "$(SCCACHE_AZURE_CONNECTION_STRING)"
       echo "##vso[task.setvariable variable=SCCACHE_WRAPPER]$SCCACHE_WRAPPER"