docs: update test automation doc (#31506)

* Update WebdriverIO documentation * Update docs/tutorial/using-selenium-and-webdriver.md Co-authored-by: Jeremy Rose <nornagon@nornagon.net> * Update docs/tutorial/using-selenium-and-webdriver.md Co-authored-by: Jeremy Rose <nornagon@nornagon.net> * docs: update automated testing docs * lint * update * Update docs/tutorial/automated-testing.md Co-authored-by: Christian Bromann <github@christian-bromann.com> * fixes Co-authored-by: Christian Bromann <github@christian-bromann.com> Co-authored-by: Jeremy Rose <nornagon@nornagon.net>
2021-10-27 19:17:43 -07:00 · 2021-10-27 19:17:43 -07:00 · ec34c6c6e7
commit ec34c6c6e7
parent 3bf42593ba
5 changed files with 267 additions and 356 deletions
--- a/docs/README.md
+++ b/docs/README.md
@ -59,10 +59,9 @@ an issue:
 * [Testing and Debugging](tutorial/application-debugging.md)
  * [Debugging the Main Process](tutorial/debugging-main-process.md)
  * [Debugging with Visual Studio Code](tutorial/debugging-vscode.md)
  * [Using Selenium and WebDriver](tutorial/using-selenium-and-webdriver.md)
  * [Testing on Headless CI Systems (Travis, Jenkins)](tutorial/testing-on-headless-ci.md)
  * [DevTools Extension](tutorial/devtools-extension.md)
-  * [Automated Testing with a Custom Driver](tutorial/automated-testing-with-a-custom-driver.md)
+  * [Automated Testing](tutorial/automated-testing.md)
  * [REPL](tutorial/repl.md)
 * [Distribution](tutorial/application-distribution.md)
  * [Supported Platforms](tutorial/support.md#supported-platforms)
--- a/docs/tutorial/accessibility.md
+++ b/docs/tutorial/accessibility.md
@ -1,48 +1,7 @@
 # Accessibility
 Making accessible applications is important and we're happy to provide
 functionality to [Devtron][devtron] and [Spectron][spectron] that gives
 developers the opportunity to make their apps better for everyone.
 ---
 Accessibility concerns in Electron applications are similar to those of
-websites because they're both ultimately HTML. With Electron apps, however,
+websites because they're both ultimately HTML.
 you can't use the online resources for accessibility audits because your app
 doesn't have a URL to point the auditor to.
 These features bring those auditing tools to your Electron app. You can
 choose to add audits to your tests with Spectron or use them within DevTools
 with Devtron. Read on for a summary of the tools.
 ## Spectron
 In the testing framework Spectron, you can now audit each window and `<webview>`
 tag in your application. For example:
 ```javascript
 app.client.auditAccessibility().then((audit) => {
  if (audit.failed) {
    console.error(audit.message)
  }
 })
 ```
 You can read more about this feature in [Spectron's documentation][spectron-a11y].
 ## Devtron
 In Devtron, there is an accessibility tab which will allow you to audit a
 page in your app, sort and filter the results.
 ![devtron screenshot][devtron-screenshot]
 Both of these tools are using the [Accessibility Developer Tools][a11y-devtools]
 library built by Google for Chrome. You can learn more about the accessibility
 audit rules this library uses on that [repository's wiki][a11y-devtools-wiki].
 If you know of other great accessibility tools for Electron, add them to the
 accessibility documentation with a pull request.
 ## Manually enabling accessibility features
@ -84,10 +43,6 @@ CFStringRef kAXManualAccessibility = CFSTR("AXManualAccessibility");
 }
 ```
 [devtron]: https://electronjs.org/devtron
 [devtron-screenshot]: https://cloud.githubusercontent.com/assets/1305617/17156618/9f9bcd72-533f-11e6-880d-389115f40a2a.png
 [spectron]: https://electronjs.org/spectron
 [spectron-a11y]: https://github.com/electron/spectron#accessibility-testing
 [a11y-docs]: https://www.chromium.org/developers/design-documents/accessibility#TOC-How-Chrome-detects-the-presence-of-Assistive-Technology
 [a11y-devtools]: https://github.com/GoogleChrome/accessibility-developer-tools
 [a11y-devtools-wiki]: https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules
--- a/docs/tutorial/automated-testing-with-a-custom-driver.md
+++ b/docs/tutorial/automated-testing-with-a-custom-driver.md
@ -1,135 +0,0 @@
 # Automated Testing with a Custom Driver
 To write automated tests for your Electron app, you will need a way to "drive" your application. [Spectron](https://electronjs.org/spectron) is a commonly-used solution which lets you emulate user actions via [WebDriver](https://webdriver.io/). However, it's also possible to write your own custom driver using node's builtin IPC-over-STDIO. The benefit of a custom driver is that it tends to require less overhead than Spectron, and lets you expose custom methods to your test suite.
 To create a custom driver, we'll use Node.js' [child_process](https://nodejs.org/api/child_process.html) API. The test suite will spawn the Electron process, then establish a simple messaging protocol:
 ```js
 const childProcess = require('child_process')
 const electronPath = require('electron')
 // spawn the process
 const env = { /* ... */ }
 const stdio = ['inherit', 'inherit', 'inherit', 'ipc']
 const appProcess = childProcess.spawn(electronPath, ['./app'], { stdio, env })
 // listen for IPC messages from the app
 appProcess.on('message', (msg) => {
  // ...
 })
 // send an IPC message to the app
 appProcess.send({ my: 'message' })
 ```
 From within the Electron app, you can listen for messages and send replies using the Node.js [process](https://nodejs.org/api/process.html) API:
 ```js
 // listen for IPC messages from the test suite
 process.on('message', (msg) => {
  // ...
 })
 // send an IPC message to the test suite
 process.send({ my: 'message' })
 ```
 We can now communicate from the test suite to the Electron app using the `appProcess` object.
 For convenience, you may want to wrap `appProcess` in a driver object that provides more high-level functions. Here is an example of how you can do this:
 ```js
 class TestDriver {
  constructor ({ path, args, env }) {
    this.rpcCalls = []
    // start child process
    env.APP_TEST_DRIVER = 1 // let the app know it should listen for messages
    this.process = childProcess.spawn(path, args, { stdio: ['inherit', 'inherit', 'inherit', 'ipc'], env })
    // handle rpc responses
    this.process.on('message', (message) => {
      // pop the handler
      const rpcCall = this.rpcCalls[message.msgId]
      if (!rpcCall) return
      this.rpcCalls[message.msgId] = null
      // reject/resolve
      if (message.reject) rpcCall.reject(message.reject)
      else rpcCall.resolve(message.resolve)
    })
    // wait for ready
    this.isReady = this.rpc('isReady').catch((err) => {
      console.error('Application failed to start', err)
      this.stop()
      process.exit(1)
    })
  }
  // simple RPC call
  // to use: driver.rpc('method', 1, 2, 3).then(...)
  async rpc (cmd, ...args) {
    // send rpc request
    const msgId = this.rpcCalls.length
    this.process.send({ msgId, cmd, args })
    return new Promise((resolve, reject) => this.rpcCalls.push({ resolve, reject }))
  }
  stop () {
    this.process.kill()
  }
 }
 ```
 In the app, you'd need to write a simple handler for the RPC calls:
 ```js
 const METHODS = {
  isReady () {
    // do any setup needed
    return true
  }
  // define your RPC-able methods here
 }
 const onMessage = async ({ msgId, cmd, args }) => {
  let method = METHODS[cmd]
  if (!method) method = () => new Error('Invalid method: ' + cmd)
  try {
    const resolve = await method(...args)
    process.send({ msgId, resolve })
  } catch (err) {
    const reject = {
      message: err.message,
      stack: err.stack,
      name: err.name
    }
    process.send({ msgId, reject })
  }
 }
 if (process.env.APP_TEST_DRIVER) {
  process.on('message', onMessage)
 }
 ```
 Then, in your test suite, you can use your test-driver as follows:
 ```js
 const test = require('ava')
 const electronPath = require('electron')
 const app = new TestDriver({
  path: electronPath,
  args: ['./app'],
  env: {
    NODE_ENV: 'test'
  }
 })
 test.before(async t => {
  await app.isReady
 })
 test.after.always('cleanup', async t => {
  await app.stop()
 })
 ```
--- a/docs/tutorial/automated-testing.md
+++ b/docs/tutorial/automated-testing.md
@ -0,0 +1,265 @@
 # Automated Testing
 Test automation is an efficient way of validating that your application code works as intended.
 While Electron doesn't actively maintain its own testing solution, this guide will go over a couple
 ways you can run end-to-end automated tests on your Electron app.
 ## Using the WebDriver interface
 From [ChromeDriver - WebDriver for Chrome][chrome-driver]:
 > WebDriver is an open source tool for automated testing of web apps across many
 > browsers. It provides capabilities for navigating to web pages, user input,
 > JavaScript execution, and more. ChromeDriver is a standalone server which
 > implements WebDriver's wire protocol for Chromium. It is being developed by
 > members of the Chromium and WebDriver teams.
 There are a few ways that you can set up testing using WebDriver.
 ### With WebdriverIO
 [WebdriverIO](https://webdriver.io/) (WDIO) is a test automation framework that provides a
 Node.js package for testing with WebDriver. Its ecosystem also includes various plugins
 (e.g. reporter and services) that can help you put together your test setup.
 #### Install the testrunner
 First you need to run the WebdriverIO starter toolkit in your project root directory:
 ```sh npm2yarn
 npx wdio . --yes
 ```
 This installs all necessary packages for you and generates a `wdio.conf.js` configuration file.
 #### Connect WDIO to your Electron app
 Update the capabilities in your configuration file to point to your Electron app binary:
 ```javascript title='wdio.conf.js'
 export.config = {
  // ...
  capabilities: [{
    browserName: 'chrome',
    'goog:chromeOptions': {
      binary: '/path/to/your/electron/binary', // Path to your Electron binary.
      args: [/* cli arguments */] // Optional, perhaps 'app=' + /path/to/your/app/
    }
  }]
  // ...
 }
 ```
 #### Run your tests
 To run your tests:
 ```sh
 $ npx wdio run wdio.conf.js
 ```
 [chrome-driver]: https://sites.google.com/chromium.org/driver/
 ### With Selenium
 [Selenium](https://www.selenium.dev/) is a web automation framework that
 exposes bindings to WebDriver APIs in many languages. Their Node.js bindings
 are available under the `selenium-webdriver` package on NPM.
 #### Run a ChromeDriver server
 In order to use Selenium with Electron, you need to download the `electron-chromedriver`
 binary, and run it:
 ```sh npm2yarn
 npm install --save-dev electron-chromedriver
 ./node_modules/.bin/chromedriver
 Starting ChromeDriver (v2.10.291558) on port 9515
 Only local connections are allowed.
 ```
 Remember the port number `9515`, which will be used later.
 #### Connect Selenium to ChromeDriver
 Next, install Selenium into your project:
 ```sh npm2yarn
 npm install --save-dev selenium-webdriver
 ```
 Usage of `selenium-webdriver` with Electron is the same as with
 normal websites, except that you have to manually specify how to connect
 ChromeDriver and where to find the binary of your Electron app:
 ```js title='test.js'
 const webdriver = require('selenium-webdriver')
 const driver = new webdriver.Builder()
  // The "9515" is the port opened by ChromeDriver.
  .usingServer('http://localhost:9515')
  .withCapabilities({
    'goog:chromeOptions': {
      // Here is the path to your Electron binary.
      binary: '/Path-to-Your-App.app/Contents/MacOS/Electron'
    }
  })
  .forBrowser('chrome') // note: use .forBrowser('electron') for selenium-webdriver <= 3.6.0
  .build()
 driver.get('http://www.google.com')
 driver.findElement(webdriver.By.name('q')).sendKeys('webdriver')
 driver.findElement(webdriver.By.name('btnG')).click()
 driver.wait(() => {
  return driver.getTitle().then((title) => {
    return title === 'webdriver - Google Search'
  })
 }, 1000)
 driver.quit()
 ```
 ## Using a custom test driver
 It's also possible to write your own custom driver using Node.js' built-in IPC-over-STDIO.
 Custom test drivers require you to write additional app code, but have lower overhead and let you
 expose custom methods to your test suite.
 To create a custom driver, we'll use Node.js' [`child_process`](https://nodejs.org/api/child_process.html) API.
 The test suite will spawn the Electron process, then establish a simple messaging protocol:
 ```js title='testDriver.js'
 const childProcess = require('child_process')
 const electronPath = require('electron')
 // spawn the process
 const env = { /* ... */ }
 const stdio = ['inherit', 'inherit', 'inherit', 'ipc']
 const appProcess = childProcess.spawn(electronPath, ['./app'], { stdio, env })
 // listen for IPC messages from the app
 appProcess.on('message', (msg) => {
  // ...
 })
 // send an IPC message to the app
 appProcess.send({ my: 'message' })
 ```
 From within the Electron app, you can listen for messages and send replies using the Node.js
 [`process`](https://nodejs.org/api/process.html) API:
 ```js title='main.js'
 // listen for messages from the test suite
 process.on('message', (msg) => {
  // ...
 })
 // send a message to the test suite
 process.send({ my: 'message' })
 ```
 We can now communicate from the test suite to the Electron app using the `appProcess` object.
 For convenience, you may want to wrap `appProcess` in a driver object that provides more
 high-level functions. Here is an example of how you can do this. Let's start by creating
 a `TestDriver` class:
 ```js title='testDriver.js'
 class TestDriver {
  constructor ({ path, args, env }) {
    this.rpcCalls = []
    // start child process
    env.APP_TEST_DRIVER = 1 // let the app know it should listen for messages
    this.process = childProcess.spawn(path, args, { stdio: ['inherit', 'inherit', 'inherit', 'ipc'], env })
    // handle rpc responses
    this.process.on('message', (message) => {
      // pop the handler
      const rpcCall = this.rpcCalls[message.msgId]
      if (!rpcCall) return
      this.rpcCalls[message.msgId] = null
      // reject/resolve
      if (message.reject) rpcCall.reject(message.reject)
      else rpcCall.resolve(message.resolve)
    })
    // wait for ready
    this.isReady = this.rpc('isReady').catch((err) => {
      console.error('Application failed to start', err)
      this.stop()
      process.exit(1)
    })
  }
  // simple RPC call
  // to use: driver.rpc('method', 1, 2, 3).then(...)
  async rpc (cmd, ...args) {
    // send rpc request
    const msgId = this.rpcCalls.length
    this.process.send({ msgId, cmd, args })
    return new Promise((resolve, reject) => this.rpcCalls.push({ resolve, reject }))
  }
  stop () {
    this.process.kill()
  }
 }
 module.exports = { TestDriver };
 ```
 In your app code, can then write a simple handler to receive RPC calls:
 ```js title='main.js'
 const METHODS = {
  isReady () {
    // do any setup needed
    return true
  }
  // define your RPC-able methods here
 }
 const onMessage = async ({ msgId, cmd, args }) => {
  let method = METHODS[cmd]
  if (!method) method = () => new Error('Invalid method: ' + cmd)
  try {
    const resolve = await method(...args)
    process.send({ msgId, resolve })
  } catch (err) {
    const reject = {
      message: err.message,
      stack: err.stack,
      name: err.name
    }
    process.send({ msgId, reject })
  }
 }
 if (process.env.APP_TEST_DRIVER) {
  process.on('message', onMessage)
 }
 ```
 Then, in your test suite, you can use your `TestDriver` class with the test automation
 framework of your choosing. The following example uses
 [`ava`](https://www.npmjs.com/package/ava), but other popular choices like Jest
 or Mocha would work as well:
 ```js title='test.js'
 const test = require('ava')
 const electronPath = require('electron')
 const { TestDriver } = require('./testDriver')
 const app = new TestDriver({
  path: electronPath,
  args: ['./app'],
  env: {
    NODE_ENV: 'test'
  }
 })
 test.before(async t => {
  await app.isReady
 })
 test.after.always('cleanup', async t => {
  await app.stop()
 })
 ```
--- a/docs/tutorial/using-selenium-and-webdriver.md
+++ b/docs/tutorial/using-selenium-and-webdriver.md
@ -1,173 +0,0 @@
 # Selenium and WebDriver
 From [ChromeDriver - WebDriver for Chrome][chrome-driver]:
 > WebDriver is an open source tool for automated testing of web apps across many
 > browsers. It provides capabilities for navigating to web pages, user input,
 > JavaScript execution, and more. ChromeDriver is a standalone server which
 > implements WebDriver's wire protocol for Chromium. It is being developed by
 > members of the Chromium and WebDriver teams.
 ## Setting up Spectron
 [Spectron][spectron] is the officially supported ChromeDriver testing framework
 for Electron. It is built on top of [WebdriverIO](https://webdriver.io/) and
 has helpers to access Electron APIs in your tests and bundles ChromeDriver.
 ```sh
 $ npm install --save-dev spectron
 ```
 ```javascript
 // A simple test to verify a visible window is opened with a title
 const Application = require('spectron').Application
 const assert = require('assert')
 const myApp = new Application({
  path: '/Applications/MyApp.app/Contents/MacOS/MyApp'
 })
 const verifyWindowIsVisibleWithTitle = async (app) => {
  await app.start()
  try {
    // Check if the window is visible
    const isVisible = await app.browserWindow.isVisible()
    // Verify the window is visible
    assert.strictEqual(isVisible, true)
    // Get the window's title
    const title = await app.client.getTitle()
    // Verify the window's title
    assert.strictEqual(title, 'My App')
  } catch (error) {
    // Log any failures
    console.error('Test failed', error.message)
  }
  // Stop the application
  await app.stop()
 }
 verifyWindowIsVisibleWithTitle(myApp)
 ```
 ## Setting up with WebDriverJs
 [WebDriverJs](https://www.selenium.dev/selenium/docs/api/javascript/index.html) provides
 a Node package for testing with web driver, we will use it as an example.
 ### 1. Start ChromeDriver
 First you need to download the `chromedriver` binary, and run it:
 ```sh
 $ npm install electron-chromedriver
 $ ./node_modules/.bin/chromedriver
 Starting ChromeDriver (v2.10.291558) on port 9515
 Only local connections are allowed.
 ```
 Remember the port number `9515`, which will be used later
 ### 2. Install WebDriverJS
 ```sh
 $ npm install selenium-webdriver
 ```
 ### 3. Connect to ChromeDriver
 The usage of `selenium-webdriver` with Electron is the same with
 upstream, except that you have to manually specify how to connect
 chrome driver and where to find Electron's binary:
 ```javascript
 const webdriver = require('selenium-webdriver')
 const driver = new webdriver.Builder()
  // The "9515" is the port opened by chrome driver.
  .usingServer('http://localhost:9515')
  .withCapabilities({
    'goog:chromeOptions': {
      // Here is the path to your Electron binary.
      binary: '/Path-to-Your-App.app/Contents/MacOS/Electron'
    }
  })
  .forBrowser('chrome') // note: use .forBrowser('electron') for selenium-webdriver <= 3.6.0
  .build()
 driver.get('http://www.google.com')
 driver.findElement(webdriver.By.name('q')).sendKeys('webdriver')
 driver.findElement(webdriver.By.name('btnG')).click()
 driver.wait(() => {
  return driver.getTitle().then((title) => {
    return title === 'webdriver - Google Search'
  })
 }, 1000)
 driver.quit()
 ```
 ## Setting up with WebdriverIO
 [WebdriverIO](https://webdriver.io/) provides a Node package for testing with web
 driver.
 ### 1. Start ChromeDriver
 First you need to download the `chromedriver` binary, and run it:
 ```sh
 $ npm install electron-chromedriver
 $ ./node_modules/.bin/chromedriver --url-base=wd/hub --port=9515
 Starting ChromeDriver (v2.10.291558) on port 9515
 Only local connections are allowed.
 ```
 Remember the port number `9515`, which will be used later
 ### 2. Install WebdriverIO
 ```sh
 $ npm install webdriverio
 ```
 ### 3. Connect to chrome driver
 ```javascript
 const webdriverio = require('webdriverio')
 const options = {
  host: 'localhost', // Use localhost as chrome driver server
  port: 9515, // "9515" is the port opened by chrome driver.
  desiredCapabilities: {
    browserName: 'chrome',
    'goog:chromeOptions': {
      binary: '/Path-to-Your-App/electron', // Path to your Electron binary.
      args: [/* cli arguments */] // Optional, perhaps 'app=' + /path/to/your/app/
    }
  }
 }
 const client = webdriverio.remote(options)
 client
  .init()
  .url('http://google.com')
  .setValue('#q', 'webdriverio')
  .click('#btnG')
  .getTitle().then((title) => {
    console.log('Title was: ' + title)
  })
  .end()
 ```
 ## Workflow
 To test your application without rebuilding Electron,
 [place](application-distribution.md)
 your app source into Electron's resource directory.
 Alternatively, pass an argument to run with your Electron binary that points to
 your app's folder. This eliminates the need to copy-paste your app into
 Electron's resource directory.
 [chrome-driver]: https://sites.google.com/a/chromium.org/chromedriver/
 [spectron]: https://electronjs.org/spectron