Learn Nightwatch

Automate your acceptance tests and run them in real browsers!

Why?

Testing what the people using your application/website will see and their ability interact with the product is (probably) the most important part of building a web app/site. You can have amazing code, a super-fast backend and gorgeous UI, but none of that matters if people are unable to use it because of a basic bug!

User Acceptance Testing (UAT) with a tool like Nightwatch (Selenium) lets you to run real-world scenarios in your Web App which will give you confidence that the app works in the chosen device(s)/browser(s).

What?

Automated Acceptance Testing using Real Browsers.

Nightwatch is quick to setup and the tests/scenarios are easy to write.

We exhaustively read through all the tutorials, blog posts and documentation for Nightwatch (including the mailing list & StackOverflow Q&A) and have condensed our findings into this step-by-step guide.
We hope you find it useful and decide to use it for your web app/site!
Please give us feedback and if you get stuck, tell us!

Background Links

• Nightwatch website: http://nightwatchjs.org/
• Github: https://github.com/nightwatchjs/nightwatch
• Guide/docs: https://github.com/nightwatchjs/nightwatch-docs (don't be put off by the lack of docs, we're here to help if you get stuck!)
• Configuration file settings: http://nightwatchjs.org/guide#settings-file

Who?

Who should learn/use Nightwatch?

• Developers - People writing code, building web apps needing to check that everything works as expected.
• QA - Quality Assurance people who have to manually "click-test" apps/sites.
• "Testers" - Many organisations still have people who's job is to write tests for software. If you describe yourself as a "Tester" and want an easier/faster way to write your acceptance tests, read on!

How?

Quick Start (5mins)

Try it on your local machine in 5 mins by following these 3 easy steps:

1. Clone

Clone the repository by copy-pasting the following command into your terminal:

git clone https://github.com/dwyl/learn-nightwatch.git && cd learn-nightwatch && cp sample.env .env

Note: if you're curious what that last part is, see: https://github.com/dwyl/env2

2. Install¹

Install the required dependencies including Selenium Server and chromedriver:

npm install

3. Run (tests)²

Run the Nightwatch tests:

npm test

You should expect to see:

Once you see the tests pass you are well on your way to testing with Nightwatch!

¹This assumes you have node.js installed. If not, https://nodejs.org/en/download/
²Selenium Requires Java/JDK see: Java Installation section below. (don't worry, you'll be up-and-running shortly...!)
Once you have Java installed re-run the Nightwatch tests (npm test).

Step-by-Step Tutorial

Now that you have had a taste for running tests with Nightwatch, let's walk through each of the steps to get this working in your project.

Installation (in detail)

Install nightwatch

First install the nightwatch node.js module from NPM:

npm install nightwatch --save-dev

Note: while the Nightwatch docs instruct to install globally (-g), we prefer to always install devDependencies locally to the project and list them explicitly in package.json so it's clear to everyone viewing/using the project exactly which version is required to run the tests.

Install selenium-download

In order to run Browser tests Nightwatch uses Selenium. We prefer to automate the installation of Selenium using selenium-download which ensures that everyone on our team always has the latest version.

npm install selenium-download --save-dev

Download Selenium (Web Driver) Standalone Server (Script)

Once you've downloaded the the selenium-download node module, put the following code at the bottom of your nightwatch.conf.js file:

require('selenium-download').ensure('./bin', function(error) {  
   if (error) {
     return console.log(error);
   } else {
     console.log('✔ Selenium & Chromedriver downloaded to:', './bin');
   }
});

We run include this script in our nightwatch.conf.js which checks for the existence of selenium.jar before trying to run our tests. You will create your nightwatch.conf.js file in the next step.

Configuration

Once you've installed nightwatch, you will need to create a configuration file.
Some Nightwatch tutorials use a nightwatch.json file; this is good for the most basic cases but if you want to use variables in your configuration we recommend using a .js file; specifically called nightwatch.conf.js. Save this file to your project directory.

The most basic configuration is: nightwatch.conf.BASIC.js

module.exports = {
  "src_folders": [
    "test/e2e"// Where you are storing your Nightwatch e2e/UAT tests
  ],
  "output_folder": "./reports", // reports (test outcome) output by nightwatch
  "selenium": { // downloaded by selenium-download module (see readme)
    "start_process": true, // tells nightwatch to start/stop the selenium process
    "server_path": "./node_modules/nightwatch/bin/selenium.jar",
    "host": "127.0.0.1",
    "port": 4444, // standard selenium port
    "cli_args": { // chromedriver is downloaded by selenium-download (see readme)
      "webdriver.chrome.driver" : "./node_modules/nightwatch/bin/chromedriver"
    }
  },
  "test_settings": {
    "default": {
      "screenshots": {
        "enabled": true, // if you want to keep screenshots
        "path": './screenshots' // save screenshots here
      },
      "globals": {
        "waitForConditionTimeout": 5000 // sometimes internet is slow so wait.
      },
      "desiredCapabilities": { // use Chrome as the default browser for tests
        "browserName": "chrome"
      }
    },
    "chrome": {
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true // set to false to test progressive enhancement
      }
    }
  }
}

One of our favourite things about using a .js file is the ability to add comments in the file.
This makes it much easier for new people to understand what's going on.
We have a slightly more evolved nightwatch.conf.js (with Saucelabs) see: github.com/dwyl/learn-nightwatch/nightwatch.conf.js

Additional Steps for Testing a Non-NPM Project

You will need to run the config file you created to download the Selenium driver.

node nightwatch.conf.BASIC.js

Create Your Nightwatch Test

Nightwatch "looks" for tests in the /tests folder of your project by default; you can change this to what ever you prefer. We keep our Nightwatch tests in test/e2e.

This is the simplest test you can write for Nightwatch

module.exports = { // addapted from: https://git.io/vodU0
  'Guinea Pig Assert Title': function(browser) {
    browser
      .url('https://saucelabs.com/test/guinea-pig')
      .waitForElementVisible('body')
      .assert.title('I am a page title - Sauce Labs')
      .saveScreenshot('ginea-pig-test.png')
      .end();
  }
};

See: github.com/dwyl/learn-nightwatch/test/e2e

Run your Test

Depending on what you named your configuration file, run it with a command resembling the following:

node_modules/.bin/nightwatch --config nightwatch.conf.BASIC.js

If you called your config file nightwatch.conf.js (as we suggested) you can run your tests without specifying the config file:

node_modules/.bin/nightwatch

We add an entry in our package.json "scripts" section to not have to type all that each time. e.g:

"scripts": {
  "e2e": "node_modules/.bin/nightwatch"
}

Then run it as:

npm run e2e

Optional (Level Up)

postinstall script

We put an entry in the "scripts" section of our package.json to run the selenium-download script after all node_modules have been installed. e.g:

  "scripts": {
    "postinstall": "node nightwatch.conf.js"
  }

Saucelabs

Most people building web apps/sites don't have easy access to several devices/browsers to test their output, if you need to test in a range of browsers/devices Saucelabs is a great option.

In our nightwatch.conf.js we have defined saucelabs as our "default" setting.

We run our tests on saucelabs by running the following npm script/command:

npm run sauce

Which corresponds to the following complete command:

./node_modules/.bin/nightwatch -e chrome,ie11,android_s4_emulator,iphone_6_simulator

This just means "Run Nightwatch using the default configuration (Suacelabs in our case) and execute all tests in this list of browsers."

Note: you will need to have the following environment variables exported for Saucelabs to run your test:

export SAUCE_USERNAME=your-username
export SAUCE_ACCESS_KEY=your-key

If you're new to Saucelabs, checkout: github.com/dwyl/learn-saucelabs

Upload Screenshots to S3

If you decide to use Saucelabs to run your tests (in several devices/browsers), it will take screenshots for you and keep them inside Saucelabs. That's nice for people who are used to using Saucelabs, but what about the other stakeholders?

We decided to upload our screenshots to S3 and created a super-simple .html file which shows a slideshow of the images.

Example: https://isearch-ui.s3-eu-west-1.amazonaws.com/1.0.21/index.html

If you want the screenshots of tests to be uploaded to S3, you will need to have the following environment variables declared:

export AWS_S3_BUCKET=yourbucket
export AWS_REGION=eu-west-1
export AWS_ACCESS_KEY_ID=IDHERE
export AWS_SECRET_ACCESS_KEY=YOURKEY

The script we wrote to perform the uploading is: github.com/dwyl/learn-nightwatch/test/upload_screenshots_to_s3.js

The screenshots taken on Saucelabs browsers/devices are saved locally and uploaded to S3 when tests succeed.

Running your Nightwatch tests on your Continuous Integration (CI)

Running your Nightwatch tests on CI is easy on CodeShip.
We usually set the required (minimum) node version in our package.json e.g:

"engines": {
  "node": "4.4.6"
},

Once you have the desired version of node installed.

Setup Commands:

# install dependencies:
npm install

Test Command:

# run tests
npm test

That's it.

Note: while the tests run seamlessly on CodeShip we were unable to get Selenium standalone working on Travis-CI if you have time to help us, please see: dwyl#8

tl;dr

More detail than you will probably need ... but we're keeping for completeness.

Background

Why Nightwatch instead of xyz...?

We first looked at NightmareJS, and even though it looks really good (fast), we saw the reaction non-technical people had when we mentioned it and did not want to have to explain the name to people/clients every time, so instead opted for nightwatch. If nightmare ever change their name, we could re-consider it.

Research

• Basic intro: http://juristr.com/blog/2014/02/nightwatch-test-automation/
• Page Object Pattern: http://martinfowler.com/bliki/PageObject.html
• Nightwatch with React: https://www.syncano.io/blog/testing-syncano/
• How to run a single Nightwatch test: http://stackoverflow.com/questions/28308990/how-to-run-a-single-test-in-nightwatch/29701199#29701199
• Custom Commands: http://nightwatchjs.org/guide#writing-custom-commands
• Nightwatch Global Variables: http://stackoverflow.com/questions/25067391/how-to-make-a-globally-accessible-variable/
• Travis with Saucelabs: http://samsaccone.com/posts/testing-with-travis-and-sauce-labs.html
• Selenium Status: http://stackoverflow.com/questions/6517501/selenium-2-how-to-check-that-server-is-running-and-stop-the-server
• Silence Selenium on Travis: http://andrew.yurisich.com/work/2014/08/30/silence-noisy-selenium-server-output-in-travis-ci/
• Intro by @mikberg: https://medium.com/@mikaelberg/zero-to-hero-with-end-to-end-tests-using-nightwatch-saucelabs-and-travis-e932c8deb695
• Intro (Angular-focussed): http://g00glen00b.be/e2e-testing-nightwatch-js/
• Nightwatchjs: how to check if element exists without creating an error/failure/exception http://stackoverflow.com/questions/31687027/nightwatchjs-how-to-check-if-element-exists-without-creating-an-error-failure-e
• Can I create reusable test steps in nightwatch.js? http://stackoverflow.com/questions/31388280/can-i-create-reusable-test-steps-in-nightwatch-js
• Nightwatch on Saucelabs: https://github.com/saucelabs-sample-test-frameworks/JS-Nightwatch.js

Setup (Detail)

Installing Java (Runtime Environment JRE)

If you see the following message while trying to run the tests:

While we prefer not to run Java on our machines for security reasons Selenium is still the best way of running tests in real browsers.

Check if you have Java installed on your local machine:

How do I install Java? https://www.java.com/en/download/help/download_options.xml pick your Operating System and follow the instructions

##### Mac OSX? (use homebrew)

If you haven't updated brew in a while, do that first:

brew update

That will install cask which is now part of Homebrew.

Now you can install Java:

brew cask install java

You should see something like this:

See: http://stackoverflow.com/questions/24342886/how-to-install-java-8-on-mac

Manual Selenium Install

If you prefer to install it manually that's an option.

Visit: http://www.seleniumhq.org/download/ and download the latest version.

When downloading the selenium-server-standalone-2.53.0.jar you may see a warning in your browser:

Click on "keep" to save the file. Once you have it, put it in the bin directory of your project and re-name it to selenium.jar (without the version number).

StackOverflow Questions

Remind me to Respond to these:

• http://stackoverflow.com/questions/24314040/getting-started-with-nightwatch-js
• http://stackoverflow.com/questions/37699036/is-it-possible-to-start-a-selenium-server-inside-travis
• http://stackoverflow.com/questions/25919673/file-upload-testing-in-nightwatch-js
• http://stackoverflow.com/questions/31388280/can-i-create-reusable-test-steps-in-nightwatch-js/31393249#31393249
• use saucelabs with nightwatch? http://stackoverflow.com/questions/36137270/how-to-use-saucelabs-with-nightwatch
• keypress: http://stackoverflow.com/questions/31812935/nightwatch-testing-sendkeys-and-keys-not-sending-key-clicks ... answer
• Run Selenium as child process: http://stackoverflow.com/questions/27408864/cant-launch-selenium-phantomjs-ghostdriver-as-child-processes
• Current running browser: http://stackoverflow.com/questions/38102543/when-running-nightwatch-js-test-how-can-i-get-the-name-of-browser-currently-runn

Cons (of using Nightwatch)

• Selenium is not the fastest way to run tests.