DATA.GOV.UK - Visual Regression Testing

This repo contains the BackstopJS configuration to run visual regression tests against the data.gov.uk admin panel.

Credits

This repo is a fork of the work done by the Digital Marketplace team

Setup

Ensure that you have the latest versions of Node js and NPM installed locally.

Clone this repo and run:

npm install

Clone the .env.example file and create a .env file at the root of your local repo. Fill in the environment variables in accordance with their descriptions in .env.example. The variables STANDARD_RESOURCE, ORGANOGRAM_RESOURCE_NO_SOURCE and ORGANOGRAM_RESOURCE_SOURCE_SPECIFIED will be auto generated by the setup script (see below) if they are left blank or will be overwritten if they are filled in and setup is re-run.

It's recommended that you run the VRT against a blank environment with only the standard DGU test data. See the data setup section below for more details including instructions on how to setup test data locally. If you run this against a populated environment, filling in the necessary env variables, then the VRT will still run as expected but it will "fail" as it isn't running against the data that it has been approved against and is therefore expecting. This isn't a problem if you just want to check specific features manually but can be an issue if you want to update the tests.

Additionally, it's recommended you set the docker ckan environment variable CKAN_DEBUG to false for your local instance. This will turn off the debug console and other debug features for the ckan admin panel. Whilst this repo includes code to handle said debug features, it slows down local instances and can make the VRT sluggish and difficult to run.

How it works

BackstopJS compares an instance of ckan that you specify in the .env file, uses puppeteer to run that instance through a set of scenarios aka: views within the ckan user journey and variations of those views, and compares screenshots taken from those scenarios to existing screenshots stored in source control. Backstop will then produce a html report for interrogation, making use of puppeteer's rich diff functionality. You as the dev working on a feature or bugfix can then assess if any changes produced are acceptable or not and approve these changes, updating the screenshots in source control.

The original intent of these tests was to assess changes between versions of ckan to ensure that the admin frontend hadn't broken when upgrading. This however has evolved as we start to think about ways to integrate this testing suite into a consistent CI flow.

If these admin journeys change at any point, the scenarios in this tool will need to be reviewed.

Running tests locally

Quick start

npm run setup OR npm run update-resource-ids

Either of these scripts will ensure that your local VRT is synced up with the data in your local ckan instance so that tests have all the data they need to run properly. Run one of these scripts before running test.

npm run test

With the basic setup as supplied, this will run an initial test on the domain specified in your .env file. By default, this will run against screenshots stored for ckan version 2.8. You can see the report by opening backstop_data/html_report/index.html in your browser. If there are any changes between the instance you are running and the stored screenshots then these tests will fail, showing diffs in the produced report. If you want to update these screenshots to the most recently run test screenshots, you can run:

npm run approve

This will update your saved screenshots to the most recently run tests. Remember you still have to commit these changes to source control. Every time you make a positive change, simply run npm run test to ensure visual fidelity and then npm run approve to update your reference screenshots.

Different ckan versions and view filters

This repo includes multiple scenario "sets" for different version of ckan. Currently, this includes a set of screenshots for ckan 2.9, 2.8 and 2.7. By default, running test will use the config for 2.8 as this is the version currently running in production. To use a different config, for example 2.9, just pass this as an argument:

npm run test 2.9

This will run your local instance against ckan 2.9 instead of 2.8. Remember that your .env file needs to reflect the scenario set that you are testing eg: Don't run a local version of 2.9 against the 2.8 scenarios (unless this is explicitly what you want to do, in which case do not approve these changes as they will override the existing 2.8 scenario set).

Running the default test against any ckan config currently command will trigger 40 scenarios, each one with 3 views (desktop, tablet and phone), totalling to 120 tests. This will take a while to run all of these and may start eating into your machine's memory in a non-trivial way. To subvert this, scenarios are broken down into the following sections:

• homepage - 1 scenario
• dataset - 11 scenarios
• publisher - 5 scenarios
• harvest - 9 scenarios
• user - 8 scenarios
• dashboard - 3 scenarios
• admin - 3 scenarios

You can use these section keywords to only target specific scenarios using the command npm run test [config] [keyword]. For example, to only test the harvest section for ckan 2.7, simply run npm run test 2.7 harvest to run only those 9 scenarios for 2.7.

For running ckan locally, it is recommended you use the docker-ckan project. Please see the documentation for that project for details on how to run ckan locally via docker.

Updating tests

To add new test scenarios, you can update configs/ckan-config.js under the scenarios key and follow the format of other entries in the list. Each scenario expects the following attributes:

• label: The name of the scenario. Please be descriptive and use a prefix where appropriate eg: if your new test is within the publisher views, the label should be "Publisher - [your test]".
• url: The url of the scenario. Remember to make use of process.env.DOMAIN as necessary.

In addition to the above, you can add the following additional attributes depending on what you need for your scenario:

• skipLogin: This will skip the login step. For instances where you are testing a view that relies on not being logged in.
• submitForm: Assumes there is a single form on your view and submits it. This is principally for instances where you want to test how form errors look on forms. Please see the Gotchas section below for details on a minor quirk within ckan forms.
• uploadField: Waits for elements specific to the image upload functionality to ensure that views using this pattern run properly for tests.
• select2: Waits for elements specific to the select2 custom dropdown vendor within to ensure that views using this pattern run properly for tests.
• slugPreview: Waits for elements specific to the slug preview functionality custom dropdown vendor within to ensure that views using this pattern run properly for tests.
• textarea: For views that use a textarea field, this attribute explicitly specifies the heigh of these fields. This is to fix an issue where the resize arrow would shift unpredictably on tests that included textareas.
• tableRows: Sets the content of data table row data to zero for views using the data table pattern to ensure that dynamic data doesn't interfere with tests.
• dashboard: Removes some specific elements related to dashboard views (dashboard default and user activity) to ensure that dynamic data doesn't interfere with tests.
• harvestJobList: An attribute for a specific view to clear dynamic data for the harvest job list view.
• tableShowMore: Removes the show more pattern from data tables to combat an unpredictable bug where the VRT would sometimes beat the js that rendered table hiding and showing on long data tables.

Make sure to review the test report before approving to check that no unwanted changes have slipped in. You can find this at backstop_data/html_report/index.html and can view it in your browser. It will populate following a test run.

If you want to add a new scenario set, you can add a new config root under the configs directory. If you are adding a new ckan version, it is recommended that you follow the setup in the other ckan config roots (currently 2.7.js, 2.8.js and 2.9.js) and reference ckan-config.js as they do. If you are adding a completely new set of scenarios relating to datagovuk but unrelated to the ckan admin panel, you will need to create an entirely new config. You can use ckan-config.js as a basis for your config. If you want to use filters in your new scenario, it is recommended that you edit test.js and amend the code to properly catch your filter.

Ckan data setup

In order to ensure that tests are based on consistent data and therefore consistent views, the tests in this repo are based on datagovuk's test data setup. You can run the commands specified in the documentation for ckanext-datagovuk from within the ckan image of the docker-ckan stack.

You can verify this within the VRT after adding the test data, as well as the test organogram dataset, a necessary dataset for the VRT tests, by running npm run setup. This will check that each piece of test data exists and add the organogram dataset via puppeteer. As part of this script, the env variables STANDARD_RESOURCE, ORGANOGRAM_RESOURCE_NO_SOURCE and ORGANOGRAM_RESOURCE_SOURCE_SPECIFIED will be auto generated. You can also run this function as a standalone command by running:

npm run update-resource-id

You can pass either organogram or default to run only the organogram or the standard resource generations respectively. Please note that if you have already populated these env variables, that they will be overwritten by this script.

Debugging issues in the code

Because a lot of the code in this repo is abstracted by puppeteer running a headless browser instance, as well as relying on an instance of docker-ckan running alongside, debugging issues can be difficult. If you encounter issues whilst running or developing upon this repo, it is recommended that you try the following:

Check the HTML report

The HTML report, found at backstop_data/html_report/index.html will often have some useful error messages for who a scenario didn't run and is typically more reliable that backstop's own CLI. An example of this is that backstop will sometimes give you an error message that looks like this:

Reference image not found [scenario_reference].png

Historically, this error has been less an indication of what a problem is and more a response from backstop to a problem and checking the HTML report has revealed that this is usually caused by a problem whilst running a particular scenario.

Look at the Gotchas section below.

It could be that the issue you're encountering has already been captured and explored.

Use puppeteer's screenshot functionality.

await page.screenshot({path: 'screenshot.png'});

In the above example, puppeteer will take a screenshot of the current instance of puppeteer and output it to screenshot.png at the root of your local repo. You can change the output location to whatever you want and put this line virtually anywhere in this code and it'll be valid. This will give you a representation of puppeteer's state at any one time compared to what you're expecting.

If you do find an issue, please create a pull request to either fix the issue or add it to the gotchas below.

Gotchas

Issues with ckan

• In ckan scenarios that involve submitting a form to test for form errors, the ideal would be to use form.submit() in line with the browser API. Unfortunately, ckan's basic-form js module hijacks the form submission flow, meaning that the submit button for that form has to be explicitly clicked. This can still be done via puppeteer, however it's not especially neat.
• On any view with an upload field, notably the organogram resource views, ckan's image-upload js module takes a little while to load and puppeteer manages to beat it 9 times out of 10. The upload-field.js script has therefore been added to wait for the js to load.
• Similar to the above around having to wait for js to load in a view, some forms have an autofill slug field, managed by ckan's slug-preview module. This both hides one field and generates markup for a non-input field which auto-updates based on an associated input field. Like above, puppeteer beats the rendering work for this 9 times out of 10, creating inconsistent view tests. The slug-preview.js script exists to account for this.

Backstop limitations

• Backstop doesn't have a testing library for IE or Edge. The reason for this is that backstop relies on the headless infrastructure provided by engines like puppeteer or caspar, something that old microsoft browsers don't support. We have a need to support IE11 at least for this project, so you may need to do some additional manual testing on top of running these tests.
• Backstop can only run against a very specific stack. If any additional data is added, because Backstop explicitly tests against visual discrepancy, then tests will fail. This means that if any additional data is added to a local stack or anything is changed, either said data will need to be removed or you will have to reset your local stack.
• Backstop storing screenshots in source control is potentially problematic. A single set of ckan scenarios (120 screenshots) comes to approximately 15MB. A long term solution needs ot be considered for how we store scenarios to avoid taking up obscene amounts of space in source control.