Add resemblejs to smoke tests (#3899)

### Installing resemblejs
#### Getting resemblejs to work in the repo
Override yarn config in smoke tests to use node runtime instead of
electron. It seems that the tests need to compile `resemblejs` to target
`node` instead of `electron` so that the correct header files are used.

Using `npm` worked because it doesn't have anything set to target the
Electron runtime. Yarn was merging in the options `<positron
root>/.yarnrc`, which sets the runtime to `electron`.

#### Building resemblejs (which depends on canvas) will require some
configuration updates:

OS X
brew install pkg-config cairo libpng jpeg giflib

Debian
sudo apt-get install -y libcairo-dev libsdl-pango-dev libjpeg-dev
libgif-dev

Windows
https://github.com/Automattic/node-canvas/wiki/Installation:-Windows

Documentation PR: posit-dev/positron-wiki#55

### Resemblejs usage

Resemblejs is being used to compare plots in Positron at runtime against
master plots stored in the repo. The master plots are generated locally
with a line of code similar to:

```
await app.code.driver.getLocator('.plot-instance .image-wrapper img').screenshot({ path: 'screenshot.png' });
```

Screenshot.png is then renamed and moved to the proper directory. Note
that screenshot.png is collected in CI and not locally. Files collected
locally differ too much from CI to be use with a reasonable comparison
threshold.

Tests fail currently if the "rawMismatchPercentage" is greater than 2%.
Locally generated snapshots seem to differ from CI snapshots by about
5.0 - 7.5%. An all black image differs by about 98% but an all white
image only differs by 4.6%. This is why the threshold is set lower that
the difference between a local and a CI image.

### QA Notes - use of resemblejs in code
```ts
// import { compareImages } from 'resemblejs';
// importing the dependency as require to avoid error once transpiled to .js
const compareImages = require('resemblejs/compareImages');
```
Using the package might need to be done like this. My early testing had
some issues with imports and it didn't know how to find `resemblejs`
during runtime.

---------

Co-authored-by: Chris Mead <[email protected]>

.github/workflows/positron-full-test.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Setup Build Environment
         run: |
           sudo apt-get update
-          sudo apt-get install -y vim curl build-essential clang make cmake git python3-pip python-is-python3 libsodium-dev libxkbfile-dev pkg-config libsecret-1-dev libxss1 dbus xvfb libgtk-3-0 libgbm1 libnss3 libnspr4 libasound2 libkrb5-dev
+          sudo apt-get install -y vim curl build-essential clang make cmake git python3-pip python-is-python3 libsodium-dev libxkbfile-dev pkg-config libsecret-1-dev libxss1 dbus xvfb libgtk-3-0 libgbm1 libnss3 libnspr4 libasound2 libkrb5-dev libcairo-dev libsdl-pango-dev libjpeg-dev libgif-dev
           sudo cp build/azure-pipelines/linux/xvfb.init /etc/init.d/xvfb
           sudo chmod +x /etc/init.d/xvfb
           sudo update-rc.d xvfb defaults
@@ -54,6 +54,8 @@ jobs:
           # Perform the main yarn command; this installs all Node packages and
           # dependencies
           yarn --immutable --network-timeout 120000
+          yarn --cwd test/automation install
+          yarn --cwd test/smoke install
 
       - name: Compile and Download
         run: yarn npm-run-all --max_old_space_size=4095 -lp compile "electron x64" playwright-install download-builtin-extensions
@@ -137,3 +139,11 @@ jobs:
         with:
           name: run-artifacts
           path: .build/logs/smoke-tests-electron/
+
+      - name: slack-smoke-test-report
+        if: always()
+        uses: ivanklee86/[email protected]
+        env:
+          SLACK_CHANNEL: C06DNFJSHPD
+          SLACK_TOKEN: ${{ secrets.SMOKE_TESTS_SLACK_TOKEN }}
+          XUNIT_PATH: .build/logs/smoke-tests-electron/test-results/results.xml

build/npm/dirs.js

@@ -64,10 +64,13 @@ const dirs = [
 	'extensions/vscode-test-resolver',
 	'remote',
 	'remote/web',
-	'test/automation',
+	// --- Start Positron
 	'test/integration/browser',
 	'test/monaco',
-	'test/smoke',
+	// no need to compile smoke tests during release builds
+	// 'test/automation',
+	// 'test/smoke',
+	// --- End Positron
 	'.vscode/extensions/vscode-selfhost-test-provider',
 ];
 

test/automation/src/positron/positronPlots.ts

@@ -50,4 +50,12 @@ export class PositronPlots {
 	async waitForNoPlots() {
 		await this.code.waitForElement(CURRENT_PLOT, (result) => !result);
 	}
+
+	async getCurrentPlotAsBuffer(): Promise<Buffer> {
+		return this.code.driver.getLocator(CURRENT_PLOT).screenshot();
+	}
+
+	async getCurrentStaticPlotAsBuffer(): Promise<Buffer> {
+		return this.code.driver.getLocator(CURRENT_STATIC_PLOT).screenshot();
+	}
 }

test/smoke/.yarnrc

@@ -0,0 +1,5 @@
+disturl ""
+target ""
+ms_build_id ""
+runtime "node"
+build_from_source ""

test/smoke/README.md

@@ -148,6 +148,10 @@ Graphviz is external software that has a Python package to render graphs. Instal
 * **Windows** - `choco install graphviz`
 * **Mac** - `brew install graphviz`
 
+## Environment Setup - Resemblejs dependency
+
+Make sure that you have followed the [Machine Setup](https://connect.posit.it/positron-wiki/machine-setup.html) instructions so that you can be sure you are set up to build resemblejs (which depends on node-canvas).
+
 ## Build step
 
 The tests are written in typescript, but unlike the main Positron typescript, the files are not transpiled by the build daemons. So when running tests you'll need to navigate into the smoke tests location and run the build watcher:
@@ -223,4 +227,10 @@ When a run is complete, you can debug any test failures that occurred using the
 
 Playwright traces can be drag and dropped to the [Trace Viewer](https://trace.playwright.dev/).  The trace will usually give you a good visualization of the failed test, but they can be sparse on details.  More details are available from the run log (smoke-test-runner.log).  It has a start and end marker for each test case.
 
+## Notes About Updating Specific Tests
+
+### Plot Tests That Use Resemblejs
+
+In order to get the "golden screenshots" used for plot comparison is CI, you will need to temporarily uncomment the line of code marked with `capture master image in CI` or add a similar line of code for a new case.  We must use CI taken snapshots because if the "golden screenshots" are taken locally, they will differ too much from the CI images to be useable with a proper threshold.  You can't compare the current runtime plot against a snapshot until you have established a baseline screenshot from CI that is saved to `test/smoke/plots`.
+
 <!-- End Positron -->

test/smoke/package.json

@@ -11,9 +11,11 @@
     "mocha": "node ../node_modules/mocha/bin/mocha"
   },
   "dependencies": {
+    "canvas": "^2.11.2",
     "mkdirp": "^1.0.4",
     "ncp": "^2.0.0",
     "node-fetch": "^2.6.7",
+    "resemblejs": "5.0.0",
     "rimraf": "3.0.2"
   },
   "devDependencies": {
@@ -22,6 +24,7 @@
     "@types/ncp": "2.0.1",
     "@types/node": "20.x",
     "@types/node-fetch": "^2.5.10",
+    "@types/resemblejs": "4.1.3",
     "@types/rimraf": "3.0.2",
     "npm-run-all": "^4.1.5",
     "watch": "^1.0.2"

test/smoke/src/areas/positron/plots/plots.test.ts

@@ -5,14 +5,40 @@
 
 
 import { expect } from '@playwright/test';
+import * as path from 'path';
 import { Application, Logger, PositronPythonFixtures, PositronRFixtures } from '../../../../../automation';
 import { installAllHandlers } from '../../../utils';
-
+import { readFileSync } from 'fs';
+import compareImages = require('resemblejs/compareImages');
+import { ComparisonOptions } from 'resemblejs';
+import * as fs from 'fs';
+import { fail } from 'assert';
 
 /*
  * Plots test cases
  */
 export function setup(logger: Logger) {
+
+	const diffPlotsPath = ['..', '..', '.build', 'logs', 'smoke-tests-electron'];
+
+	const options: ComparisonOptions = {
+		output: {
+			errorColor: {
+				red: 255,
+				green: 0,
+				blue: 255
+			},
+			errorType: 'movement',
+			transparency: 0.3,
+			largeImageThreshold: 1200,
+			useCrossOrigin: false
+		},
+		scaleToSameSize: true,
+		ignore: 'antialiasing',
+	};
+
+	const githubActions = process.env.GITHUB_ACTIONS === "true";
+
 	describe('Plots', () => {
 
 		// Shared before/after handling
@@ -57,6 +83,20 @@ plt.show()`;
 
 				await app.workbench.positronPlots.waitForCurrentPlot();
 
+				// capture master image in CI
+				// await app.code.driver.getLocator('.plot-instance .image-wrapper img').screenshot({ path: path.join(...diffPlotsPath, 'pythonScatterplot.png') });
+
+				const buffer = await app.workbench.positronPlots.getCurrentPlotAsBuffer();
+
+				const data = await compareImages(readFileSync(path.join('plots', 'pythonScatterplot.png'), ), buffer, options);
+
+				if (githubActions && data.rawMisMatchPercentage > 2.0) {
+					if (data.getBuffer) {
+						fs.writeFileSync(path.join(...diffPlotsPath, 'pythonScatterplotDiff.png'), data.getBuffer(true));
+					}
+					fail(`Image comparison failed with mismatch percentage: ${data.rawMisMatchPercentage}`);
+				}
+
 				await app.workbench.positronPlots.clearPlots();
 
 				await app.workbench.positronPlots.waitForNoPlots();
@@ -86,6 +126,20 @@ IPython.display.display_png(h)`;
 
 				await app.workbench.positronPlots.waitForCurrentStaticPlot();
 
+				// capture master image in CI
+				// await app.code.driver.getLocator('.plot-instance.static-plot-instance img').screenshot({ path: path.join(...diffPlotsPath, 'graphviz.png') });
+
+				const buffer = await app.workbench.positronPlots.getCurrentStaticPlotAsBuffer();
+
+				const data = await compareImages(readFileSync(path.join('plots', 'graphviz.png'), ), buffer, options);
+
+				if (githubActions && data.rawMisMatchPercentage > 2.0) {
+					if (data.getBuffer) {
+						fs.writeFileSync(path.join(...diffPlotsPath, 'graphvizDiff.png'), data.getBuffer(true));
+					}
+					fail(`Image comparison failed with mismatch percentage: ${data.rawMisMatchPercentage}`);
+				}
+
 				await app.workbench.positronPlots.clearPlots();
 
 				await app.workbench.positronPlots.waitForNoPlots();
@@ -205,6 +259,20 @@ title(main="Autos", col.main="red", font.main=4)`;
 
 				await app.workbench.positronPlots.waitForCurrentPlot();
 
+				// capture master image in CI
+				// await app.code.driver.getLocator('.plot-instance .image-wrapper img').screenshot({ path: path.join(...diffPlotsPath, 'autos.png') });
+
+				const buffer = await app.workbench.positronPlots.getCurrentPlotAsBuffer();
+
+				const data = await compareImages(readFileSync(path.join('plots', 'autos.png'), ), buffer, options);
+
+				if (githubActions && data.rawMisMatchPercentage > 2.0) {
+					if (data.getBuffer) {
+						fs.writeFileSync(path.join(...diffPlotsPath, 'autosDiff.png'), data.getBuffer(true));
+					}
+					fail(`Image comparison failed with mismatch percentage: ${data.rawMisMatchPercentage}`);
+				}
+
 				await app.workbench.positronPlots.clearPlots();
 
 				await app.workbench.positronPlots.waitForNoPlots();