fix: Fix flaky tests in CI #681
Conversation
Some visual regression tests inconsistently fail when running in GitHub Actions. I think this is because the `govuk-frontend-supported` class is added to the `<body>` when JavaScript is enabled. This takes a second or two to initialise, and Playwright thinks the page has stabilised before the class is applied.
| @@ -8,6 +8,10 @@ test.describe('@visual-regression', () => { | |||
| test.describe(exampleName, () => { | |||
| test.beforeEach(async ({ page }) => { | |||
| await page.goto(`./components/${component}/${exampleName}`) | |||
| // The `govuk-frontend-supported` class is added to the `<body>` when JavaScript is enabled. | |||
| // This takes a second or two to initialise, and Playwright thinks the page has stabilised before the class is applied. | |||
There was a problem hiding this comment.
This makes sense. It would explain why this suddenly started happening when updated the code to support v5 of govuk-frontend.
It might not be this class being added but initAll that causes the layout shift, though. Does this only happen on pages with components that have associated JavaScript?
There was a problem hiding this comment.
No - it seems to also happen with components that don't have JavaScript e.g. Tag. That's why I didn't originally think that this was the issue.
There was a problem hiding this comment.
I think you're right - I've tried await expect(page.locator('body')).toHaveClass(/govuk-frontend-supported/) and await page.waitForSelector('body.govuk-frontend-supported') but we're still getting failing tests
Will fix when I rebase |
Some visual regression tests inconsistently fail when running in GitHub Actions. I think this is because the
govuk-frontend-supportedclass is added to the<body>when JavaScript is enabled. This takes a second or two to initialise, and Playwright thinks the page has stabilised before the class is applied.This PR adds a short timeout to visual regression tests between the page loading and the page being compared to the 'golden screenshot' to allow the
govuk-frontend-supportedclass to be applied and the page to stabilise.