[Tracking]: Unified UI Testing 🔮

[ghengeveld]

🧑‍🤝‍🧑 Who: @ghengeveld, @yannbf and @valentinpalkovic

This is a tracking issue the Unified UI Testing 🔮 project. The purpose of this issue is to keep tracking of the overall status of the project and tasks, and plan everything around it.

⚠️ Problem

Storybook has the potential to be the go-to tool for comprehensive UI testing, unifying both the existing Visual Regression Testing (VRT) and Component Testing (CT) experiences. However, the current implementation falls short in two key areas:

1️⃣ Fragmented workflow

As of 8.3, users begin their Storybook CT experience with Vitest via the CLI, then transition into Storybook for visual debugging. More specifically, this process includes:

1. Configuring the storybookScript option in the Vitest plugin.
2. Running vitest in watch mode from the CLI, which spawns Storybook in the background.
3. Viewing test failure outputs in the CLI and clicking a link to open Storybook for visual debugging.

This workflow centers on Vitest, making Storybook feel like a "sidekick" (albeit a powerful one). The constant switching between Vitest and Storybook disrupts the development flow and makes the process feel fragmented.

2️⃣ Incoherent design and under-utilized CT experience

Storybook already provides a powerful VRT experience via Visual Test Addon. We have an opportunity to create a more coherent and optimized experience that fully leverages both VRT and CT capabilities within Storybook.

🏁 Goals

The high-level goals of this project are:

• Introduce a way to trigger test runs from the Storybook Manager UI.
• Support both the Storybook Test addon (Vitest) and the Visual Test addon (Chromatic).
• Support scoping down to groups, components and stories (when supported by the test provider).
• Update (aggregate) story status indicators on groups, components and stories with new icons.

🚩 Milestones

1. Testing module essentials

Beta Give feedback

1. Spec and implement Test Provider API #29092
   +0
   
2. Spec for Channel events to trigger test run #29093
   +0
   
3. Update Storybook Test addon to register with Test Provider API #29094
   +0
   
4. Update Storybook Test addon to listen for test run events #29095
   +0
   
5. Testing module UI scaffold #29096
   +0
   

2. Updating VTA and Interactions addon

Beta Give feedback

1. Update VTA to use Test Provider API #29097
   +0
2. Update VTA onboarding to point at Testing module #29098
   +0
3. Update Interactions to handle discrepancy in test results #29120
   +0

3. Watch mode

Beta Give feedback

1. Support watchable property for test providers
2. Render an "eye" toggle button which emits a _WATCH event
3. Handle watch event in Storybook Test addon

4. Design updates

Beta Give feedback

1. Testing module UI design refinements
2. Group/component/story dropdown menu UI update

5. Run individual stories

Beta Give feedback

1. Spec for Story Actions API #29108
   +0
2. Ellipsis story dropdown menu toggle on hover #29107
   +0
3. Story dropdown menu (Story Actions API) #29101
   +0
4. “Run component tests” action (Channel API)

6. Static builds

Beta Give feedback

1. Disable expandability of the testing module

7. Enhanced story status

Beta Give feedback

1. Add support for a count property (shown on the right in the dropdown)
2. Aggregate (sum) these counts on groups and components

Out of scope

• Integrating with A11y addon
• Support for code coverage reporting