Migrate Harbor UI E2E tests from Robot + Selenium to Playwright

[bupd]

Tracking Issue: goharbor/harbor#22134

[MinerYang]

Hi @bupd ,

We may need more thoroughly mindful investigation on the harbor whole robot-case code base, especially on the UI part.
Generally , robot framework has more maturely ecosystem and plentiful extensions that utilized or could be utilized in the future cases, however playwright may has more narrower usability-scope. The key criteria is we need a Cross-environment, Containerized, Cli-keyworkd-driven orchestration with Unified traceable report for the whole test framework instead of just simply running on the git-actions.
If you go through more details on our current robot cases and running manually by yourself, you could easily get that there's some limitations that current robot case could handle easily however playwrite does not.

• Cross-environment orchestration (shell, processes, SSH)
  Robot suites invoke host/container CLIs heavily via OperatingSystem and Process keywords (docker/helm/imgpkg/cosign/notation/oras/singularity), and remote ops via SSHLibrary. This makes complex end-to-end, infra-level flows first-class.
  In Playwright, you’d have to re-implement these as custom Node/Python scripts (subprocess/child_process, ssh2/paramiko) with your own retry, logging, and reporting around them which would not be a best practice. especially in a graduate project.

    *** Settings ***
    Library  OperatingSystem
    Library  String
    Library  Collections
    Library  requests
    Library  Process
    Library  SSHLibrary  1 minute
    Library  DateTime
    Library  SeleniumLibrary  60  10
    Library  JSONLibrary
    Resource  Harbor-Util.robot
   ...

• Keyword-driven composition, readability, and reuse across infrastructure and UI
  Robot’s .robot resources layer UI pages, infra utilities, and APIs into readily composable keywords, with built-in retries and waits in keywords. Plus, keywords driven would also benefit the case controlling across environment.
  Playwright Test is excellent for UI, but mixing in infra operations typically results in imperative code; step-level reporting for those custom flows requires additional work.

• Unified HTML logs/reporting for UI + infra steps
  Robot provides rich keyword-level HTML logs with screenshots and nested steps for both browser and non-browser actions, out of the box.
  Playwright’s HTML reporter is strong for UI steps but you must explicitly annotate custom infra steps to reach parity, and you’ll still have to stitch multi-tool outputs and artifacts yourself.

• Containerization: what works smoothly with Robot vs Playwright
  *Robot is easily to achieve cmd in a single image with infra CLIs
  The current suite expects system-level tools and paths (/usr/bin/google-chrome, /go/selenium-server-standalone-3.4.0.jar, /drone/download), making it natural to build a purpose image that bundles:
  Browsers + chromedriver + Selenium JAR
  CLI tools: docker, helm, imgpkg, cosign, notation, oras, singularity, curl, jq, etc.
  Python libs: Robot Framework, SeleniumLibrary, requests, JSONLibrary, SSHLibrary, etc.
  Running via CLI is trivial: robot [options] tests/robot-cases/....
  Playwright image with infra CLIs is possible but heavier and trickier
  The official Playwright images bundle browsers; adding all the additional CLIs and enabling docker/helm interactions typically requires:
  Larger image and maintenance for many native deps.
  Privileged mounts (e.g., -v /var/run/docker.sock:/var/run/docker.sock) or DinD; plus credentials, kube config, etc.
  Re-implementing SSH, process orchestration, and retries in code or utilities.
  It’s doable, but you’ll lose the turnkey keyword orchestration and reporting that Robot already provides for non-UI tasks.

[bupd]

@MinerYang

The key criteria is we need a Cross-environment, Containerized, Cli-keyworkd-driven orchestration with Unified traceable report for the whole test framework instead of just simply running on the git-actions.

I don't get this. Why do we need a cross environment - what cross environment mean here ?? Please explain.

Playwright image with infra CLIs is possible but heavier and trickier
The official Playwright images bundle browsers; adding all the additional CLIs and enabling docker/helm interactions typically requires:
Larger image and maintenance for many native deps.
Privileged mounts (e.g., -v /var/run/docker.sock:/var/run/docker.sock) or DinD; plus credentials, kube config, etc.
Re-implementing SSH, process orchestration, and retries in code or utilities.
It’s doable, but you’ll lose the turnkey keyword orchestration and reporting that Robot already provides for non-UI tasks.

This makes no sense. The goal is clearly mentioned here https://github.com/goharbor/community/pull/268/files#diff-e9c149ab1ab59f390feca2254d8ccd8c76643c1016b5e6651c7dcd712341ada2R26
Replace UI/XPath-based Robot tests with Playwright.

We are not removing Robot from areas where it might be the right tool. I am sure that for UI tests robot is a inadequate tool.

We are eliminating XPath-dependent UI flakiness by shifting only those tests to Playwright.

[MinerYang]

Hi @bupd ,

Generally, One of our key acceptance standards is do not decrease the current quality which would reflects in these criteria that you need to take it consideration:

• Cross-environment and containerization
  We need an orchestration that needs convenient to be portable, which allows us to be easily imported as a tool such as a container that could be utilized in different environments like Jenkins, any shell, or CICD env for different network topology (like air-gaped) rather than just running on a fixed public git action CI. And would the containerized playwright image become more heavier? AFAK Playwright is not optimized for heavy system orchestration.

• Keyword-driven infra
  We need an keyword-driven infrastructure that each of the case is relatively dependent and plugable, which allow tester to define which test-case need to running in different to be skip or running easily by a cli keyword.

• Unified HTML logs/reporting
  We need an unified, readable and easy-tracking log/report for the test process. You could compare the current harbor robot case log and investigate more could the playwright generate a equivalent trace report and how to achieve so.

• System process and tools maintenance
  If your dig deeper into the current harbor robot case, you would see some of cases heavily rely on the system process (docker.daemon), system libraries(SSHLibraries, JSONLibrary etc. ), apt tools(curl) or third-party tools like helm, cosign, notation. Here are some race conditions that I know playwright may not handle well. e.g. If there's any gap like SSHLibraries hasn't well-inplemented in the playwright ecosystem, how do you plan to handle situations? Or running CLI tools like docker push or cosign verify are network and I/O heavy, when we have this along with our test-flow, would the performance be affect if when we are using execAsync...
  RobotFramework can manage resources explicitly via setup/teardown keywords and runs tasks sequentially by default. However Playwright parallel test workers may all try to start dockerd at the same time which leads to failures, hangs like port conflicts socket block or unpredictable error.

• Other thoughts
  Besides some of our current case that needs key or credential which can’t run safely in sandboxed CI environments (like GitHub Actions or GitLab runners) which would leads to accidentally leaks credentials or keys in logs.

Could you please help to demonstrate in your proposal how to achieve above goals or solutions for the potential issues?

Best,
Miner

[reasonerjt]

@Vad1mo @OrlinVasilev

I believe there're still disagreements about this proposal. Miner has expressed a lot of concerns in last week's community meeting.

We should make sure the comments are resolved before this proposal can be merged.

[MinerYang]

Hi @bupd
Please help to update the proposal per this comment and the discussion we had last month.
#268 (comment)

Best,
Miner

[bupd]

@Vad1mo @OrlinVasilev

I believe there're still disagreements about this proposal. Miner has expressed a lot of concerns in last week's community meeting.

We should make sure the comments are resolved before this proposal can be merged.

@reasonerjt I would like to clarify that the concerns have been addressed in the previous community meeting. I was in OSS summit for past week. currently getting back on this.

Only thing pending on this is to update the proposal with screenshots of reports and add more info on the running this containerised, environments. as we discussed earlier

I am currently recovering will update the proposal shortly.

Thanks for understanding

[bupd]

@MinerYang @stonezdj @reasonerjt All concerns have been addressed.

[MinerYang]

For the whole containerized process, I 'believe that you've had a PoC of all these criteria. Could you please help to adding more details , like:

• How''s the ready-to-use Dockerfile that build for our play-wright e2e-UI engine would looks like? ref to https://github.com/goharbor/harbor/blob/main/tests/test-engine-image/Dockerfile.ui_test
• How' s the running cmd should looks like.
• Does play-wright support key-word driven execution? How's the cmd should looks like?

e.g.

E2E_IMAGE=goharbor/harbor-e2e-engine:play-wright-ui
# Run tests by title
docker run -i -v <....>  -e <....>  -w <...> $E2E_IMAGE npx playwright test -g "add a todo item"

[bupd]

• https://github.com/bupd/harbor/blob/87b9f9722d3113653aad792480a17fc11731d4f0/src/portal/e2e/Dockerfile this is the POC dockerfile

• playwright do support key-word driven execution like robot-framework. They have an extensive official documentation around this here - https://playwright.dev/docs/running-tests#run-specific-tests

[bupd]

i will update the proposal with details of these. also give a simple example

[MinerYang]

Thanks, And all these details would be better to presents in the proposal.

[MinerYang]

I would like better to have a full-chain build and running process demo for several specific cases, and presents the demo in the proposal as a Poc.

• Test Case - Cosign And Cosign Deployment Security Policy
• Verfiy System Level CVE Allowlist

These cases demo should have presents these criteria I have mentioned before.

[bupd]

• feat: Migrate Trivy.robot(Security Hub Testcase) to playwright harbor#22591 pr covering bigger testcase with securityhub interacting with harbor UI, trivy etc. this PR also showcases full build and test workflow.

[bupd]

@MinerYang @stonezdj @reasonerjt
I have updated the proposal to now include POC Dockerfile (Ubuntu 20.04-based, matching Robot's structure with same permissions and all tools: docker, helm, cosign, notation, chrome, etc., just with Playwright instead of Robot Framework), demonstrates keyword-driven execution, superior reporting with screenshots, and containerization for Jenkins/GitLab/air-gapped environments while scoping to UI tests only.

also added relevant screenshots of test reports, debugging, networking capabilities. please take a look.

Thanks