test(release): Add Python script to test UIs #3232
Conversation
There was a problem hiding this comment.
Nice!
What we're looking for is a mix between this and the now-closed #2519: something we can run as part of a pass/fail e2e test that's run with the rest of our CI checks. We'll leave the manual verification in scripts/release.sh in place while we build confidence in the e2e test.
It's fine if you want to also run the headless browser test in scripts/release.sh, but it should be pass/fail, rather than involving manual inspection of the output of a python script. What we're looking for would be cases where the automated test didn't catch something, but the manual verification did.
Thanks for the feedback. So I imagine that I can have something very similar to the closed MR, especially for the YAML configurations, right?
Then maybe it can be added after the manual inspection, just to confirm that new warnings or new errors are not introduced by a new version. |
No need for any equivalent of that e2e-selenium.yml workflow, since we'll be adding one to e2e workflow. |
mmicu
force-pushed
the
test-uis-with-playwright
branch
from
7222215
to
278054b
Compare
There was a problem hiding this comment.
I've let CI run for this PR, and it bugged out in a bit unexpected place: https://github.com/dfinity/sdk/actions/runs/5591534054/jobs/10227915747?pr=3232
Nothing obvious jumps at me why it crashed there, so I've compared .jobs.test.steps and .jobs.ui_test.steps; these were some of the differences I was able to find:
|
CI run is in progress https://github.com/dfinity/sdk/actions/runs/5602950032?pr=3232 |
|
we're almost there :) https://github.com/dfinity/sdk/actions/runs/5602950032/jobs/10249202379 |
|
I found a silly bug in the script. It should solve most of the issues. However, I cannot make |
|
I'm afraid yes, Btw, have you maybe looked into how hard it would be to also include tests on mobile (android chrome & ios safari)? Having those is basically an endgame for us |
In the Frontend UI, the
Good point. Nope, I did not check it yet. |
|
What OS are you running this on? I'll check tomorrow on macOS First thing I'd try to do is try to run launch the browser in headed mode, have you maybe tried that already? |
Ubuntu 22.04 (WSL).
Not yet. Please, try with the following snippet: from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
# Empty value
browser = playwright.webkit.launch(headless=True)
# OK
# browser = playwright.firefox.launch(headless=True)
context = browser.new_context()
page = browser.new_page()
frontend_url = '...'
page.goto(frontend_url)
page.get_by_label('Enter your name:').fill('foo')
page.get_by_role('button', name='Click Me!').click()
section_element = page.query_selector('#greeting')
print(section_element.text_content())
browser.close() |
|
That's why it is not working, which cites Safari keeps forcing HTTPS on localhost. I am still checking how to solve it from Playwright. EDIT: maybe changing the headers in the response could help. It seems that it is supported from version 1.29. I will test it in the evening and let you know. |
|
Yes, looks like you're right... I tried to look into how to generate the certificate but didn't found anything so far Also, when I try to open my locally deployed canister in Safari, the website doesn't load CSS or images. When I try to click on the button, it errors out with |
mmicu
force-pushed
the
test-uis-with-playwright
branch
from
9d36248
to
77a4561
Compare
|
I was able to replicate and fix the issues locally. Not sure if we actually need to parse the warnings and errors, since we still have the manual checks during the release process. |
To follow up on this here: this is happening on Safari due to the security policy headers that we introduced in dfx 0.13.1. |
mmicu
force-pushed
the
test-uis-with-playwright
branch
from
4212f0a
to
f3e7c97
Compare
mmicu
force-pushed
the
test-uis-with-playwright
branch
from
f3e7c97
to
d928060
Compare
mmicu
force-pushed
the
test-uis-with-playwright
branch
from
0a2f2e7
to
27f8687
Compare
mmicu
force-pushed
the
test-uis-with-playwright
branch
from
27f8687
to
01e8d66
Compare
Co-authored-by: Marcin Nowak-Liebiediew <[email protected]>
|
Hi @ericswanson-dfinity, could you please review the PR? The script seems to work now. |
Description
Add a Python script that automate the testing of the
FrontendandCandidUIs by using playwright.How Has This Been Tested?
The script was executed locally. It should produce a similar output.
Moreover, it fails immediately if it does not find the expected fields as well as warnings and errors.
Checklist: