Fix test workflow, bis #4252
Fix test workflow, bis #4252
Conversation
If I understand the documentation correctly, it is not strictly necessary that these variables be read from the workflow file; they may also be read directly by the target processes.
This reverts commit 02c3b73.
In Node 6, running Karma gave the following warning: Error during loading "/home/runner/work/backbone/backbone/node_modules/karma-sauce-launcher" plugin: Unexpected token function I had no such error running karma after installing karma-sauce-launcher locally in Node 14, so this upgrade might fix some things. We are not exactly in the business of supporting old Node versions, so this should be relatively painless.
I mostly copied this from the current config in Underscore.
My account currently allows only two concurrent runs, so let's rule out that this is biting us.
I chose this datacenter when creating my account. Maybe it matters.
I tried a couple of different OS version, and this one was the first where Firefox 11 seems to connect correctly at least some of the time. (To clarify: the tests were passing in Firefox 11 on all OS versions I tried, but in most cases a connection error prevented Karma from knowing this. I only know because I saw the reports in the Sauce web interface.)
jgonggrijp
force-pushed
the
patch-1
branch
2 times, most recently
from
8f9af60 to
91b190b
Compare
|
@ogonkov @paulfalgout I got the Sauce Labs tests to run, mostly reliably, with the exception of Firefox 11. See the details of the above (failed) workflow run. If you agree with the logic changes you see here, we can merge this. However, we need to make a decision about the lowest Firefox version. The tests do pass in Firefox 11, as you can see on Sauce Labs: https://app.eu-central-1.saucelabs.com/tests/08d1010d4bea46dda25b29f2a6a9a698. However, something is going wrong in the connection with Karma. I tried this on a couple of OS versions and I had a single attempt on Windows 10 where the connection suddenly worked (run 53). So far, I haven't been able to reproduce this, but this is the reason why the I also tried Firefox 18, 25 and 32, but they all behaved similarly flaky, although versions 18 and 32 eventually succeeded on second attempts. See run 55. I see three possible courses of action:
Please let me know what you think! |
|
Can you run sause labs config with |
|
That was a good suggestion. The visible difference between FF11 and the other browsers was that FF11 was creating additional socket connections and then immediately closing them again. I tried increasing Karma's Haven't been able to find a solution for "Response has empty body" or "Request failed with status 204" yet. Still searching the web. |
|
Some requests (which one?) has been responded with empty body, that was unexpected for client. There also warn about old client. May be it is something related to old client, used by plugin? |
|
Yes, good suggestion. I was just considering to upgrade the |
|
I'm stuck again because Sauce Labs hasn't converted my trial account to Open Sauce yet and my trial has run out of testing minutes. To be continued. |
|
Update: I haven't heard back from Sauce Labs since January 17. I just made a third attempt at reestablishing contact, using a different email address. |
|
Update again: they finally enabled Open Sauce, so we can run browser tests again. I will resume the debugging labour soon. |
.github/workflows/tests.yml
Outdated
| - run: ./node_modules/.bin/karma start karma.conf-sauce.js | ||
| - name: Install dependencies | ||
| run: | | ||
| npm install --no-audit |
There was a problem hiding this comment.
We want npm ci here, since we had package lock
There was a problem hiding this comment.
Thanks. I will have a look at the documentation, because I'm not really familiar with the ci subcommand.
As suggested by @ogonkov in #4252 (comment)
|
Open Sauce doesn't seem to be really enabled. To be continued... |
|
May be we can mark sause labs tests as non blocking for workflow and merge it in master, to fix units in PRs? |
|
Yes that makes sense. Good idea. How do you do that? |
As suggested by @ogonkov in #4252 (comment)
Lower tolerances, more concurrency.
|
@ogonkov Turns out Open Sauce was enabled on the US cluster while my account is on the EU cluster. This has now been fixed. I decided to just disable IE 9, IE 10, Chrome 26 and Chrome 40 for the time being (#4253), as the selection of browsers that remains seems still better than nothing. If you agree with the current state of the code, we can merge this (I already rebased for cleanup). |
|
High five! |
The updated docco puts these annotated sources in a different place. Rather than forcing them back in the old place, it seems easiest to just go with the flow.
The updated docco puts these annotated sources in a different place. Rather than forcing them back in the old place, it seems easiest to just go with the flow.
This is a continuation of @ogonkov's work at #4251, which I'm personally pushing directly to the main repo in order to rule out any effects from permissions. I also wanted to try a couple of things. If I end up merging this branch, #4251 will automatically be recognized as merged as well.