Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Implement end-to-end testing with Playwright #420

Merged
merged 45 commits into from
Oct 10, 2024

Commits on Aug 28, 2024

  1. test: Implement end-to-end testing with Playwright

      This is an in-progress cleaned version of the final PR (numberscope#361) of
      the Delft student user interface project, which it supersedes.
    
      Comments from the original PR that remain relevant:
      * The end-to-end tests run using Firefox and Chromium.
      * New tests are in the e2e folder.
      * The tests often depend on specific classes and IDs, so they may
        need to be updated upon changes to Numberscope.
      * The tests can be executed as the following npm script:
        `npm run test:e2e`
      * An interactive testing UI and debugger can be executed as the
        following npm script: `npm run test:e2e:ui`
    
      Caveats concerning trying this cleaned PR and its status:
      * Make certain to run `npm install` after pulling this PR.
      * Many of the tests do not yet pass, perhaps because of the "specific
        classes and IDs" point mentioned above and the fact that ui2 has
        diverged significantly from the Delft PR series.
      * Tests are not yet run automatically prior to commit.
      * I do not think there are any image tests yet, we need to try to add
        them.
      * Tests are not yet performed in the continuous integration checks
        to be run on GitHub; they should be.
    gwhitney committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    1b6aea7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1249a46 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fd32bf9 View commit details
    Browse the repository at this point in the history
  4. fix: Repair scope tests.

    gwhitney committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    3b81741 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    44869dc View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. test: Add image tests to e2e

      The strong desire for image tests led to a cascade of changes in this
      commit, mostly driven by the need to have reproducible images:
    
      - Removes all use of `sketch.noLoop()` and `sketch.loop()` in favor
        of the previously existing `stop()` and `continue()` visualizer methods,
        to allow:
      - Adds a `frames=NNN` query parameter to URLs to set the maximum number
        of frames a visualization may draw
      - Switches from the "static" instance of mathjs to a "dynamic" one, to
        allow its random number configuration to be controlled. In conjunction
        with this, moves all math functions into a single math module, as
        extensions of mathjs.
      - Removes all use of `Math.random()` in favor of the mathjs random
        generator
      - Adds a `randomSeed=AAAA` query parameter to URLs to make the mathjs
        random generator reproducible.
      - Documents all of the above changes.
    gwhitney committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    b98114a View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. doc: describe running and creating code tests

      Also adds new tests for `src/shared/defineFeatured.ts` and corrects
      the documentation extraction facility for the package manager
      scripts.
    
      Resolves numberscope#25.
      Resolves numberscope#73.
      Resolves numberscope#246.
    gwhitney committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    11a1def View commit details
    Browse the repository at this point in the history
  2. test: Test that the caching mechanism won't double-calculate

       In other words, it should never call calculate twice for the
       same index. This is tested by 10K random accesses to indices
       less than 1M, followed by accessing the first 10K entries,
       followed by accessing the last 10K entries. Hopefully that
       should suffice.
    
       Resolves numberscope#54.
    gwhitney committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    c988513 View commit details
    Browse the repository at this point in the history
  3. fix: Prevent ModFill from hanging on extremely large input

      This is an initial pass at addressing numberscope#113.
      Note, however, that ModFill is not reporting to the person doing
      visualization that it is running with different parameter values
      than shown. So that still must be done, but for that part we will need
      a resolution to numberscope#112, which will be a sufficiently involve change that
      we should leave it to a spearate PR from this numberscope#420.
    gwhitney committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    de5f16c View commit details
    Browse the repository at this point in the history
  4. doc: Update PR checklist.

      Resolves numberscope#174.
    gwhitney committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    3ca1474 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    d8ccce2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    5cccf8c View commit details
    Browse the repository at this point in the history
  7. maintenance: Run typecheck in CI

      Resolves numberscope#292.
    gwhitney committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    571390a View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    3d1d6fa View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. test: Add OEIS Sequence-Visualizer test grid

      Use a list of "stressful sequences" to perform at least two image tests on
      each visualizer. As this change uncovered several existing errors, there
      are numerous other changes in this commit to return to a state of all tests
      passing. Here is an enumeration of other changes, in no particular order:
    
      * Change husky pre-commit actions so that the end-to-end tests are not
        run if there is an existing successful test since any files in the project
        last changed.
      * On most image tests, snapshot only the visualizer canvas.
      * Make 'axios' a runtime dependency, rather than just development, as it
        is used when obtaining OEIS sequence values in the running frontend.
      * So many other changes are made to Histogram.ts that this PR also adds
        hatching to show the elements with unknown factorization. Adds the
        "p5.brush" package to draw the hatching. Since this necessitates WebGL,
        adds a new 'P5GLVisualizer' base class for visualizers that wish to use
        a WebGL canvas.
      * Updates several dependencies to latest to make sure their out-of-dateness
        was not contributing to test issues.
      * Switches to the (ES) "module" import system from "commonjs" to ensure
        that was not contributing to test issues.
    
    Resolves numberscope#294.
    
      * Specifies font locations in a way that vite understands how to
        relocate in both the 'dev' and 'build' versions of the app.
      * Properly marks both factor cache and value cache as empty at the
        initialization of an OEIS sequence.
      * Allows ±Infinity as valid values of INTEGER param fields for convenience.
      * Allows specimenQuery to take the output of parseSpecimenQuery to
        recreate the same specimen query as was parsed, for the sake of testing.
      * Turns off the browser default context menu on visualizers, which wasn't
        particularly useful and was interfering with the UI for some visualizers.
      * Fixes typos in Differences visualizer (this.first -> sequence.first)
      * Makes FactorHistogram visualizer a p5 WebGL-based visualizer, and adds
        hatching to the "0 factors" bar to indicate the terms with unknown
        factorizations.
      * Uses the WebGL default controls to implement panning, zooming, and (rather
        uselessly but somewhat spectacularly) rotating the plot in three
        dimensions for the FactorHistogram visualizer.
      * On FactorHistogram's first pass over the sequence data, collects the
        counts for each possible number of factors, as opposed to the number
        of factors for each entry, to avoid a possibly disastrously long loop
        on the second pass that accumulates bin counts. This also streamlines
        the computation of the largest number of factors in the data.
      * Factors out repeated code in FactorHistogram, for labeling bars and
        displaying text.
      * Fits the hover box in FactorHistogram to the text to be displayed.
      * FactorHistogram displays a temporary message if factoring is taking a
        long time.
      * Moves the bar labels of FactorHistogram just under their respective bars,
        to make sure they are visible.
      * Selects y-axis ticks at round numbers for FactorHistogram, and moves the
        tick labels closer to their ticks.
      * Prevents FactorHistogram from looping except when there is mouse activity.
      * Prevents ModFill from freezing up if too large a "mod dimension" is
        chosen. Note TODO: display a warning when a smaller mod dimension than
        requested is actually used.
      * Prevents Vue's reactivity system from attempting to modify the behavior
        of p5 sketches, using the Vue `markRaw` method. This change prevents
        some instances of infinite loops caused by cascading change notifications.
      * Tightens up the typing of P5Visualizer so that it is possible to derive
        another visualizer base class P5GLVisualizer from it. Also splits up the
        inhabit method so that P5GLVisualizer can modify it as needed.
      * Allow negative start indices for Show Factors visualizer.
      * Show at most 100 terms in Show Factors (no room on screen for more than
        that).
      * Updates TypeScript target versions of JavaScript
    
    Resolves numberscope#226.
    gwhitney committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    044a054 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f1f7d30 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Configuration menu
    Copy the full SHA
    bb45388 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    95d8f57 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. feat: Uniform sequence bounds controls

      All params for controlling which terms of a sequence will be used in
      the visualization are removed from individual visualizers. Instead, there
      are uniform params in the Sequence classes themselves.
    
      Resolves numberscope#411.
    
      In this implementation, several other changes are necessary and/or
      expedient to make. In particular, the type of sequence indices is changed
      to bigint and bounds to ExtendedBigint (the union of bigint and ±Infinity),
      in preparation for addressing numberscope#455. Additional changes include:
    
      * Removal of SequenceDefault class, as everything now derives from Cached.
      * More care in caching of OEIS sequences, to provide some help with numberscope#459.
      * Individual-parameter validation functions now take a validation status
        to update based on the finding. This refactor avoids a frequent need to
        merge status objects.
      * Individual-parameter validation functions are called in a context
        of `this` set to the Paramable object, so that other data from the
        paramable can be accessed.
      * New `math` functions for dealing with ExtendedBigints.
      * NumberGlyph visualizer now tries to display as many terms as are
        available and will fit on the screen, except in case there are infinitely
        many terms available, in which case it still defaults to 64 terms. This
        change is needed so that the general length parameter for Sequences would
        affect what NumberGlyph displayed.
      * Puts reactivation of "known" OEIS sequences into a function
        rather than at top level of sequences.ts so that the timing of
        when it occurs can be controlled.
      * Renames property `_size` of P5Visualizer to `size` as multiple derived
        classes seem to need this value (i.e., not just used internally in the base
        class).
      * Makes sure the p5 loop is restarted if need be every time `.show()` is
        called on a visualizer.
      * Temporarily disables A000521 transversal tests until we can
        get incremental OEIS loading to work.
      * Adds a test for starting a visualization deep into a sequence.
    gwhitney committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d811eba View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Configuration menu
    Copy the full SHA
    33386c0 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. test: First working e2e tests with text and transversal

      Unfortunately, due to the intricacies of end-to-end testing with image
      comparison, this is a very large commit. It introduces running tests
      in a Docker container for the sake of reproducibility. (So note that
      henceforth you must have docker installed and running on your machine to
      perform testing, and hence to make a commit.)
    
      However, it turns out not all tests can be run in a Docker container --
      Firefox is not able to supply a WebGL context inside of Docker. Hence,
      some tests still need to be run directly on the host machine.
    
      To deal with all this, I felt it necessary to introduce the "make" tool.
      In particular, creating the necessary Docker image is slow, and I didn't
      want to have to repeat that except when truly necessary.
    
      That, in turn, means there are now many more configuration files, as
      well as auxiliary files created by the Makefile. The directory tree was
      becoming hopelessly cluttered. To keep things straight in my head, I felt
      it was really necessary to reorganize the directory structure. The biggest
      change is to put as many of the configuration files as possible in a new
      etc/ directory (the old-school unix name for where to put such things).
      I managed to get almost everything put in there. The one major holdout is
      the tsconfig files; I just couldn't find a way to get TypeScript to run
      without its config files at the top level of the project. Slightly
      annoying, but then, there are so many more annoying things about TypeScript.
    
      Anyhow, accomplishing the configuration file rearrangement resulted in
      updating eslint. Unfortunately, it went through a _major_ rewrite from
      version 8 to 9, and prettier-eslint has not been updated to work with
      the new version. I tried to work on that update myself, but prettier-eslint
      is incredibly intricate in the way that it handles configurations because
      it is trying to provide a great deal of flexibility and power, and it is
      trying to infer prettier configuration from eslint configuration (and maybe
      even vice-versa as well, I'm not sure). Since we were only using a tiny
      fraction of that power, it turned out to be significantly easier to just
      replace prettier-eslint with a custom script tools/prettiest.js that runs
      first prettier, then eslint. It also meant we could get rid of the
      lint-staged tool, so there is that. (While I was at it, I ran the "depcheck"
      tool and got rid of some other unneeded cruft that had accumulated.)
    
      But now it all works, and as a side benefit since just about every npm
      script now runs through make, you should never need to worry about
      running install before dev or build before preview, etc. Make keeps track
      of what depends on what and whether/how to redo the prerequisites before
      taking the requested action. Even better, if you just ran successful
      end-to-end tests, which take a while now, then `git commit` will not have
      to re-run them.
    
      Those are the major points. In addition, there are a number of minor changes
      resulting from errors uncovered in getting all the tests to work, etc:
    
      - Reordering of element attributes and other reformatting in vue components,
        because with the lint updates and reconfiguration, it now seems
        significantly stricter about vue templates.
      - First pass at an updated User Guide for ui2; it definitely needs more work.
      - Some other more minor documentation updates.
      - Updates tools/editor/autoformat.el so that emacs can use the new
        "prettiest" tool for formatting (even though I know I am likely the only
        one in the world who will ever use that).
    gwhitney committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    63ddc6c View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. test: Add skipped tests reflecting known shortcomings

      As per Numberscope discussion today, this is the last missing element
      of the end-to-end testing PR. With these skipped tests, we are flagging
      that there are known concerns we want to resolve at least by beta, if not
      alpha. Also fixes a small gap in the docs for running from source that
      Aaron noticed in the meeting.
    
      With this, the PR will be marked ready for final review.
    gwhitney committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    bca3bf4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    22d77fb View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b5cfc4a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8f3f58e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c02cfd8 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    4a212e3 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    5214d32 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    acb49bb View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    68d9294 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    ff375c7 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    345a72b View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Configuration menu
    Copy the full SHA
    599662b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7374837 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Configuration menu
    Copy the full SHA
    be2a7d5 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Configuration menu
    Copy the full SHA
    7567594 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    14b24f6 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Configuration menu
    Copy the full SHA
    31bbe08 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    94669e5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f72c78c View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Configuration menu
    Copy the full SHA
    fbdaedc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    feb8084 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1fb03d2 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Configuration menu
    Copy the full SHA
    4c78deb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4322aa7 View commit details
    Browse the repository at this point in the history