Questions about Goals and Future Direction for Matrix Multiplication

[epsilonError]

Hello Everyone,

I worked on some tests and type fixes for multiplyMatrices.js and wanted to share some edge cases and ask about future implementation goals and priorities.

I previously mentioned I was going to open an issue to outline what I found while working on tests and types, but there's enough to talk about that I thought it would be better to try opening a discussion instead.

After some consensus is reached, I'll breakdown the decisions into individual issues and more focused pull requests.

Reference Implementation

I'm testing possible implementation details and running tests with an adaptation of color.js's multiplyMatrices.js in a private typescript project. The typescript code is available in a github gist (hover to see when last updated).

I already have testing and code coverage setup for that project so it's easier for me to play around with things there. And I plan on porting the relevant tests to hTest and color.js in future.

The private project also includes tests for simulating prota-/deutera-/trita-nopia on sRGB colors using color.js and compares reference results from DaltonLens-Python. So I've got quite a lot of tests already. (After making the simulation work with non-sRGB colors I plan on open sourcing it, but that is a long way off and not part of this discussion.)

Observations

Priority

What I gleaned from a previous version of the API docs is that multiplyMatrices.js isn't focused on ensuring the correctness of every arbitrary matrix product. The priority seemed to be 1) correctness for common use cases, and 2) performance to reduce the overhead of the frequent transformation of coordinates from one Color Space to another.

Arguments & Return Types

In color.js (as of 2024-08-10) there are 25 non-test uses of multiplyMatrices(), roughly the types work out to be:

Assigned/Result Type	A	B	Returns
[number number number]		11	7
[number number number][]
number[]		10	16
number[][]	25	2	2
any		2

My Guess of Type	Times	Out of
any seems to be 3-Array:	2	2
number[] seems to be 1×3:
number[] seems to be 3×1:	10	26
number[] seems to be 3-Array:	10	26
number[][] seems to be 3×3:	25	29

I did a rough scan of the code to get these numbers, they aren't definitive.

Current multiplyMatrices Implementation Notes

1. Lines 65–69 seem unreachable. At that point, AM is a row-ordered matrix so the row argument of the first map will always be an Array. (Code Coverage never reached this code in my tests.)
2. Line 73 uses (col[i] || 0) to handle possible undefined access of the col variable. This is guaranteed to be an invalid product if the || 0 is used. Another possible choice is using NaN instead of 0.{See comment below.}
3. Since the for loop on line 72 loops until the length of the row in the A matrix, the returned value's shape is always constrained by A. Iterating based on col.length, and using (row[i] || 0), returns the same correct answers and is also a reasonable choice.
4. The general types in the function overloads work well, but it could be worthwhile to add a TransformationMatrix (3×3) type and use it along with a non-null Coords type to specify precise return types.
5. Along with 3 and 4 above, the shape of the return type will always be constrained by either A's rows or B's columns. Which means either A or B is guaranteed to constrain the final returned shape.
6. A vector × vector generally returns a scalar, but multiplyMatrices returns a [number] result.
7. Overly specified vectors arguments are permissible and fulfill the parameter type requirements (i.e., [[]], [[1]], [[1],[2],[3]], etc.)
8. Overly specified vector arguments don't fulfill the return type guarantee. They are all number[][] type but are mathematically equivalent to vectors. Notationally 1 is the same as [1] is the same as [[1]], but they are not the same computationally. Computational equivalence requires properly packing and unpacking these nested arrays.
9. Empty B vectors throw an error (can't call .length on undefined). We could throw our own errors, attempt to return an obviously incorrect value (but type/shape correct) result, or leave as is for performance reasons
10. If B is [] then calling map on it will assign [] to BM, rather than the expected [[]].
11. m is actually assigned the value of n, when A is a number[].

The Future

My Next Steps Recommendation

• Add more tests to account for notes-1,2,3,5,9,10,11
• Add a TransformationMatrix type and use it and/or non-null Coords type to highlight "happy path" usages of multiplyMatrices (ref notes-4,5)

Needs Discussion

• How do we want to pack and/or unpack mathematically equivalent nested arrays? (Some Options: Unpack to simplest equivalent. Pack to match provided arguments return type.) [related to notes-6,7,8]
• How much error checking and handling do we want to do? (Some Checking Options: None, An empty edge-case, All possible empty edge-cases, full dimension checking. Some Handling Options: Throw error for some/each check, return obviously incorrect values (0/NaN) in the proper return shape) [related to notes-9,2,3]
• Since Coords includes null and numbers as acceptable values, should multiplyMatrices accept them as well? (I'm not sure how multiplyMatrices could handle this correctly for all cases. So it's probably better for each Color Space to resolve null values.) [related to notes-4,2,9]

For Now

Thanks for reading and let know what you think. I'll probably focus on adding more test in the near future. But I'm open to hearing other perspectives and recommendations.

[epsilonError]

2. Line 73 uses (col[i] || 0) to handle possible undefined access of the col variable. This is guaranteed to be an invalid product if the || 0 is used. Another possible choice is using NaN instead of 0.{See comment below.}

NaN is not a reasonable choice if || is used for comparison. 0 is a falsy value and so all 0's in a matrix would be turned into NaN. So using NaN wouldn't be a good choice.

[lloydk]

Thanks for the detailed analysis. Looks like some of the types need to be cleaned up.

I was doing some profiling of a benchmark that used Color.js on the weekend and found that multiplyMatrices was the function that contributed the most amount of time to the benchmark. Using a specialized function to transform a [number, number, number] by a 3x3 matrix resulted in a significant performance improvement. Here is the PR with the changes and benchmark timings: #585

Since Coords includes null and numbers as acceptable values, should multiplyMatrices accept them as well? (I'm not sure how multiplyMatrices could handle this correctly for all cases. So it's probably better for each Color Space to resolve null values.)

I agree that null should be resolved to 0 before calling and of the matrix multiplication functions.

[epsilonError]

I was doing some profiling of a benchmark that used Color.js on the weekend and found that multiplyMatrices was the function that contributed the most amount of time to the benchmark. Using a specialized function to transform a [number, number, number] by a 3x3 matrix resulted in a significant performance improvement. Here is the PR with the changes and benchmark timings: #585

That's a good use of the specialized function, and with type-checking a great tradeoff for the performance.

It makes sense to me that multiplyMatrices is a bottleneck, it's generic enough to handle a lot of things and is consistently called whenever a color space is traversed.

As just a nit-pick, I prefer the semantics of multiplyMatrices' parameters over transform's. Even though they are functionally not much different. transform reads to me as transform input by matrix while optionally using some existing vector. While multiplyMatrices hints at a chain of matrices all lined up together and finally a column vector at the end reifies the abstract linear transformations into a specific position. But I'm a sucker for a reduce/scan pattern so I like imagining 𝐀𝐁𝐂𝐃𝐱ᵀ over 𝑓(𝐱, 𝐀𝐁𝐂𝐃) in my head.

I'm happy to keep working on multiplyMatrices if a generic way to do matrix multiplication with scalars, vectors, and/or matrices is needed. But I suspect a specialized function for 3x3 matrices and vectors will always outperform it.

[epsilonError]

I found some benchmarks on matrix math. It only benchmarks element-wise addition of matrices but could be useful in structuring future specialized functions.