Quantization: Error-Free Modeling

[teleprint-me]

I think I cracked error-free quantization of single samples in isolation.

This allows for quantization of data in real-time, controlled granularity through flexible step sizes, and near-zero error rates with values that are within the domain.

---

Error-Free-Modeling

---

// Quantization structure
typedef struct Quant {
    float scalar; /**< Scaling factor for quantization of input */
    uint8_t bits; /**< Quantized value */
} Quant;

typedef Quant Q8;

// 8-bit integer quantization
Q8 quantize_q8(float value) {
    Q8 q8;

    // Set the range for the integer domain [-128, 127]
    int z_domain = 255;

    // Calculate the dynamic range for the real domain [-value, value]
    float r_domain = fabsf(value); // reflect and compound input

    // if value is greater than z_max or value is less than z_min, otherwise 1
    float squeeze = 1;
    if (r_domain > z_domain) {
        squeeze = z_domain / r_domain; // ratio of the effective range
    }

    // Calculate the scaling factor and preserve the sign
    q8.scalar = (squeeze * r_domain) / z_domain;
    q8.scalar = (value >= 0) ? q8.scalar : -q8.scalar;

    // Quantize the value
    q8.bits = (uint8_t) roundf(value / q8.scalar);

    return q8;
}

// Reconstruct the real value using the scalar
float dequantize_q8(Q8 q8) {
    return (float) (q8.scalar * q8.bits); // / q8.alpha;
}

// Function to sample floating-point values in the range [-n, n-1]
void sampler(double* x, int length, int range) {
    assert(x != NULL);
    assert(length > 0);
    assert(range > 1);

    for (int i = 0; i < length; ++i) {
        double normalized = (double) rand() / (double) RAND_MAX;
        x[i] = -(range + 1) + (normalized * ((2 * range) - 1));
    }
}

// Compute absolute and relative errors
void error(double x, double x_prime, double* abs_error, double* rel_error) {
    *abs_error = fabs(x - x_prime);
    *rel_error = (fabs(x) > 1e-6) ? (*abs_error / fabs(x)) : 0.0;
}

---

03:58:28 | /mnt/valerie/private/alt
(.venv) git:(main | Δ) λ cmake --build build --config Debug -j $(nproc)
[ 20%] Built target alt
[ 80%] Built target mlp
[ 80%] Built target random
[ 80%] Built target data_types
[ 80%] Building C object examples/CMakeFiles/q8.dir/q8.c.o
[ 80%] Built target qk8
[ 93%] Built target compare_float
[100%] Linking C executable q8
[100%] Built target q8
03:58:29 | /mnt/valerie/private/alt
(.venv) git:(main | Δ) λ ./build/examples/q8
=== Running tests with seed: 1337 ===

=== 8-bit Signed Test (Range: [-128, 127]) ===
Input: -93.526154, Quant: 255, Prime: -93.526154
Absolute Error: 0.000001, Relative Error: 0.00%

Input: 65.081794, Quant: 255, Prime: 65.081795
Absolute Error: 0.000001, Relative Error: 0.00%

Input: -97.874577, Quant: 255, Prime: -97.874580
Absolute Error: 0.000003, Relative Error: 0.00%

Input: -10.680528, Quant: 255, Prime: -10.680529
Absolute Error: 0.000000, Relative Error: 0.00%

Input: -58.695369, Quant: 255, Prime: -58.695370
Absolute Error: 0.000001, Relative Error: 0.00%

Input: 53.465564, Quant: 255, Prime: 53.465565
Absolute Error: 0.000000, Relative Error: 0.00%

Input: -87.541037, Quant: 255, Prime: -87.541039
Absolute Error: 0.000002, Relative Error: 0.00%

Input: -21.535200, Quant: 255, Prime: -21.535200
Absolute Error: 0.000000, Relative Error: 0.00%

Input: -22.142080, Quant: 255, Prime: -22.142080
Absolute Error: 0.000001, Relative Error: 0.00%

Input: 43.950995, Quant: 255, Prime: 43.950996
Absolute Error: 0.000002, Relative Error: 0.00%

=== 8-bit Unsigned Test (Range: [-256, 255]) ===
Input: -190.351824, Quant: 255, Prime: -190.351822
Absolute Error: 0.000002, Relative Error: 0.00%

Input: 220.212923, Quant: 255, Prime: 220.212921
Absolute Error: 0.000002, Relative Error: 0.00%

Input: -243.350204, Quant: 255, Prime: -243.350204
Absolute Error: 0.000001, Relative Error: 0.00%

Input: 97.262971, Quant: 255, Prime: 97.262970
Absolute Error: 0.000001, Relative Error: 0.00%

Input: -125.004597, Quant: 255, Prime: -125.004601
Absolute Error: 0.000004, Relative Error: 0.00%

Input: -250.875754, Quant: 255, Prime: -250.875748
Absolute Error: 0.000006, Relative Error: 0.00%

Input: 146.897301, Quant: 255, Prime: 146.897308
Absolute Error: 0.000007, Relative Error: 0.00%

Input: -98.576232, Quant: 255, Prime: -98.576233
Absolute Error: 0.000001, Relative Error: 0.00%

Input: -68.553835, Quant: 255, Prime: -68.553833
Absolute Error: 0.000002, Relative Error: 0.00%

Input: 193.520184, Quant: 255, Prime: 193.520187
Absolute Error: 0.000003, Relative Error: 0.00%

=== 16-bit Signed Test (Range: [-32536, 32535]) ===
Input: 15071.229040, Quant: 223, Prime: 222.999985
Absolute Error: 14848.229055, Relative Error: 98.52%

Input: -10267.707150, Quant: 28, Prime: -28.000000
Absolute Error: 10239.707150, Relative Error: 99.73%

Input: -20218.884026, Quant: 251, Prime: -251.000000
Absolute Error: 19967.884026, Relative Error: 98.76%

Input: 11123.455052, Quant: 115, Prime: 115.000000
Absolute Error: 11008.455052, Relative Error: 98.97%

Input: -25925.801902, Quant: 70, Prime: -70.000000
Absolute Error: 25855.801902, Relative Error: 99.73%

Input: -7613.772435, Quant: 190, Prime: -190.000000
Absolute Error: 7423.772435, Relative Error: 97.50%

Input: -4239.673874, Quant: 144, Prime: -144.000000
Absolute Error: 4095.673874, Relative Error: 96.60%

Input: -29251.144851, Quant: 67, Prime: -67.000000
Absolute Error: 29184.144851, Relative Error: 99.77%

Input: 18200.443258, Quant: 24, Prime: 24.000000
Absolute Error: 18176.443258, Relative Error: 99.87%

Input: -27935.016269, Quant: 31, Prime: -31.000000
Absolute Error: 27904.016269, Relative Error: 99.89%

Average Absolute Error: 16870.412791
Average Relative Error: 98.93%
=== Completed 3 test cases ===

Since the values are scaled using an $\alpha$ factor to "squeeze" the values into the integer domain, there's no need for clipping.

The cost of not introducing the $\alpha$ value in the reconstruction phase is that there is no way to restore the original value which ends up causing massive error spikes in the results.

This is expected as they would normally be completely clipped and instead retain some form a semantic meaning instead of generating noise as a result.

I'm ready to be roasted. Please be gentle. I mean well.