Incorporate major PDB table row fix

Author: brunchboy

CHANGELOG.md

@@ -6,10 +6,15 @@ This change log follows the conventions of
 
 ## [Unreleased][unreleased]
 
-### Changed
+### Fixed
 
-- Reworked the database export Kaitai Struct definition to incorporate some important discoveries by the Mixxx and rekordcrate developers (thanks once again [@Swiftb0y](https://github.com/Swiftb0y)): all tables that use string offsets have subtypes which control whether those offsets are eight or sixteen bits.
+- Reworked the database export Kaitai Struct definition to incorporate some important discoveries by the [Mixxx](https://mixxx.org) and [rekordcrate](https://github.com/Holzhaus/rekordcrate) developers (thanks once again [@Swiftb0y](https://github.com/Swiftb0y)): all tables that use string offsets have subtypes which control whether those offsets are eight or sixteen bits.
   We had previously only noted that for the Artists table, but the Album, Tag, and Tag Track tables behave this way as well, and in fact even the Track table follows this pattern, but its offsets always use the sixteen bit variant because the rows are so big.
+- The understanding of row counts in DeviceSQL data pages has been broken since the very beginning (though we had some clumsy workarounds that were good enough for reading).
+  Thanks to [Robin McCorkell](https://github.com/RobinMcCorkell) we now can properly interpret these non-byte-aligned numbers. 
+
+### Changed
+
 - Updated the analysis file Kaitai Struct definition to cope with the fact that rekordbox sometimes puts truly bizarre values (we have seen `f3` and `f9`) in the track bank byte of song structure tags.
   Previously this cause construction of the entire tag object to fail because no matching enumeration value could be found.
   Now we have a separate `raw_bank` field that holds the numeric value, and `bank` is a value instance that can be `null` when `raw_bank` is not recognizable.
@@ -36,7 +41,7 @@ May the Fourth be with you.
 - Upgraded Kaitai Struct to version 0.10, which includes a number of
   fixes and adds linting of mapped values.
   > :wrench:  This is a backwards-incompatible change.
-- Since we are already backwards incompatible with pervious releases,
+- Since we are already backwards incompatible with previous releases,
   changed some mapped value names to correspond to
   the KSY style guide and fix linter errors reported by KSC 0.10:
   1. In `rekordbox_pdb.ksy` renamed `num_groups` to `num_row_groups`.

doc/modules/ROOT/pages/exports.adoc

@@ -151,14 +151,13 @@ These each have the size specified by __len_page__ in the above diagram, and the
 (draw-box (text "next_page" :math) {:span 4})
 (draw-box (text "version" :math) {:span 4})
 (draw-box (text "unknown" :math [:sub "2"]) {:span 4})
-(draw-box (text "n" :math [:sub "rs"]))
-(draw-box (text "num" :math [:sub "rv"]) {:span 2})
+(draw-box (text "row counts") {:span 3})
 (draw-box (text "p" :math [:sub "f"]))
 (draw-box (text "free" :math [:sub "s"]) {:span 2})
 (draw-box (text "used" :math [:sub "s"]) {:span 2})
 
 (draw-box (text "u" :math [:sub "5"]) {:span 2})
-(draw-box (text "num" :math [:sub "rl"]) {:span 2})
+(draw-box (text "unk" :math [:sub "rows"]) {:span 2})
 (draw-box (text "u" :math [:sub "6"]) {:span 2})
 (draw-box (text "u" :math [:sub "7"]) {:span 2})
 (draw-gap "heap")
@@ -196,20 +195,11 @@ This might be used by the history feature, or might be some kind of data integri
 
 We don't know what _unknown~2~_ contains, though it is often/always zero. It may be an extension of _version_ to 64-bits?
 
-Next, __num_rows_small__ (abbreviated _n~rs~_ in the byte field diagram above) holds the number of row offsets (valid or not) that are present in the page, unless __num_rows_large__ (below) holds a value that is larger than it (but not equal to `1fff`). To find the actual rows, you need to scan all 16 entries of each of the row groups present in the page, ignoring any whose <<#row-presence-bits,row presence bit>> is zero.
-This seems like a strange mechanism for dealing with the fact that some tables (like playlist entries) have a lot of very small rows, too many to count with a single byte.
-But then why not just always use __num_rows_large__?
-
-__num_rows_valid__ (abbreviated _num~rv~_) holds the number of valid rows multiplied by 0x20 (i.e. the first 5 bits are not part of the counter). This count does reflect invalidated rows unlike __num_rows_small__.
-
-.Example row counts
-[example]
-* Fresh page containing 1 row: _num_rows_small_=1, _num_rows_large_=0, _num_rows_valid_=32
-* Page containing 2 rows: _num_rows_small_=2, _num_rows_large_=1, _num_rows_valid_=64
-* Page containing 2 valid rows and 1 deleted row: _num_rows_small_=3, _num_rows_large_=1, _num_rows_valid_=64
-* Page containing 6 valid rows and 2 deleted rows: _num_rows_small_=8, _num_rows_large_=6, _num_rows_valid_=192
-* Page containing 8 valid rows: _num_rows_small_=8, _num_rows_large_=7, _num_rows_valid=256
-* Page containing 7 valid rows and some deleted rows: _num_rows_small_=10, _num_rows_large_=0x1fff, _num_rows_valid_=224
+The three bytes{nbsp}``8``–``a``, labeled “row counts”, actually contain two non-byte-aligned numbers.
+The first 13 bits, __num_row_offsets__ keeps track of how many row offsets have ever been allocated in the heap, even if some of them are no longer valid.
+This number only ever increases over time, and can be used to calculate how many row groups are present in the page.
+The final 11 bits, __num_rows__, report the number of valid rows that are currently present in the table.footnote:[It is unclear why these two values are packed into three bytes like this.
+ We did not understand this structure, and had to rely on clumsy workarounds, until https://github.com/RobinMcCorkell[Robin McCorkell] figured this out in December 2025.]
 
 Byte{nbsp}``1b`` is called __page_flags__ (abbreviated _p~f~_ in the diagram).
 According to Mr. Lesniak, “strange” (non-data) pages will have the value `44` or `64`, and other pages have had the values `24` or `34`.
@@ -219,7 +209,7 @@ Bytes{nbsp}``1c``-`1d` are called __free_size__ (abbreviated _free~s~_ in the di
 
 Bytes{nbsp}``20``-`21`, _u~5~_ , are of unclear purpose. Mr. Lesniak labeled them “(0→1: 2).”
 
-Bytes{nbsp}``22``-`23`, __num_rows_large__ (abbreviated _num~rl~_ in the diagram) hold the number of entries in the row index at the end of the page when that value is too large to fit into __num_rows_small__ (as mentioned above), and that situation seems to be indicated when this value is larger than __num_rows_small__, but not equal to `1fff`.
+Bytes{nbsp}``22``-`23`, labeled __unk~rows~__, hold a value that seems related to the number of rows in the table in an unclear way, but sometimes instead equals `1fff`.
 
 _u~6~_ at bytes{nbsp}``24``-`25` seems to have the value `1004` for strange pages, and `0000` for data pages.
 And Mr. Lesniak describes _u~7~_ at bytes{nbsp}``26``-`27` as “always 0 except 1 for history pages, num entries for strange pages?”

src/main/kaitai/rekordbox_pdb.ksy

@@ -163,6 +163,8 @@ types:
       heap in which row data is found. At the end of the page there is
       an index which locates all rows present in the heap via their
       offsets past the end of the page header.
+    meta:
+      bit-endian: le
     seq:
       - id: gap
         contents: [0, 0, 0, 0]
@@ -194,22 +196,14 @@ types:
         doc: |
           @flesniak said: "sequence number (0->1: 8->13, 1->2: 22, 2->3: 27)"
       - size: 4
-      - id: num_rows_small
-        type: u1
-        doc: |
-          Holds the value used for `num_rows` (see below) unless
-          `num_rows_large` is larger (but not equal to `0x1fff`). This
-          seems like some strange mechanism to deal with the fact that
-          lots of tiny entries, such as are found in the
-          `playlist_entries` table, are too big to count with a single
-          byte. But why not just always use `num_rows_large`, then?
-      - type: u1
-        doc: |
-          @flesniak said: "a bitmask (1st track: 32)"
-      - type: u1
-        doc: |
-          @flesniak said: "often 0, sometimes larger, esp. for pages
-          with high real_entry_count (e.g. 12 for 101 entries)"
+      - id: num_row_offsets
+        type: b13
+        doc: |
+          Seems to hold the number of row offsets that have ever been
+          allocated, including those that are no longer valid.
+      - id: num_rows
+        type: b11
+        doc: The number of valid rows currently present in the page.
       - id: page_flags
         type: u1
         doc: |
@@ -226,17 +220,13 @@ types:
       - type: u2
         doc: |
           @flesniak said: "(0->1: 2)"
-      - id: num_rows_large
+      - id: unk_rows
         type: u2
         doc: |
-          Holds the value used for `num_rows` (as described above)
-          when that is too large to fit into `num_rows_small`, and
-          that situation seems to be indicated when this value is
-          larger than `num_rows_small`, but not equal to `0x1fff`.
-          This seems like some strange mechanism to deal with the fact
-          that lots of tiny entries, such as are found in the
-          `playlist_entries` table, are too big to count with a single
-          byte. But why not just always use this value, then?
+          This was previously believed to take the place of a one-byte `num_rows`
+          field, when that got too large to fit, but that was based on an incorrect
+          understanding of the `num_row_offsets` and `num_rows` bit fields. We do
+          not yet have a new explanation for the purpose of this value.
       - type: u2
         doc: |
           @flesniak said: "1004 for strange blocks, 0 otherwise"
@@ -246,24 +236,15 @@ types:
           entries for strange pages?"
       - id: heap
         size-eos: true
-        if: false  # never true, but stores pos
+        if: 'false'  # never true, but stores pos
     instances:
       is_data_page:
         value: page_flags & 0x40 == 0
         -webide-parse-mode: eager
       heap_pos:
         value: _io.pos
-      num_rows:
-        value: |
-          (num_rows_large > num_rows_small) and (num_rows_large != 0x1fff) ? num_rows_large : num_rows_small
-        doc: |
-          The number of rows that have ever been allocated on this
-          page (controls the number of row groups there are, but some
-          entries in each group may not be marked as present in the
-          table due to deletion or updates).
-        -webide-parse-mode: eager
       num_row_groups:
-        value: '(num_rows - 1) / 16 + 1'
+        value: '(num_row_offsets - 1) / 16 + 1'
         doc: |
           The number of row groups that are present in the index. Each
           group can hold up to sixteen rows, but `row_present_flags`