Skip to content

Configurable Binary Token Parsing #234

Draft
nickbabcock wants to merge 10 commits intomasterfrom
ng
Draft

Configurable Binary Token Parsing #234
nickbabcock wants to merge 10 commits intomasterfrom
ng

Conversation

@nickbabcock
Copy link
Contributor

@nickbabcock nickbabcock commented Jan 19, 2026

Related #233

In a recent HOI4 patch (patch 1.17, save_version=30) they deviated from the established binary format norms. The usual F32 token now signals F64 data. Currently the code is tied to associating 0x000d with 32 bits of data.

This PR plays around with allowing each game to define a stateful parser to handle how HOI4, CK3, and EU5 may differ in how something is decoded.

pub trait BinaryFormat {
    fn visit(&mut self, reader: &mut ParserState, id: LexemeId) -> Result<TokenSignal, Error>;
}

I created a reader-ng benchmark that shows an example implementation with MyFormat. I found it to be 10-20% slower than reader-token-kind, which is not ideal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant