Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: faster zip entry extraction #1661

Merged
merged 1 commit into from
Aug 26, 2024
Merged

Conversation

Snd-R
Copy link
Contributor

@Snd-R Snd-R commented Aug 22, 2024

Currently ZipFile read will scan entire file to read each entry's local file header on each request for book page image.
This can be quite a slow operation depending on archive file size. In my tests 800mb archive with ~400 entries stored on hdd takes ~3 seconds on initial read on Linux system.
After initial read OS will cache file and subsequent reads become really fast ~50-100ms, but in my opinion it's better to not rely on OS for caching

If we only read central directory header entry retrieval takes ~100-150ms instead of 3 seconds which is about 30 times faster
The downside to only reading central directory is that filename can use Unicode extra field and that field is (for some reason) only added to local file header.
In that case retrieving entry might fail and we fallback to iterating each entry local file header and set name from Unicode extra field if it exists

I've also updated Apache Compress library version which deprecated ZipFile constructor calls and replaced it with Builders

@gotson
Copy link
Owner

gotson commented Aug 23, 2024

thanks for the PR, i'll have a look when i have time, and will run some tests too

@gotson gotson changed the title feat: zip entry read fast path perf: faster zip entry extraction Aug 26, 2024
@gotson gotson merged commit eeb5898 into gotson:master Aug 26, 2024
3 checks passed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants