Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add http caching to reqwest #75

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add http caching to reqwest #75

wants to merge 1 commit into from

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Aug 11, 2023

before

Benchmark 1: pacquet
  Time (mean ± σ):     799.0 ms ±  17.2 ms    [User: 141.8 ms, System: 818.4 ms]
  Range (min … max):   777.5 ms … 827.8 ms    10 runs

after

Benchmark 1: pacquet
  Time (mean ± σ):     656.5 ms ±  26.4 ms    [User: 152.1 ms, System: 811.8 ms]
  Range (min … max):   625.6 ms … 709.3 ms    10 runs

@anonrig anonrig requested a review from KSXGitHub August 11, 2023 15:31
Comment on lines -26 to +37
clap = { workspace = true }
futures-util = { workspace = true }
rayon = { workspace = true }
reflink-copy = { workspace = true }
reqwest = { workspace = true }
node-semver = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
tokio = { workspace = true }
clap = { workspace = true }
http-cache-reqwest = { workspace = true }
futures-util = { workspace = true }
moka = { workspace = true }
rayon = { workspace = true }
reflink-copy = { workspace = true }
reqwest = { workspace = true }
reqwest-middleware = { workspace = true }
node-semver = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
tokio = { workspace = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See? This is the huge diff that I talked about.

@KSXGitHub
Copy link
Contributor

Anyway, I will review this later. Maybe tomorrow, or maybe next week.

@codecov
Copy link

codecov bot commented Aug 11, 2023

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (40f1291) 85.39% compared to head (406dfc8) 85.39%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #75   +/-   ##
=======================================
  Coverage   85.39%   85.39%           
=======================================
  Files          24       24           
  Lines        1239     1253   +14     
=======================================
+ Hits         1058     1070   +12     
- Misses        181      183    +2     
Files Changed Coverage Δ
crates/registry/src/lib.rs 0.00% <ø> (ø)
crates/cli/src/package.rs 100.00% <100.00%> (ø)
crates/cli/src/package_manager.rs 93.33% <100.00%> (+7.61%) ⬆️
crates/registry/src/package.rs 88.15% <100.00%> (-1.32%) ⬇️
crates/registry/src/package_version.rs 91.11% <100.00%> (-2.23%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions
Copy link

Benchmark Results

Linux

group                          main                                   pr
-----                          ----                                   --
tarball/download_dependency    1.05     10.2±0.60ms   426.6 KB/sec    1.00      9.7±0.53ms   447.2 KB/sec

@KSXGitHub
Copy link
Contributor

I will make a prediction: This will likely be rendered irrelevant once a proper hashmap-based caching mechanism is implemented.

Copy link
Contributor

@KSXGitHub KSXGitHub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should fix the cargo deny.

@anonrig
Copy link
Member Author

anonrig commented Aug 14, 2023

You should fix the cargo deny.

I think we shouldn't follow this pattern, and start working on the hashmap-based caching mechanism.

@KSXGitHub
Copy link
Contributor

@anonrig Since we have merged hashmap based caching mechanism and full parallelization, you can now do a benchmark (in pacquet install) to see if this PR could bring performance improvement.

@await-ovo
Copy link
Member

It looks like we've just added mem-cache to the tarball download, should we add a cache for fetching the package from the registry as well?

Whether we implement mem-cache or not, perhaps http_caching stored on disk still has some value?

@KSXGitHub
Copy link
Contributor

@await-ovo Currently, all network requests in pacquet is performed via a single wrapper: pacquet-network. Cache may be added there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants