Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pacman 6.0 ignores the pacredir proxy after three 404 errors #10

Open
haraldkoch opened this issue Jun 3, 2021 · 9 comments
Open

pacman 6.0 ignores the pacredir proxy after three 404 errors #10

haraldkoch opened this issue Jun 3, 2021 · 9 comments

Comments

@haraldkoch
Copy link

With pacman 6.0:

error: failed retrieving file 'core.db' from 127.0.0.1:7077 : The requested URL returned error: 404
error: failed retrieving file 'extra.db' from 127.0.0.1:7077 : The requested URL returned error: 404
error: failed retrieving file 'community.db' from 127.0.0.1:7077 : The requested URL returned error: 404
warning: too many errors from 127.0.0.1:7077, skipping for the remainder of this transaction

After this, pacman and friends ignore the pacredir mirror completely, and (re)download the packages from the internet.

This appears to be caused by a combination of two changes:

5be95f3 - do not redirect if database file is too old

And a change to pacman: https://bugs.archlinux.org/task/29293

Unfortunately, the pacman change is hardcoded to 3 attempts; they did not add a configuration option.

I'm not sure what to suggest to fix it, but pacredir is basically unusable with the pacman change in place...

Thanks!
@eworm-de
Copy link
Owner

eworm-de commented Jun 3, 2021

This issue is known and I am trying to find a solution. Stay tuned...

eworm-de added a commit that referenced this issue Jun 7, 2021
@eworm-de
add info about caveat with pacman's server error limit
997be8d 
... also reported in #10.
eworm-de added a commit that referenced this issue Jun 7, 2021
@eworm-de
add info about caveat with pacman's server error limit
1326a40 
... also reported in #10.
@eworm-de
Copy link
Owner

eworm-de commented Jun 7, 2021 •
edited
Loading

Ok, some words about what happened...

I've tried to make pacman support http header to indicate an expected failure, but that was refused.

Currently the only option is to rebuild pacman with the server error limit feature disabled. Alternatively install pacman from the repository I created:

[pacredir]
Server = https://pkgbuild.com/~eworm/$repo/$arch/

I've pushed a new pacredir release to the repositories that gives the same instructions:

With its latest release pacman now supports a server error limit,
which breaks pacredir. Rebuild pacman with that feature disabled
or install from this repository:
[pacredir]
Server = https://pkgbuild.com/~eworm/$repo/$arch/

In the long run we can hope to see FS#23407 - Allow soft failures on Server URLs being implemented. We can add pacradir as (additional) cache server then.

@eworm-de
Copy link
Owner

Sent my current implementation of CacheServer for pacman here:
https://lists.archlinux.org/pipermail/pacman-dev/2022-January/025505.html

Anybody wants to give it a try? Add this in your pacman.conf:

[cache-servers]
Server = http://pkgbuild.com/~eworm/$repo/

@limitedAtonement
Copy link

I added that to my pacman.conf then pacman -Syu, but didn't get a new pacman. I assume I was supposed to get a new pacman that replaces the one from the official repositories?

sudo pacman -Syu
:: Synchronizing package databases...
 core                                                 157.5 KiB   543 KiB/s 00:00 [###############################################] 100%
 extra                                               1718.3 KiB  2.59 MiB/s 00:01 [###############################################] 100%
 community                                              6.7 MiB  4.89 MiB/s 00:01 [###############################################] 100%
 multilib                                             172.6 KiB   664 KiB/s 00:00 [###############################################] 100%
 cache-servers                                       1206.0   B  1103   B/s 00:01 [###############################################] 100%
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (5) gnupg-2.2.36-1  imagemagick-7.1.0.41-1  nodejs-18.5.0-1  perl-uri-5.11-1  sdl2_ttf-2.20.0-1

Total Download Size:   16.07 MiB
Total Installed Size:  59.09 MiB
Net Upgrade Size:       0.11 MiB

:: Proceed with installation? [Y/n] n

Then I aborted thinking I should try to reinstall pacman:

sudo pacman -Syu pacman
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
 cache-servers is up to date
warning: pacman-6.0.1-5 is up to date -- reinstalling
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (6) gnupg-2.2.36-1  imagemagick-7.1.0.41-1  nodejs-18.5.0-1  perl-uri-5.11-1  sdl2_ttf-2.20.0-1  pacman-6.0.1-5

Total Download Size:   16.07 MiB
Total Installed Size:  63.78 MiB
Net Upgrade Size:       0.11 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 perl-uri-5.11-1-any                                   80.0 KiB  1096 KiB/s 00:00 [###############################################] 100%
 sdl2_ttf-2.20.0-1-x86_64                             633.3 KiB  1336 KiB/s 00:00 [###############################################] 100%
 gnupg-2.2.36-1-x86_64                                  2.3 MiB  2.13 MiB/s 00:01 [###############################################] 100%
 imagemagick-7.1.0.41-1-x86_64                          2.8 MiB  1894 KiB/s 00:02 [###############################################] 100%
 nodejs-18.5.0-1-x86_64                                10.2 MiB  4.31 MiB/s 00:02 [###############################################] 100%
 Total (5/5)                                           16.1 MiB  6.65 MiB/s 00:02 [###############################################] 100%
(6/6) checking keys in keyring                                                    [###############################################] 100%
(6/6) checking package integrity                                                  [###############################################] 100%
(6/6) loading package files                                                       [###############################################] 100%
(6/6) checking for file conflicts                                                 [###############################################] 100%
(6/6) checking available disk space                                               [###############################################] 100%
:: Processing package changes...
(1/6) upgrading gnupg                                                             [###############################################] 100%
(2/6) reinstalling pacman                                                         [###############################################] 100%
(3/6) upgrading imagemagick                                                       [###############################################] 100%
(4/6) upgrading nodejs                                                            [###############################################] 100%
(5/6) upgrading perl-uri                                                          [###############################################] 100%
(6/6) upgrading sdl2_ttf                                                          [###############################################] 100%
:: Running post-transaction hooks...
(1/3) Arming ConditionNeedsUpdate...
(2/3) Warn about old perl modules
(3/3) Updating the info directory file...

@eworm-de
Copy link
Owner

eworm-de commented Jul 8, 2022

Something is wrong there. Did you add the lines before core repository?

@limitedAtonement
Copy link

Thank you, that helped! Now a pacman -Syu; downloaded and installed pacman-6.0.1-5.80-x86_64.

I added CacheServer = http://127.0.0.1:7077 to my /etc/pacman.conf in the [options] section. Now when I run, for instance, pacman -S firefox;, I get warning: config file /etc/pacman.conf, line 44: directive 'CacheServer' in section 'options' not recognized.. I think this means I don't have your pacman installed properly? Would you verify the version string above? What else should I do?

@eworm-de
Copy link
Owner

eworm-de commented Jul 8, 2022

This belongs into each repository block.

@limitedAtonement
Copy link

limitedAtonement commented Jul 8, 2022 •
edited
Loading

It works here. To recap:

  1. Add a new repository: 
    [cache-servers]
Server = https://pkgbuild.com/~eworm/$repo/
    above the [core] repository in /etc/pacman.conf
  2. sudo pacman -Syu pacredir; should install the fixed pacman and pacredir
  3. sudo systemctl enable --now pacserve pacredir; (ss -antl | grep 7077; should show a process listening on port 7077)
  4. Add CacheServer = http://127.0.0.1:7077/ at the top of each of the repository blocks in /etc/pacman.conf
  5. To review logs, sudo journalctl -u pacredir -u pacserve; on both machines

I'm not sure how to verify that pacman is using the local network redirect server, though. The way I verified is that my download speed was well over 100 MiB/s (I don't go over 7 MiB/s over the internet).

EDIT: Updated to add another step for verification.

@eworm-de
Copy link
Owner

eworm-de commented Jul 8, 2022

You should see messages from pacredir in your local logs, and access logs on the peers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@eworm-de @haraldkoch @limitedAtonement