status: offical pacman support

[RubenKelevra]

I've talked a while ago about this on the pacman dev mailing list - but it was kind of rejected.

https://lists.archlinux.org/pipermail/pacman-dev/2020-April/024179.html

This would still require the databases to be centralized stored in a cluster, as well as the packages, but it would be done by an "official" team instead of me - in the best case.

This way the IPFS update push would happen automatically and the updates would be faster and somewhat more reliably than a rsync to ipfs script written by some random guy on the internet.

It would also allow having multiple writing servers on the same cluster, which can do the updates seamlessly - since when they do the same update, the cluster would just merge them as the same change. This means we can completely eliminate any single point of failure.

Originally posted by @RubenKelevra in #40 (comment)

[RubenKelevra]

Some more thoughts on that:

Linux software is already pretty well available, since there are many generous providers which setup an rsync with http.

Just look at what’s available for Arch Linux:

https://www.archlinux.org/mirrors/status/

So my intention is not to fix an issue with availability, but help to accelerate a transition from sharing updates via static servers to spread updates from the developer machine to the cluster, accessing them via their CID instead of their filename and everyone help to share the file, making them longer and better available.

If we would transition from static compressed databases to directory listings in IPFS as ‘update lists’, we could also safe a lot of traffic, since IPFS is able to deliver delta updates.

A background process for pacman could refresh the package list constantly, without using a lot of traffic for this. Avoiding any delay for ‘syncing’.

A background process could also start to pin new updates when they are available and when they are fully available a gui program could offer a popup to the user, that new updates are available for installation.

I like also the idea of local community networks, and have a history building them. Software like IPFS can help to reduce internet traffic if there’s just low bandwidth available by sharing updates and software with other computers on the local network.

Appeared first here.

[tobixen]

I would still like to use IPFS for upgrading my systems, though due to lack of time and practical problems I'm currently using the official mirrors.

I have four motivations for using IPFS:

• Because it's cool!
• Eventually, I do tend to have quite many laptops and other computers around with Archlinux installed, and sometimes the uplink is a bottle neck. There are probably are other tools and methods that are better suited for avoiding duplicated file transport over the uplink, but ideally I expect that IPFS would reduce the amount of duplicated traffic over the upstream link when several computers on the local network that should upgrade the same packages.
• Since I'm primarily using Archlinux on laptops and since I'm frequently on the move, having a static mirror list is not always optimal (and any particular setup to utilize local cache servers may fail).
• Sometimes one may want to roll back to old packages, but that's difficult as they are typically disappearing from all the mirror servers rapidly. With IPFS that would probably not be a big problem.

[RubenKelevra]

Hey @tobixen,

can you share what your practical problems currently are, which should be fixed by a pacman support of IPFS?

Rolling back to an older version would be somewhat simpler to implement with IPFS, since you can fetch it from the network from a random node that still caches it instead of having to rely on a centralized server that needs to hold all the data.

[tobixen]

It's many years since I last gave pacman over ipfs a serious try, but as far as I remember I had those issues ...

• Content discovery being incredibly slow on IPFS, not only that IPFS was magnitudes slower than using a mirror, but also that I would get timeouts all the time.
• I wanted to fail back to using a regular mirror, but making whatever I downloaded available through IPFS. I don't think I found any smart ways of doing that. ("ipfs mount" on /var/cache/pacman/pkg/ would probably solve that, but there were always problems with "ipfs mount"). I did set up some crontab to put everything under /var/cache/pacman/pkg/ available by ipfs and expire content from that directory, but frequently ran into issues that package files were missing from /var/cache/pacman/pkg/
• Duplicated storage, as package files would be both available in /var/cache/pacman/pkg/ and in ipfs. (ipfs mount would probably solve this). (cue "disk is virtually free nowadays" ... no, that's not always trivial to just add more disk).
• It never got truly distributed as I needed to rely on random third parties to make everything available on ipfs.

[RubenKelevra]

There's a new function being discussed by @eworm-de which might fit what we need: It would allow us to specify ipfs as a cache server and if ipfs fails to deliver a package a regular server will be used.

https://lists.archlinux.org/pipermail/pacman-dev/2022-January/025492.html

[eworm-de]

Sent my current implementation here:
https://lists.archlinux.org/pipermail/pacman-dev/2022-January/025505.html

Anybody wants to give it a try? Add this in your pacman.conf:

[cache-servers]
Server = http://pkgbuild.com/~eworm/$repo/

[RubenKelevra]

@eworm-de the pacman binary just segfaults on me :/

Can you share the PKGBUILD, please? :)

[eworm-de]

Just apply this commit on top of official PKGBUILD: eworm-de/pacman@83d57f3

[eworm-de]

The package from my repository works fine for me... Verified just now.
Can you give any detail from your configuration that causes this? Any issue with pacman from official repository?

[RubenKelevra]

Can you give any detail from your configuration that causes this?

Hm. I don't think I use something special. Pretty much the default config (and no cache servers configured) plus ZFS stuff:

[options]
HoldPkg     = pacman glibc
CleanMethod = KeepInstalled
Architecture = auto

IgnorePkg   = linux-lts linux python2 binutils-blackikeeagle

Color
CheckSpace
ParallelDownloads = 1

SigLevel    = Required DatabaseOptional
LocalFileSigLevel = Optional

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

[multilib]
Include = /etc/pacman.d/mirrorlist

[archzfs]
# Origin Server - France
Server = http://archzfs.com/$repo/x86_64
# Mirror - Germany
Server = http://mirror.sum7.eu/archlinux/archzfs/$repo/x86_64
# Mirror - Germany
Server = https://mirror.biocrafting.net/archlinux/archzfs/$repo/x86_64
# Mirror - India
Server = https://mirror.in.themindsmaze.com/archzfs/$repo/x86_64

[zfs-linux-lts]
Server = http://kernels.archzfs.com/$repo/

[zfs-linux]
Server = http://kernels.archzfs.com/$repo/

Any issue with pacman from official repository?

Nope, just fetched the regular pacman package and overwrote the binary and the issue went away.

Maybe that was just a mismatch of a library. 🤷

[eworm-de]

Wait, you took just the pacman binary from my package? Of course that breaks, as libalpm has API changes.

[RubenKelevra]

@eworm-de no I just added your server and made an update :)

Anyway, why don't you just create an AUR package for people to try? Better than adding some binaries from a random server :D

[m00nwtchr]

Not sure where to post about this, but i just found about the Filestore feature in IPFS, which sounds like a pretty important thing for this project, to avoid duplicating the pacman cache: https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipfs-filestore

Note: It does look like it only works if the path is under the IPFS repo directory, but i managed to solve that with a bind mount (well, actually a subvolume mount because i'm on BTRFS, but a bind mount should work, too)

[RubenKelevra]

@m00nwtchr I know this feature. It's experimental and if a file is "gone" locally it will fail phenomenal. Also I don't think it can actually handle duplicate blocks. So if you add a file the blocks won't be removed from the ipfs storage until you get to the next round of garbage collection.

So yeah, not helpful atm.

But there's work done to mount ipfs more reliable than currently, this could be an option.