2.0.0b5 : borg2 --check -r repo seems to be stuck running for a while without any update even with --verbose

[AkechiShiro]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE/QUESTION (how to get more verbose debug ?? to see what's going on?)

Your borg version (borg -V).

Only client : borg2 2.0.0b5 (no server involved)

Operating system (distribution) and version.

Running on Linux Debian 12.7 (Proxmox 8)

• Kernel :

Linux pve 6.8.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-1 (2024-08-05T16:17Z) x86_64 GNU/Linux

• OS_RELEASE :

cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Hardware / network configuration, and filesystems used.

Hardware is a laptop with ext4 filesystem but the data and the borg repo is on an external HDD which uses the NTFS filesystem (I should've formatted to something else before using it...)

How much data is handled by borg?

About 465GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg2 --check -r --verbose

Describe the problem you're observing.

Just stuck for hours showing :

• Starting repository check

Mind that this is running on an external HDD (USB 3.0), plugged in a USB 3.0 port.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I don't think I can, the backup must've been interrupted mid way because the machine had shut down due to power loss, so there is bound to be issues with the repo.

I would like to ideally get back the data that has been saved, if not it's alright I've got other backups.

Include any warning/errors/backtraces from the system logs

I saw no issues in dmesg logs.

[ThomasWaldmann]

Maybe Ctrl-C / kill that run and try again, giving also the --progress option.

Only in case the repo is locked, make sure no borg is accessing it and use borg break-lock on it. Then try the check again.

Also check the USB HDDs activity LED. After borg has loaded the files and chunks cache from your internal SSD, there should be quite some activity on the USB HDD.

[AkechiShiro]

It seems to be showing some progress but it's incrementing by 0.2% every 30 seconds, I'd say, I'll wait until it gets to 100% :

Starting repository check
Checking segments 2.3%   

Maybe it was working but just needed a few more hours

[AkechiShiro]

@ThomasWaldmann

Finished full repository check, no problems found.
Starting archive consistency check...
Local Exception
Traceback (most recent call last):
  File "/usr/lib/borgbackup2/borg/archiver/__init__.py", line 595, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/archiver/__init__.py", line 523, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/archiver/_common.py", line 156, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/archiver/check_cmd.py", line 52, in do_check
    if not args.repo_only and not ArchiveChecker().check(
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/archive.py", line 1778, in check
    self.key = self.make_key(repository)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/archive.py", line 1830, in make_key
    return key_factory(repository, cdata)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/crypto/key.py", line 113, in key_factory
    assert manifest_data, "manifest data must not be zero bytes long"
AssertionError: manifest data must not be zero bytes long

Platform: Linux pve 6.8.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-1 (2024-08-05T16:17Z) x86_64
Linux: Unknown Linux  
Borg: 2.0.0b5  Python: CPython 3.11.2 msgpack: 1.0.3 fuse: pyfuse3 3.2.1 [pyfuse3,llfuse]
PID: 76568  CWD: /mnt/PC-backup
sys.argv: ['/usr/bin/borg2', 'check', '--progress', '--verbose', '-r', 'borg-bak']
SSH_ORIGINAL_COMMAND: None

[ThomasWaldmann]

Slow speed: maybe you use some FUSE filesystem for ntfs and is slow due to that implementation.

Tracback in previous comment: looks like the repository manifest was corrupted.

borg check --repair should be able to repair this (assuming your hardware and filesystem works correctly).

[ThomasWaldmann]

BTW, we are at 2.0.0 beta9 meanwhile and guess after my big PR will get merged, I'll release a b10 ASAP.

Don't use the betas for production backups, only for additional backups intended for beta-testing. b10 might bring very big changes.

[AkechiShiro]

Well at this point, I should just try the borg check --repair and hope for the best if that doesn't work, I do have other backups of the data.

I thought check would tell me to run check --repair not show this traceback, I guess since this is a beta that's how it currently is.

You're correct about the FUSE filesystem for ntfs (I'm using ntfs-3g)

Thanks also for the mention about b9, b10 and the advice of not using betas for prod backups.