feat: Log the backup stats summary to a file

[gerardbosch]

I was missing to be able to track how the backup repo grows, and finally came up with a log to persist what desktop notification shows after the backup (many times I miss the notification and don't realized if I unconsciously added heavy files to the home backup).

When something gets added unintentionally to the backup I usually check/update the ~/.backup_exclude.txt and manually run sudo resticw forget --prune <offending snapshot ID>. The stats log will help to better track these situations.

The stats log feature could be disabled (opt-out) by setting empty the new environment variable $RESTIC_BACKUP_STATS_DIR.

The log records the added, removed and snapshot size after each backup to a CSV log. This is how it looks:

❯ cat /var/log/restic-automatic-backup-scheduler/2023-stats.log.csv    
Date, Snapshot ID, Added, Removed, Snapshot size
2023-12-11 23:05:00, fe91dd73, 3.383 MiB, 3.375 MiB, 8.042 GiB
2023-12-11 22:27:27, a3fa2de3, 2.413 MiB, 2.443 MiB, 8.042 GiB
2023-12-11 21:37:27, e414fb73, 4.285 MiB, 4.265 MiB, 8.042 GiB

❯ csvlook /var/log/restic-automatic-backup-scheduler/2023-stats.log.csv
|                Date |  Snapshot ID |  Added     |  Removed   |  Snapshot size |
| ------------------- | ------------ | ---------- | ---------- | -------------- |
| 2023-12-11 23:05:00 |  fe91dd73    |  3.383 MiB |  3.375 MiB |  8.042 GiB     |
| 2023-12-11 22:27:27 |  a3fa2de3    |  2.413 MiB |  2.443 MiB |  8.042 GiB     |
| 2023-12-11 21:37:27 |  e414fb73    |  4.285 MiB |  4.265 MiB |  8.042 GiB     |

Why a CSV? Just to avoid to fill a file with the same text message on each line. I thought on simply adding a header and just write the data, not the full message each time. New records are prepended in the log, the latest backup sits below the header (descending order), easy to inspect.

Organized with yearly file, e.g. 2023-stats.log.csv

[erikw]

Great thank you @gerardbosch, and for using functions!

Juat one comment

[erikw]

Could we make this opt-in please, to make it as simple as possible for new users. Advanced users can enable it. We can include an out-commented line as an example of how to enable it.

[gerardbosch]

Hi! sure!! But just want to double-check it :)

Do you think it is better to have the log disabled by default? Probably I'm biased and looking from my use-case perspective only, but I guessed that would be a sane default to have a summary log enabled by default for all users, and make it an opt-out. For myself I find to log this info an essential feature.

Maybe looking from my limited use case here, hehe (I'm just doing a daily backup of home). The only cons I see of having it enabled by default is that it needs to run restic snapshots and restic diff (I stated it on the README 🙂), which takes a little more time to finish the systemd service (not the backup) and I guess requires extra talk to the remote repo (but as it runs unattended on background I don't care). The benefit is that you can quickly check how the backup increased on every snapshot without ceremony, and helps to investigate the backup.

Let me know, and if you still think it is better to be an opt-in I could just comment # this line and add the (optional) comment 😃, leaving the example directory as-is /var/log/restic-auto... but commented + update the README accordingly. Cheers!

[gerardbosch]

P.S. I'm not sure if the data transfer supposedly required for restic diff —which is used (single time) for both, log and notification— may be a problem for general public, I don't actually know how it works, or if it computes against a local cache or something like this, couldn't find it in the docs https://restic.readthedocs.io/en/latest/040_backup.html#comparing-snapshots

[erikw]

Hi, I see your point that it makes sense to have. I personally would like it. Though the project scope has grown a bit too much making maintenance harder (I have much less time at the moment for it). Thus I would much appreciate if this was opt-in. 🙂

[gerardbosch]

Yep, no problem! I disabled it (left it commented), you can check now ccdf7b7