how to monitor dashboard Failures when performing backup and other operations

[GuitarBilly]

npbackup3.0.0-rc13-pub
linux Pop!_OS 22.04 LTS

As of now i have a working setup including the grafana dashboard. When I first started a backup on this machine i got multiple Failure operations logged.
One was easy to spot from the Error Messages window; one file in the backup path had a permission issue. It does not need to be backed up so I added that to the exclude list.
With that change the backup from the GUI runs perfect without any error. However at each backup I keep getting red Failures actions in the grafana NPBackup v3 20241104 dashboard.

One failure at action=backup, and one failure at action=snapshots.
To be clear, I also have one Green succesful backup count, which matches what the GUI suggests.

The only thing I see from the commandline is an ERROR: permission denied: /var/log/npbackup-gui.log because I run this as normal user not as root or sudo. But it reverts to use /tmp/ofunctions.logger_utils.log instead.

Running npbackup-cli with the gui config file and with --snapshots gives the same snapshots failure in the dashboard and no clue in the terminal log apart from the already mentioned log file folder error and workaround.

How would I go ahead and investigate the cause of the reported dashboard Failures?f

EDIT: I run the rest-server in append-only mode, the gui is configured with Full permissions and the default Retention policy is still active. Could that conflict? I will change it and try again and report back later.

[GuitarBilly]

experiment 1.
change permissions from Full Permissions to the more restricted permissions or only Backup permissions.
Grafana Dashboard : no error with action=backup, still error on action=snapshots

experiment 2.
launch npbackup-gui with sudo. now i notice that the Current permissions are reset to Full permissions again.
(now i remember i had already reduced the permissions as normal user yesterday too. somehow that now did not get saved)
Grafana Dashboard : no error in Failures, one green addition in the Succesful runs box.

so this seems to boil down to a problem with permissions?

[GuitarBilly]

note to self: inspect the prometeus (pushgateway) data. that could be the answer to my question.

[deajan]

So basically, there are two important metrics.

• restic_backup_failure which is restic backend backup status (only appears on backup operations)
• npbackup_exec_state{action='backup} which is the full backup status (appears on every operation, whith action=operation)

While the first metric shows that the backup itself suceed, the second metric shows if backup succeed and eveything around backup (pre/post exec scripts, backup minimum size checks...) have succeed also
The second metric will show an error if there is any log message with a level greater than warning.

I guess I could document that a bit better.

In your case, having a log permission error will trigger an error on the second metric, even if the actual backup succeed.

Does this answer your question ?

[deajan]

Perhaps I should change the titles on the dashboard to "backend backup status" and "overall backup status" ?

[GuitarBilly]

This clarifies it, thanks.

What confused me most is that there is no log permission error shown at all in the GUI.
Only when I launch from the terminal you can see the error in the terminal output.

• Is that a design philosophy or a bug? I could argue either way 8-)

p.s. not sure why the sudo would be required to avoid the snapshot-action-failures, I do not see it on my linux ubuntu computer. I will likely access that Pop!_OS machine sometime next week.

[deajan]

TBH, the GUI doesn't show all logs, only import errors like operation failures.
I considered auto opening a log window on bad log events (#89), but that's another thread development which is quite horrible, and is on a FR list.

[GuitarBilly]

This clarifies it, thanks.

What confused me most is that there is no log permission error shown at all in the GUI. Only when I launch from the terminal you can see the error in the terminal output.

* Is that a design philosophy or a bug? I could argue either way   8-)

p.s. not sure why the sudo would be required to avoid the snapshot-action-failures, I do not see it on my linux ubuntu computer. I will likely access that Pop!_OS machine sometime next week.

finally came around to access the Pop!_OS machine. All failures boil down to the /var/log/ logfile permission error that was already discussed before. I fixed that and now there are no failures anymore whether normal user or root is executing.
For me another step forward and this chapter is closed.