Skip to content

Restore from a backup

Jump to bottom
benoit74 edited this page Oct 23, 2023 · 2 revisions

Download a backup

Backups are in borgbase. To download them you need the read-only credentials:

# those are all static values you need to enter
# those are all for the _slave_ (aka readonly) bitwarden account
export BW_CLIENTID=user.xxxxxxxxx
export BW_CLIENTSECRET=xxxxxxxxxxxx
export BW_PASSWORD=xxxxxxxxxxxx

Select a backup

docker run --rm -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name zimfarm-dispatcher-postgresdb --list

zimfarm-dispatcher-postgresdb is the name of the Borgbase repository in which we archive the Zimfarm DB backups.

Output would look like

List avaible archives ...
Remote: Warning: Permanently added the ECDSA host key for IP address '95.216.113.224' to the list of known hosts.
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] yes (from BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK)
zimfarm-dispatcher-postgresdb__backup__2023-05-12T11:44:28 Fri, 2023-05-12 11:44:29 [6011272d62e793acb48efa39ab96873a9605d776f11036e065d4fa1dacba551e]
zimfarm-dispatcher-postgresdb__backup__2023-05-13T10:05:40 Sat, 2023-05-13 10:05:41 [9014fde264e23b890931f957a32946ca702247b9ac7aefb93935f3651d2a5ef8]
zimfarm-dispatcher-postgresdb__backup__2023-05-14T10:05:40 Sun, 2023-05-14 10:05:41 [a7e67dd90804da62aff0716577194d0e082c1b213159267f15e9e7643e816406]
zimfarm-dispatcher-postgresdb__backup__2023-05-15T15:20:47 Mon, 2023-05-15 15:20:48 [9310d0e4391bde5626261914cfee2fef8dbeb2ab1831d1b1dfb4062a4f931094]
zimfarm-dispatcher-postgresdb__backup__2023-05-16T10:05:47 Tue, 2023-05-16 10:05:48 [65af3dae0104339bfa8adf4c7087c0dc3690e2cbb190ec49da18e0820da0a698]
zimfarm-dispatcher-postgresdb__backup__2023-05-17T10:05:41 Wed, 2023-05-17 10:05:42 [1b9009636cd6c2b556d21d576ebf321502d5a395fa7caea1ef1262dad4e1c618]
zimfarm-dispatcher-postgresdb__backup__2023-05-18T10:05:41 Thu, 2023-05-18 10:05:42 [e5c862866bac5ec223cfece8a32250782b0cf99924cb45b3452ef3c188eb144c]
zimfarm-dispatcher-postgresdb__backup__2023-05-19T10:05:41 Fri, 2023-05-19 10:05:42 [c1d82b98d748c00cf7f57feb354433399cedc7840d18f1922c89d32b8ec96c27]

Choose one based on its date. Check the default backup periodicity in borg-backup tool and potential customization in k8s backup cronjob.

Note: the archive name is the first column (stops at first space). ex: zimfarm-dispatcher-postgresdb__backup__2023-05-19T10:05:41.

Extract a Backup file

With your selected archive name, download+extract it to your filesystem:

docker run -v /data/restore:/restore:rw -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name zimfarm-dispatcher-postgresdb --extract "zimfarm-dispatcher-postgresdb__backup__2023-05-19T10:05:41"

Zimfarm backup is a single Custom Database Dump file that would be extracted to /data/restore in this example. The file has no extension ; move it to a more practical location

mv /data/restore/root/.borgmatic/postgresql_databases/api-postgres-db-service/zimfarm /data/restore/zimfarm

Test the dump file

# start a new postrges server.
# note that this will create the `zimfarm` database and its credentials
docker run -v /data/restore:/data -it --name pg-tester --rm -e POSTGRES_DB=zimfarm -e POSTGRES_USER=zimfarm -e POSTGRES_PASSWORD=zimpass -p 5432:5432 postgres:15.2-bullseye
# import the dump in the same container
docker exec -it pg-tester pg_restore -U zimfarm -d zimfarm /data/zimfarm

Check the DB structure and data using any postgres client (DBeaver)

Production Restore Procedure

If the zimfarm is running

  • shutdown the API deployment by scaling it to 0.
  • shutdown the postgres sts by scaling it to 0.
  • Cleanup the volume
  • Move the dump file to the volume folder
  • Start the postgres sts by scaling it to 1
  • Open a shell on the postgres container
  • Restore the dump pg_restore -U zimfarm -d zimfarm /var/lib/postgresql/data/zimfarm
  • Start a PortForward and ensure the data is there
  • Check the DB structure and data using any postgres client
  • Start the zimfarm API by scaling it to 1

Downloading dump into volume

In order to get the dump file into the volume, one needs to launch borg-backup into the cluster. This would be done with a temporary Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: borg-accessor
  namespace: zimfarm
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app: borg-app
    spec:
      containers:
      - name: borg-backup
        image: ghcr.io/kiwix/borg-backup
        command: ["restore", "--name", "zimfarm-dispatcher-postgresdb", "--extract", "zimfarm-dispatcher-postgresdb__backup__2023-05-19T10:05:41"]
        imagePullPolicy: Always
        env:
        - name: BW_CLIENTID
          value: "xxxx"
        - name: BW_CLIENTSECRET
          value: "xxxx"
        - name: BW_PASSWORD
          value: "xxx"
        volumeMounts:
        - name: data-volume
          mountPath: "/restore"
          readOnly: false
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: zimfarm-api-postgres-db-pvc
      restartPolicy: Never
      nodeSelector:
        k8s.kiwix.org/role: "services"
Clone this wiki locally