High availability for a small personal mail server

[timkgh]

I'd like to host my own mail server with custom domains on a VPS. But I'd like to do it in a highly available way so I do not lose emails. I'm thinking of using 2 VPS VMs, different providers. The volume of mail is low, I'm not looking for a load balancer in front, just a primary MX (mx1) and a secondary/backup MX (mx2). Most of the traffic should be handled by mx1 using the DNS MX priorities. I understand that some of the traffic may still end up on mx2 even though mx1 is available (broken senders, abusers/spammers, etc.) and that should be fine.

mx1 and mx2 can communicate via a private Wireguard connection if they need to replicate anything or forward email between them.

When mx1 is up and mx2 receives mail I'd like mx2 to just send it to mx1 for normal processing (filters, forwarding, etc.). If mx1 is down, I'd like mx2 to keep a local copy (so I can still get to it via IMAP, web interface, etc.) if I can't get around to fixing mx1 quickly (e.g. if I'm traveling), but also try to queue the mail and send it to mx1 when it's back online for normal processing.

I'm trying to keep costs to a minimum and not install and maintain fancy replicated databases or shared filesystems. Local filesystem and SQLite should be just fine for the load, single user.

Is such a setup possible with Stalwart? Other ideas to achieve high availability in a simple, cost effective setup? Not sure whether I want a "backup MX" that stores and forwards or 2 identical instances that somehow stay synchronized, settings, database in some form of active-passive setup.

@mdecimus - perhaps you could blog with ideas about such a setup 😄

[mdecimus]

Hi,

Stalwart already supports clustering and high availability. Just start multiple instances of Stalwart and point them to the same store(s) and that's it!
Currently what's pending to implement is broadcasting notifications (for the IMAP IDLE command for instance) but everything else works. How to run a cluster is going to be documented shortly.

[timkgh]

For a simple setup, 2 small/cheap VMs at 2 different cloud providers, using the same stores is not trivial to achieve. I assume same stores means a distributed/replicated database.

Does Stalwart support "backup MX" store-and-forward? Something like this. But I'd also like to be able to see the mail while the backup MX is storing it, not sure that is possible.

[mdecimus]

For a simple setup, 2 small/cheap VMs at 2 different cloud providers, using the same stores is not trivial to achieve. I assume same stores means a distributed/replicated database.

You could install Stalwart and FoundationDB (or PostgreSQL) on both machines and that will allow you to replicate your data as well.

Does Stalwart support "backup MX" store-and-forward? Something like this. But I'd also like to be able to see the mail while the backup MX is storing it, not sure that is possible.

Yes, you can set that up using routing rules. The messages will be stored in the queue and retried until the main MX is available.

[timkgh]

The messages will be stored in the queue and retried until the main MX is available.

Is it possible to get to the messages on the backup MX via IMAP (or webmail) while the main MX is down? Scenario: I can't get to fix the main MX quickly but I need to see queued mail on the backup MX, e.g. for a 2FA code. I think what I'm asking is whether a backup MX can deliver locally and also forward to the main MX.

[mdecimus]

Yes, you could create a Sieve script that modifies the envelope and adds a local e-mail address that exists in the local server

[martino87r]

I'm actually in the middle of setting up something like that, I'm not really aiming for 99.9999% uptime so I basically have a postgres database which is used for everything (except blobs, I use FS for that and borg to keep it deduped and backedup), it's replicated to a second server via WAL streaming. In practice the failover will not be automatic in an event of host failure, but I don't mind running a script to do it manually.
In relation to the mail, I've banged up a config where mailserver1 has the lowest priority in the MX record, then mailserver2; in the event of a failure of the primary node, the secondary will receive the mail for the domain without much fuss and once primary becomes available simply ship via LMTP. The documentation is a bit vague but it boils down to:

• setup an LMTP listener on the primary (use a dedicated vlan/wireguard/private lan for this)
• setup the secondary to act as a relay host for the primary (with a big enough queue)

In the next days I will be testing this setup (it's not completed) and report back here FYI