Regarding creation of alertmanager gossip cluster with across kubernetes cluster

[ghegdeakamai]

Hi, I have 2 kubernetes clusters in different geo locations and I want to form a gossip alertmanager HA with one alertmanager on kubernetes cluster 1 and another on kubernetes cluster 2 . Is it possible to setup gossip among alertmanagers on different kubernetes clusters? Does gossip protocol support alertmanagers distributed across multiple kubernetes clusters?

[TheMeier]

I have such a setup. This is how do it. Each pod of the alertmanager sts has a dedicated metallb endpoint, that is reachable across clusters. For each instance the cli argument --cluster.peer is then set for all gossip members. Roughly like this in the pod spec of the sts:

echo "Hostname: $my_hostname"
export LB_IP=$(cat /etc/alertmanager/config/${my_hostname})
echo "LB_IP: $LB_IP"
/bin/alertmanager \
  --config.file=/etc/alertmanager/config/alertmanager.yaml \
  --storage.path=/alertmanager \
  --cluster.listen-address=0.0.0.0:9094 \
  --web.listen-address=:9093 \
  --web.route-prefix=/ \
  --web.external-url=https://anycast-ingres.example.domain/ \
  --cluster.advertise-address=${LB_IP}:9094 \
  --cluster.peer=alertmanager-0.cluster1.example.domain:9094 \
  --cluster.peer=alertmanager-1.cluster1.example.domain:9094 \
  --cluster.peer=alertmanager-0.cluster2.example.domain:9094 \
  --cluster.peer=alertmanager-1.cluster2.example.domain:9094 \
  --cluster.reconnect-timeout=5m

/etc/alertmanager/config/${my_hostname} cointains matching entries with the "static" IPs of the metallb endpoints for each pod in the corresponding datacenter so alertmanager will know wich IPs others use to talk to it.