Recommendations for loxilb+talos+cilium in a hybrid cluster?

[justyns]

Loxilb looks like a cool project so far. I'm still reading about it, but I was wondering what the recommendation would be for in a cluster set up like this?

• Talos Linux as the OS + Omni
• All nodes accessible to each other over a vpn (either talos kubespan or a generic wireguard/tailscale vpn)
• Cilium as the cni
• Mix of bare metal dedicated servers and vps
• Some servers are behind NAT with no direct public ipv4
• Some servers are not behind NAT and have a direct public ipv4 attached
• Worker nodes are spread over multiple datacenters, so not on the same physical network

Can loxilb be deployed to help in a situation like this? E.g. traffic from the internet would only be able to come in to the nodes that have a public ip attached. If it can, are there any guides or write ups with a similar architecture that I may have missed?

Specifically one area where I am getting hung up on is loxilb's active-active HA seems like it'd require bgp or other network configuration that may not be available. I was thinking more of having loxilb deployed only to the nodes with public/external ips attached and then just having dns point to one or all of those nodes. FWIW this would only be for http/https traffic.

[nik-netlox]

Hi @justyns,
Thanks for contacting us. Please find answers to your question inline.

Can loxilb be deployed to help in a situation like this? E.g. traffic from the internet would only be able to come in to the nodes that have a public ip attached. If it can, are there any guides or write ups with a similar architecture that I may have missed?

Yes, LoxiLB can be deployed in such scenarios. We are actively working to support such scenarios. Please follow this.

Specifically one area where I am getting hung up on is loxilb's active-active HA seems like it'd require bgp or other network configuration that may not be available. I was thinking more of having loxilb deployed only to the nodes with public/external ips attached and then just having dns point to one or all of those nodes. FWIW this would only be for http/https traffic.

BGP is one of the way to distribute the incoming traffic, it is not mandatory. LoxiLB can also be deployed as you are suggesting. Here is the guide where LoxiLB can be deployed in EKS in active-active mode without BGP where DNS redirects the traffic to the nodes where LoxiLB is running.

Also, Please have a look at the architecture diagram for the multi-cloud HA scenario.

If you have further questions, please join our slack channel. We would be happy to help you.

[justyns]

Hi @nik-netlox , thanks for the response.

I'm looking through the links you provided, and also trying to do some tests in my cluster with loxilb. I'm still a little confused on how certain parts work, but right now the main question I have is about the external IP.

If I deploy loxilb in-cluster with a config like the one in https://github.com/loxilb-io/loxilbdocs/blob/main/docs/eks-incluster.md - then I can create a load balancer service which shows up like this:

NAMESPACE      NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default        nginx-lb1                        LoadBalancer   10.102.92.97     llbanyextip   55002:32329/TCP          13h

I'm assuming I can't change --externalCIDR=0.0.0.0/32 because I have multiple nodes that each have different external/public ips and can't share each other's ips?

I'm also using external-dns to automatically create dns records that point to svc/lb ips. Is there anything special I'd have to do to get that working? Or would I need to hardcode the IPs somewhere?

Would deploying loxilb on nodes outside of the cluster be easier?

[nik-netlox]

Hi @justyns,
"llbanyextip" - this will create the service with all the IPs on all the nodes where loxilb is running. You just have to configure your DNS with the public IP of all those worker nodes.

[justyns]

I'm not sure if this is a better question for external-dns or here, but do you happen to know how to get external-dns to automatically find those worker node public ips and use them?
I don't want to hardcode the IPs in dns unless I need to.

[nik-netlox]

We are working on the integration with external-dns. We will update you about its completion. For now, you can hardcode the values.