How can I measure the performance impact of L4 vs L7 mode of LoxiLB load balancer, when it is deployed in-cluster with 5G Core ?

[souravpaul8]

I want to measure, CPU usage and memory usage when LoxiLB is deployed inside Kubernetes in L4 and L7 mode.
Please give me some direction on how to go about it ?

In L4, I understand that I would need to measure the metrics for loxilb pod, which does the actual load balancing. But in L7 mode as given in theis article:
https://www.loxilb.io/post/ngap-load-balancing-with-loxilb

Only kube-loxilb is present. No loxilb pod. Does L7 only work in external mode where the docker container running outside cluster does the L7 load balancing ?

[backguynn]

Hello. First, for loxilb L7 loadbalancing, loxilb in-cluster mode also supports it.

[TrekkieCoder]

sctp L7 loadbalancing has not yet been tested in-cluster because we need to integrate with kubernetes ingress api. This is a work in progress. Tcp(http/https) L7 LB however is available currently in-cluster.

Regarding L4 load-balancing in-cluster mode, it is difficult to measure exact cpu utilization by getting pod stats since it uses eBPF running in kernel space. You can try to.give this a try.

[souravpaul8]

There are numerous ebpf programs LoxiLB. How do I get the total cpu usage of the loxiLb L4 LB ?

[TrekkieCoder]

Hi @souravpaul8 tc_packet_func_ is the entry point which does most of the heavy lifting. But it can branch out to other tail-called eBPF programs in certain cases. So, for exact measurements , you might want to sum it all up.

[souravpaul8]

Okay got it

[UltraInstinct14]

Usually L4 LB is deployed infront of L7 LB. But if there is no such requirement, you can deploy loxilb in-cluster and use either L4 or L7 as per your need. In L4 mode, you need to use eBPF specific tool to measure CPU usage etc like previous comment. For L7 mode, currently SCTP LB does not use eBPF so you can use normal Kubernetes specific tools to check pod CPU usage.

[TrekkieCoder]

I think this is due to the fact, kube-loxilb needs some additional permissions now to support in-cluster multus.

diff --git a/manifest/in-cluster/kube-loxilb.yaml b/manifest/in-cluster/kube-loxilb.yaml
index fb2d8ef..7629825 100644
--- a/manifest/in-cluster/kube-loxilb.yaml
+++ b/manifest/in-cluster/kube-loxilb.yaml
@@ -33,6 +33,7 @@ rules:
     resources:
       - endpoints
       - services
+      - namespaces
       - services/status

If you add the above line to your kube-loxilb.yaml and reapply , the errors should go away.

[souravpaul8]

@TrekkieCoder Yes adding namespaces, solved the issue, now got the Ip

[souravpaul8]

@TrekkieCoder I am testing the L4 mode. Now External IP and Load Balancing is working fine for TCP connection, for example nginx.
But for SCTP, it is accessible from inside the cluster using cluster IP but doesn't respond, when accessed from outside the cluster using External IP. I have used the sctp.yaml from https://github.com/loxilb-io/loxilb/blob/main/cicd/k3s-flannel-incluster/sctp.yml
Same is happening with the connection between gNB and AMF for SCTP connection. I also deployed the LB in L7 mode, in that SCTP connection is getting aborted and resulting in Connection Reset by Peer.

One more question, does Load Balancing work on accessing the service from within the cluster using cluster IP ?

[TrekkieCoder]

For sctp L4 LB, you need to do install another loxilb module in the k8s nodes for in-cluster SCTP LB to work :

curl -sfL https://github.com/loxilb-io/loxilb-ebpf/raw/main/kprobe/install.sh | sh -

For L7, I am not sure what might be the exact problem.

[souravpaul8]

@TrekkieCoder Thanks, This solves the issue with the L4 Load Balancer

[prudentcircle]

@souravpaul8 If you are curious how much CPU,Mem,Network a Pod uses, my suggestion would be to use Prometheus along with Kube-state-metrics.

There are very active projects like Kube-Prometheus-Stack that lets you quickly install Prometheus and kube-state-metrics in one helm command.

Once you do that, you can find out how much a Pod is using CPU, MEM, and Network.