How do iptables rules work and not create routing loops

[AlexHockey]

Hi everyone. I'm new to linkerd and fairly new to iptables so apologies if this is a silly question. I'm trying to understand how linkerd's iptables rules work and do not cause a routing loop. There's probably something obvious about iptables or linkerd that I'm missing, but I can't for the life of me figure out what.

From the linkerd architecture page I understand that linkerd-init sets up iptables rules to redirect inbound and outbound traffic to the proxy.

Here are the iptables rules that are set up in one my pods (I've told linkerd to ignore port 8000).

$ iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N PROXY_INIT_OUTPUT
-N PROXY_INIT_REDIRECT
-A PREROUTING -m comment --comment "proxy-init/install-proxy-init-prerouting/1607688012" -j PROXY_INIT_REDIRECT
-A OUTPUT -m comment --comment "proxy-init/install-proxy-init-output/1607688012" -j PROXY_INIT_OUTPUT
-A PROXY_INIT_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 2102 -m comment --comment "proxy-init/redirect-non-loopback-local-traffic/1607688012" -j PROXY_INIT_REDIRECT
-A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment "proxy-init/ignore-proxy-user-id/1607688012" -j RETURN
-A PROXY_INIT_OUTPUT -o lo -m comment --comment "proxy-init/ignore-loopback/1607688012" -j RETURN
-A PROXY_INIT_OUTPUT -p tcp -m multiport --dports 8000 -m comment --comment "proxy-init/ignore-port-8000/1607688012" -j RETURN
-A PROXY_INIT_OUTPUT -p tcp -m comment --comment "proxy-init/redirect-all-outgoing-to-proxy-port/1607688012" -j REDIRECT --to-ports 4140
-A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 4190,4191,25,443,587,3306,11211 -m comment --comment "proxy-init/ignore-port-4190,4191,25,443,587,3306,11211/1607688012" -j RETURN
-A PROXY_INIT_REDIRECT -p tcp -m comment --comment "proxy-init/redirect-all-incoming-to-proxy-port/1607688012" -j REDIRECT --to-ports 4143

So taking the case of an incoming connection to the pod, I think what's happening is:

• The packet hits the PREROUTING chain then the PROXY_INIT_REDIRECT chain, and is redirected to the proxy port 4143.
• The proxy accepts the connection and extracts the original destination address.
• The proxy creates a new connection to the original address.

At this point, I would expect the outbound packet to go through the OUTPUT and POSTROUTING steps (as it is now leaving a local process), hit the NIC, then enter PREROUTING as a new packet. But because the packet is destined for the original address, why doesn't it get redirected back to the proxy server port?

In the case of an outgoing connection, the packet hits the OUTPUT chain which invokes PROXY_INIT_OUTPUT which redirects to the proxy on port 4140. But doesn't the packet then go through POSTROUTING -> NIC -> PREROUTING? At which point why isn't the packet redirected to port 4143 by PROXY_INIT_REDIRECT?

Thanks in advance! Alex.

[cpretzer]

@AlexHockey thanks for the question, I'm also not fluent in iptables rules, but I think I know what's going on here and if I'm wrong, someone else may be able to chime in.

In the PROXY_INIT_OUTPUT chain, the proxy user 2102 is ignored, with this rule:
-A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment "proxy-init/ignore-proxy-user-id/1607688012" -j RETURN

So, in the scenario you mention, the proxy has accepted the packet and will send a the request to the underlying service as the proxy user 2102, in which case, the jump to RETURN prevents the loop that you've described.

I think that's what's happening after looking through the rules in your post.

[AlexHockey]

Thanks for replying!

I think that would only work if the POSTROUTING and PREREOUTING chains are not invoked for local -> local packets (since in both my examples it's the PREROUTING chain that messes things up). That would make sense to me, but I've not been able to find any iptables documentation that says that's actually the case.

[adleong]

My understanding that the PREROUTING chain is not invoked for local connections (though it's hard to find authoritative documentation about this).

[AlexHockey]

My understanding that the PREROUTING chain is not invoked for local connections (though it's hard to find authoritative documentation about this).

I've just done a test that confirms this 🙂 .

[cpretzer]

@AlexHockey Awesome! Thanks for confirming