Behavorial difference in IP allocation between Podman and Docker

[JCBird1012]

This may become irrelevant whenever Netavark supports --ipam-driver dhcp - but I wanted to ask anyway.

After migrating to Podman, I started to see issues with containers on my network having stale DNS records cached - apparently, when containers are either restarted/recreated - Podman (Netavark?) will more aggressively assign new IPs than what I'm used to in Docker.

I know that the proper answer is to statically assign IPs to each container - but I just wanted to point out this difference, because it may be troublesome for people making the migration.

I'll also preface by saying that I don't think either behavior is technically incorrect - I actually like Podman's behavior more - but the consequence is the stale cached DNS records, and with Aardvark's TTL of 86400, that can mean containers won't know about another container's new IP until at most a day later.

Podman

[root@dealloc-test nginx]# podman run -d --name=nginx-test --network=test -v './nginx.conf:/etc/nginx/nginx.conf' docker.io/nginx
2ca8e3cccda0cb8c0f5863f206bcad0c090d5939dacd7d0bcc0c72f92bf98d7c

[root@dealloc-test nginx]# dig +short nginx-test @10.89.0.1
10.89.0.2

[root@dealloc-test nginx]# podman stop nginx-test && podman start nginx-test
nginx-test
nginx-test

[root@dealloc-test nginx]# dig +short nginx-test @10.89.0.1
10.89.0.3

[root@dealloc-test nginx]# podman stop nginx-test && podman rm nginx-test
nginx-test
nginx-test

podman run -d --name=nginx-test --network=test -v './nginx.conf:/etc/nginx/nginx.conf' docker.io/nginx
a07147aaec5ed3a067fc527e4436dddd78dd07ec38eb9142f3e7cb92578e5a52

[root@dealloc-test nginx]# dig +short nginx-test @10.89.0.1
10.89.0.4

Docker

[root@dealloc-test nginx]# docker run -d --name=nginx-test --network=test-docker -v './nginx.conf:/etc/nginx/nginx.conf' docker.io/nginx
1e9ed441ad6bf4045ea1ebce561fbd846577074195d62c97f0ae432974264945

[root@dealloc-test nginx]# docker container inspect nginx-test | jq '.[0].NetworkSettings.Networks | .[].IPAddress'
"172.18.0.2"

[root@dealloc-test nginx]# docker stop nginx-test && docker start nginx-test
nginx-test
nginx-test

[root@dealloc-test nginx]# docker container inspect nginx-test | jq '.[0].NetworkSettings.Networks | .[].IPAddress'
"172.18.0.2"

[root@dealloc-test nginx]# docker stop nginx-test && docker rm nginx-test
nginx-test
nginx-test

[root@dealloc-test nginx]# docker run -d --name=nginx-test --network=test-docker -v './nginx.conf:/etc/nginx/nginx.conf' docker.io/nginx
9d4b70a15645b42c16e6011d29a350ae2489943bc6068d4be70ca1d9c5940dd5

[root@dealloc-test nginx]# docker container inspect nginx-test | jq '.[0].NetworkSettings.Networks | .[].IPAddress'
"172.18.0.2"

I don't think Docker is doing MAC-based DHCP leases. Based on above, in Docker, you can create and and destroy a container - which will assign a new MAC to it - but Docker will still (most of the time) reassign its old IP to it. If it sees 172.18.0.2 was assigned previously, but is no longer in use, it prefers to use that before using 172.18.0.3.

It feels like Docker's IPAM will prefer to be conservative and reuse deallocated and not-in-use IPs, before moving on to ones that have never been allocated. Netavark doesn't.

---

This isn't a leak in IPAM - Netavark allows assigning a static IP that was previously assigned and not in use - but it won't assign those IPs again on its own until the range pool is depleted.

[root@dealloc-test nginx]# dig +short nginx-test @10.89.0.1
10.89.0.3

[root@dealloc-test nginx]# podman run -d --name=nginx-test-2 --ip=10.89.0.2 --network=test -v './nginx.conf:/etc/nginx/nginx.conf'
91b9cbf94b64740bef6d62e96f96bb83f3cfa434f1c174cb4758130e386c0aba

[root@dealloc-test nginx]# dig +short nginx-test-2 @10.89.0.1
10.89.0.2

---

There was some discussion a long time ago about IP stability in Docker - moby/moby#2801 - some of the concerns about aggressively assigning new IPs were mentioned there with regards to DNS.

If this is intended behavior, I guess I'll go over to Aardvark and ask for configurable TTLs - I think 1 day is too high to not be configurable.

[baude]

@Luap99 @flouthoc thoughts? I know this is working as designed ... but the question/observation seems fair enough.

[Luap99]

I never have thought about it but the default TLL seems way to high. AFAIK most dns clients will not cache dns records at all so that's why we haven't seen this come up yet. IMO we can lower the TTL to 60 which should help because even when we try to reuse the IP there will be cases were reuse is not possible.

---

As for the IP resuse part I simply implemented the IPAM in the most simple manner, we can change it but it doesn't look like a trivial change and I don't see this as a priority.

[Luap99]

Note that IPAM is handled in the c/common go code not netavark: https://github.com/containers/common/blob/main/libnetwork/netavark/ipam.go

[JCBird1012]

Yeah, I have a few apps that strictly follow TTLs, so if I need to clear their cache, I have to restart their containers - which can get tiresome - especially when running watchtower which recreates containers all the time.

I’d be perfectly okay with a new TTL of 60 - that seems like a good tradeoff with performance, but still makes this less of an issue.

[Luap99]

containers/aardvark-dns#312 to switch to the 60 seconds TTL