Example run commands in Dockerhub readme don't work

[jarrett]

Summary: The example run commands in the Dockerhub readme don't work, but they can be fixed by adding or discussing the --privileged or --ulimit flags.

The Dockerhub readme suggests the following docker run commands:

docker run -d --name my-running-haproxy --sysctl net.ipv4.ip_unprivileged_port_start=0 my-haproxy
# Or:
docker run -d --name my-running-haproxy \
  -v /path/to/etc/haproxy:/usr/local/etc/haproxy:ro --sysctl net.ipv4.ip_unprivileged_port_start=0 haproxy:2.3

When running either of these commands, HAProxy crashes with the following error message:

[ALERT] 040/170647 (8) : [haproxy.main()] Cannot raise FD limit to 8030, limit is 1024.

Running Docker in privileged mode lifts the 1024 limit on file descriptors. So that's one possible way to get past this error:

docker run -d --name my-running-haproxy --sysctl net.ipv4.ip_unprivileged_port_start=0 --privileged my-haproxy
# Or:
docker run -d --name my-running-haproxy \
  -v /path/to/etc/haproxy:/usr/local/etc/haproxy:ro --sysctl net.ipv4.ip_unprivileged_port_start=0 --privileged haproxy:2.3

Perhaps the readme should be updated to either:

1. discuss the --privileged and --ulimit flags, their pros, their cons, and alternatives; or
2. simply add --privileged to the example docker run commands.

Option 1 is probably better so as not to encourage use of --privileged without knowledge of the consequences.

I tested this against haproxy:latest and haproxy:alpine, which as of today map to HAProxy 2.3.5.

[wglambert]

I'm not able to reproduce, could be something with the host's environment or something specific to the haproxy configuration

Using http://git.haproxy.org/?p=haproxy-2.3.git;a=blob_plain;f=examples/content-sw-sample.cfg

$ docker run -d --name haproxy --sysctl net.ipv4.ip_unprivileged_port_start=0 haproxy:test
09a55ad5f31c7bd68c117d5044eebd32ee5b68c71a12cbf824c9350fa27020ad

$ docker logs -f haproxy
[NOTICE] 040/181411 (1) : New worker #1 (9) forked
[WARNING] 040/181411 (9) : Server static/statsrv1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 36ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 040/181411 (9) : Server dynamic/dynsrv1 is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 26ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 040/181412 (9) : Server dynamic/dynsrv2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 79ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 040/181412 (9) : Server dynamic/dynsrv3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 135ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 040/181412 (9) : Server static/statsrv2 is DOWN, reason: Layer4 timeout, check duration: 1000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[NOTICE] 040/181412 (9) : haproxy version is 2.3.5-5902ad9
[NOTICE] 040/181412 (9) : path to executable is /usr/local/sbin/haproxy
[ALERT] 040/181412 (9) : backend 'static' has no server available!
[WARNING] 040/181413 (9) : Server dynamic/dynsrv4 is DOWN, reason: Layer4 timeout, check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 040/181413 (9) : backend 'dynamic' has no server available!
^C

$ docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS     NAMES
09a55ad5f31c   haproxy:test   "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes             haproxy

$ docker exec -it haproxy sh
# ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              unlimited
nofiles              1048576
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

[yosifkit]

Yeah, this is not specific to the haproxy image, but is a host configuration problem. The ulimit on open files is related to how your Docker daemon is running and configured: https://docs.docker.com/engine/reference/commandline/dockerd/#default-ulimit-settings

--privileged is definitely not the correct workaround, since it removes much of the isolation/protection of containers. But ulimits are easily adjusted per container if your default is too low:

$ docker run -it --rm --ulimit nofile=123456 debian
root@e33d81fec4ac:/# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 255577
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 123456
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[jarrett]

You're right @yosifkit, --ulimit is the better solution. With that in mind, what are your thoughts on adding the below to the readme?

Although one could argue that teaching users about Docker's behavior is outside the scope of this readme, I think it might make pragmatic sense to do it here. For one thing, Googling HAProxy's error message doesn't yield what is actually a simple answer. Additionally, the error message's terminology doesn't align with Docker's—the user would have to know that "FD limit" and "U limit" both relate to the same thing.

File Descriptor Limits

Depending on its configuration, the Docker daemon may limit the number of file descriptors available to containers. If this limit is below what HAProxy requires, HAProxy may crash with a message similar to the following:

Cannot raise FD limit to 8030, limit is 1024.

To resolve this error, you can raise the limit imposed by Docker with the --ulimit flag on docker run:

docker run --ulimit nofile=9000 [other arguments] haproxy