Pihole won't start in docker: Failed to set capabilities on file `/usr/bin/pihole-FTL' (Operation not supported)

[JohnDoeIsKIng]

Since a week I'm trying to get pihole:latest running on a docker/portainer combination on DSM7.
I updated from DSM6, portainer 1.24 and a very old pihole (5.*).

Details

I've been trying portainer (marked as deprecated) and portainer-ce on docker. While starting, first I got the error

ERROR: Unable to set capabilities for pihole-FTL. Cannot run as non-root.
If you are seeing this error, please set the environment variable 'DNSMASQ_USER' to the value 'root'

Doing so, now I'm running into

Failed to set capabilities on file `/usr/bin/pihole-FTL' (Operation not supported)
The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file

System info:

Linux DS918plus 4.4.180+ #42661 SMP Fri Apr 1 15:33:06 CST 2022 x86_64 GNU/Linux synology_apollolake_918+

Related Issues

• I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar

#963

How to reproduce the issue

1. Environment data

• Operating System: DSM 7.1-42661 Update 1
• Hardware: Synology DS918+
• Kernel Architecture: x86_64
• Docker Install Info and version:
  • Software source: docker.io
  • Supplimentary Software: Portainer 1.24 and 2
• Hardware architecture: x86

2. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here
4. any additional info to help reproduce

I've tried pihole releases back to 2021.9 - none started without DSNMASQ_USER=root and none did run pihole-FTL.
I've tried with portainer and portainer-ce.
Before any creation of the container from scratch I deleted my volume binds and recreated them empty.

These common fixes didn't work for my issue

• I have tried removing/destroying my container, and re-creating a new container
• I have tried fresh volume data by backing up and moving/removing the old volume data
• I have tried running the stock docker run example(s) in the readme (removing any customizations I added)
• I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me)
• I have tried running without my volume data mounts to eliminate volumes as the cause

If the above debugging / fixes revealed any new information note it here.
Add any other debugging steps you've taken or theories on root cause that may help.

[rdwebdesign]

Please, what is the output of docker version command?

[JohnDoeIsKIng]

Thanks for replying!

Client:
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.17.1
 Git commit:        55f0773
 Built:             Wed Feb  9 04:04:10 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.1
  Git commit:       b487c8f
  Built:            Wed Feb  9 04:04:31 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        3fa00912415f3e9c6f82dd72119179d599efd13b
 runc:
  Version:          v1.0.0-rc93
  GitCommit:        31cc25f16f5eba4d0f53e35374532873744f4b31
 docker-init:
  Version:          0.19.0
  GitCommit:        ed96d00

Hope it helps.
Best regards

JD.

[JohnDoeIsKIng]

Update:

I tried something on the console:

root@pihole:/# getcap /usr/bin/pihole-FTL 
Failed to get capabilities of file '/usr/bin/pihole-FTL' (Operation not supported)

I guess if getting file caps isn't possible setting them is neither.
An inspection of the container showed caps AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, MK added and AUDIT_CONTROL, BLOCK_SUSPEND, DAC_READ_SEARCH, IPC_LOC dropped. Some findings on the internet showed a relation of the error to the NET_BIND_SERVICE cap. Maybe somebody could test this as I'm not able to do so ?
Best regards

JD.

[rdwebdesign]

If you are using Portainer, you can set them using "Capabilities" tab, in Portainer web interface.

Pi-hole image should run using this capabilities:

Note:
You should disable "Privileged mode" option, if it's enabled.

[JohnDoeIsKIng]

Thanx for replying,

I've set the caps exactly as in your hint - no change, pihole-FTL doesn't run.
Also I've read before about the privileged mode and so I've never had it turned on.
Do you eventually know an image version that 's supposed to run in docker on a synology ?
Regards

JD.

[rdwebdesign]

Sorry, I can't answer that. I never used a Synology device.

You could try to ask this at our Discourse Forum.

[dschaper]

Have you tried without Portainer? I'll try to set up a similar configuration on my DS, I need to update to v7 though.

[dschaper]

Can you post your environment settings?

[JohnDoeIsKIng]

Have you tried without Portainer? I'll try to set up a similar configuration on my DS, I need to update to v7 though.

No, I haven't yet.
Here 's my environment:

Best regards

JD.

[JohnDoeIsKIng]

Update 2:

I've searched around further and found this:
alexbers/mtprotoproxy#157 (comment)
Could that be the cause for my problem ?
Because on my system:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.3
 Storage Driver: aufs
  Root Dir: /volume1/@docker/aufs
  Backing Filesystem: extfs
  Dirs: 16
  Dirperm1 Supported: true
 Logging Driver: db
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs db fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3fa00912415f3e9c6f82dd72119179d599efd13b
 runc version: 31cc25f16f5eba4d0f53e35374532873744f4b31
 init version: ed96d00 (expected: de40ad0)
 Security Options:
  apparmor
 Kernel Version: 4.4.180+
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.688GiB
 Name: DS918plus
 ID: GAJ3:THYJ:AK7H:EJHU:NYEF:ZFOS:FBCU:B4VU:NP4R:AWCZ:XYRI:B62F
 Docker Root Dir: /volume1/@docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

I've tried the manual from the link https://docs.docker.com/storage/storagedriver/overlayfs-driver/, but my docker-fs is still aufs.
Eventually somebody with a running pihole on docker could check if the kernel config of the pihole-container has

CONFIG_AUFS_XATTR=y

Regards

JD.

[dschaper]

Yes, if AUFS does not allow capabilities to be set then there will be issues starting.

And we don't configure the kernel in the image, it uses the kernel of the host OS.

[JohnDoeIsKIng]

Okay, that means for me pihole on docker/synology isn't an option any more.
As a last try I've edited

/var/packages/Docker/etc/dockerd.json

hoping to change the storage driver from aufs to overlay2 or btrfs but nothing changed - driver is still aufs.
So I'll remove pihole from docker and probably install it on another hardware.
Sad but true.
May I'll try to set it up without portainer as in https://discourse.pi-hole.net/t/setup-on-synology-docker/18067.
Regards

JD.

[dschaper]

There are a number of users that have Pi-hole running on Synology. Do you know what filesystem you have the Synology set to use?

My Synology Docker setup shows 'btrfs' since I'm using the HybridRaid format with my 2 disk unit. It sounds like you may have ext4 format which ends up as 'aufs' under Docker.

[JohnDoeIsKIng]

You've supposed right - All of the hdds have ext4-fs. I'm wondering where the issue comes from since before the update to DSM7.1-42661 there weren't any issues with cap settings.
I guess if I have to change one hdd to btrfs that I have to format that one and remove temporarily the data on it?

[rdwebdesign]

... I'm wondering where the issue comes from since before the update to DSM7.1-42661 there weren't any issues with cap settings.

I'm not sure, but this could be a combination of different things.

Recently (before v20.10.14), docker discovered a bug (GHSA-2mm7-x5h6-5pvq) and changed the way capabilities are inherited.
To address this change, we changed how pi-hole images handle capabilities. We also changed from buster base to bullseye base image.

Any of these changes (or any combination of them) could be the reason why you are seeing different behavior after the upgrade.

[JohnDoeIsKIng]

Dear community,

I've managed to get pihole:latest running on docker/portainer. I've transformed the file system of one hdd into btrfs and it worked nearly out of the box.
So bringing this to an end, I've one last question: I have a copy of my pihole-folder of the old version I used to run. Is it possible to import at least the block lists and my edited white and black lists into the new version ? I have NO teleporter tar file, just the content of the folder:

d---------+ 3 0 0       4096 May  1 17:21 .
d---------+ 4 0 0       4096 May  1 17:26 ..
----------+ 1 0 0        242 May  1 17:21 adlists.list
----------+ 1 0 0        596 May  1 17:21 dns-servers.conf
----------+ 1 0 0         17 May  1 17:21 GitHubVersions
----------+ 1 0 0  139624448 May  1 17:21 gravity.db
----------+ 1 0 0    2501728 May  1 17:21 list.0.raw.githubusercontent.com.domains
----------+ 1 0 0     594616 May  1 17:21 list.1.mirror1.malwaredomains.com.domains
----------+ 1 0 0        521 May  1 17:21 list.2.s3.amazonaws.com.domains
----------+ 1 0 0      43529 May  1 17:21 list.3.s3.amazonaws.com.domains
----------+ 1 0 0   45056207 May  1 17:21 list.4.raw.githubusercontent.com.domains
----------+ 1 0 0         20 May  1 17:21 localbranches
----------+ 1 0 0         31 May  1 17:21 local.list
----------+ 1 0 0         39 May  1 17:21 localversions
d---------+ 2 0 0       4096 May  1 17:21 migration_backup
----------+ 1 0 0          0 May  1 17:21 pihole-FTL.conf
----------+ 1 0 0 1214263296 May  1 17:21 pihole-FTL.db
----------+ 1 0 0        443 May  1 17:21 setupVars.conf
----------+ 1 0 0        443 May  1 17:21 setupVars.conf.update.bak

Do I have to re-initialize some databases ?
Best regards

JD.

[sancoder]

Why not make some extra code to detect Synology just for this bug? Is it too much a deviation?

I was trying to upgrade from pihole installed back in 2019 to something fresher, and bumped into this change of behavior. Before upgrade - pihole was running successfully in docker on Synology, and after upgrading it cannot start because pihole cannot set capabilities on file '/usr/bin/pihole-FTL' just as in the first message of this thread.
Yes, I have ext4 FS, and I don't have an option to reformat all of HDDs in NAS.

Or, introduce an env variable - something like $DONT_FIX_CAPS defaulting to 0 and if a user sets it as 1, then code in fix_capabilities just ignores error coming from setcap call.

[dschaper]

pihole-FTL won't run without the capabilities.

[sancoder]

Error from setcap doesn't mean pihole-FTL not capable to do what asked.

[PromoFaux]

@sancoder - There were some changes made in #1086 that might (strong might) fix the issues you're seeing

Have just tagged 2022.05, which is just building (though if you're super desperate to try it before it builds, you can try the :dev tag

[sancoder]

@PromoFaux thanks, still not working as non-root (adding DNSMASQ_USER=root works).

pihole2022 | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
pihole2022 | [s6-init] ensuring user provided files have correct perms...exited 0.
pihole2022 | [fix-attrs.d] applying ownership & permissions fixes...
pihole2022 | [fix-attrs.d] 01-resolver-resolv: applying...
pihole2022 | [fix-attrs.d] 01-resolver-resolv: exited 0.
pihole2022 | [fix-attrs.d] done.
pihole2022 | [cont-init.d] executing container initialization scripts...
pihole2022 | [cont-init.d] 05-changer-uid-gid.sh: executing...
pihole2022 | [cont-init.d] 05-changer-uid-gid.sh: exited 0.
pihole2022 | [cont-init.d] 20-start.sh: executing...
pihole2022 |  ::: Starting docker specific checks & setup for docker pihole/pihole
pihole2022 | cap[cap_net_admin] not permitted
pihole2022 | cap[cap_sys_nice] not permitted
pihole2022 | Failed to set capabilities on file `/usr/bin/pihole-FTL' (Operation not supported)
pihole2022 | The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file
pihole2022 | ERROR: Unable to set capabilities for pihole-FTL. Cannot run as non-root.
pihole2022 |        If you are seeing this error, please set the environment variable 'DNSMASQ_USER' to the value 'root'
pihole2022 | [cont-init.d] 20-start.sh: exited 1.
pihole2022 | [cont-finish.d] executing container finish scripts...
pihole2022 | [cont-finish.d] done.
pihole2022 | [s6-finish] waiting for services.
pihole2022 | [s6-finish] sending all processes the TERM signal.
pihole2022 | [s6-finish] sending all processes the KILL signal and exiting.

For others who found this issue because of the error message: if docker is on Synology, please see #963

[b-m-f]

I am having the same issue on a raspberry pi, running pihole with Podman.

From all the approaches I have found only setting the ENV variable DNSMASQ_USER=root seems to do the trick for now.

[dschaper]

I am having the same issue on a raspberry pi, running pihole with Podman.

What versions of the OS and Podman?

[b-m-f]

• Debian 11
• podman 3.0.1