GPU operator issue on OKD 4.7 with fedora coreos 34

[takyon77]

Hi,

I'm trying to install the GPU operator on a cluster OKD 4.7 running on fedora coreos 34 with the kernel 5.12.7-300.fc34.x86_64.

I'm building the driver image using https://gitlab.com/zvonkok/driver. After some work and bug correction, I was able to compile the nvidia kernel module but then it fails to insert in the kernel with the error message: 'Permission denied'.
I tried also with a basic hello world kernel module and I have the same error so apparently, I don't have the right to insert any module in the kernel from a container which is using a privileged SCC.
I was able to run the GPU operator in OKD 4.6 using a similar installation method so apparently something has changed in OKD 4.7.
I did some googling but I was not able to find any documentation on how to deal with kernel modules inside containers which is probably not a good sign ( Or I'm bad at googling which is totally possible...)

Is there a special way to insert a module in the kernel from a container in OKD 4.7? Am I doing something stupid here? Is there anyone here who managed to run the GPU operator on OKD 4.7?

Thanks,

Chris.

[takyon77]

So for anyone interested, I managed to solve the issue. It was in fact caused by a selinux 'bug' on the host running the container. Apparently a new lockdown class was introduced for selinux in Fedora 34 and this was an unwanted side-effect (https://www.spinics.net/lists/fedora-devel/msg284833.html). I just created a selinux policy on the host running the container to go around the problem and now I can insert kernel modules from the container.

[Sahun123]

@takyon77 Having the same issue with error message "Permission denied" on OKD 4.7 with kernel 5.11.20-300.fc34.x86_64. How can i create the selinux policy on the host running the container?

[takyon77]

@Sahun123 I just installed setroubleshoot-server on the server where I was trying to do the insmod. I think this is how I did the install:

curl https://download-ib01.fedoraproject.org/pub/fedora/linux/updates/34/Everything/x86_64/Packages/l/libxml2-2.9.12-4.fc34.x86_64.rpm -o libxml2-2.9.12-4.fc34.x86_64.rpm
rpm-ostree override replace ./libxml2-2.9.12-4.fc34.x86_64.rpm
systemctl reboot
rpm-ostree install setroubleshoot-server python3-six
systemctl reboot

Once this is done you will be able to see the SElinux error in the logs using 'journalctl -t setroubleshoot'. The log will tell you how to solve the problem which should be something like:

ausearch -c 'modprobe' --raw | audit2allow -M modprode
semodule -X 300 -i modprode.pp

Of course you will then have to copy the pp file and create the SElinux policy on all your GPU nodes.

Hope this help.