Openshift Image Registry cannot write to persistent volume

[ALL-SPACE-Anas]

Hi,

I have a NFS server that I've been using to create persistent volumes for OKD. I've created one for openshift image registry with the following permissions within the NFS server:

ls -al

drwxrws--- 2 root root 2 May 3 14:47 openshift-image-registry

I'm using OKD 4.15 baremetal and followed the steps in the documentation to deploy the registry. It was working nicely and was writing to the persistent volume, which out of the blue, it has stopped doing. I can't think of anything that could've caused the issue. I have tried with a debug pod to see the container user groups and I can see that it's part of "root" group.

id

uid=1000330000(1000330000) gid=0(root) groups=0(root),1000330000

Despite this, when trying to access the persistent volume from within the container, I get permission denied.

cd /registry/

sh: cd: /registry/: Permission denied

I am using podman to manage images. When trying podman push dummy-registry.route/ns-name/image, I get the following:

Getting image source signatures
Error: trying to reuse blob sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820 at destination: failed to read from destination repository ns-name/image: 500 (Internal Server Error)

Pod logs show the following errors:

time="2024-05-03T13:59:34.533032489Z" level=error msg="unable to find blob sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820: filesystem: stat /registry/docker/registry/v2/blobs/sha256/d4/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820/data: permission denied" go.version="go1.20.12 X:strictfipsruntime" http.request.host=dummy-registry.route http.request.id=092639be-a486-477c-913b-5bed70544903 http.request.method=HEAD http.request.remoteaddr=123.45.67.89 http.request.uri="/v2/ns-name/image/blobs/sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820" http.request.useragent="containers/5.16.0 (github.com/containers/image)" openshift.auth.user="user name" openshift.auth.userid=754993a4-6cc9-416e-993a-f6be7a4b888e vars.digest="sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820" vars.name=ns-name/image
time="2024-05-03T13:59:34.533215799Z" level=error msg="response completed with error" err.code=unknown err.detail="filesystem: stat /registry/docker/registry/v2/blobs/sha256/d4/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820/data: permission denied" err.message="unknown error" go.version="go1.20.12 X:strictfipsruntime" http.request.host=dummy-registry.route http.request.id=092639be-a486-477c-913b-5bed70544903 http.request.method=HEAD http.request.remoteaddr=123.45.67.89 http.request.uri="/v2/ns-name/image/blobs/sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820" http.request.useragent="containers/5.16.0 (github.com/containers/image)" http.response.contenttype=application/json http.response.duration=10.516451ms http.response.status=500 http.response.written=253 openshift.auth.user="user name" openshift.auth.userid=754993a4-6cc9-416e-993a-f6be7a4b888e vars.digest="sha256:d4fc045c9e3a848011de66

Which as far as I can tell, is pointing to the same issue of persistent volume not accessible anymore.

I've tried deleting everything including the deployment, service, route, persistent volume and persistent volume claims; and then rebuilding the registry again but no luck. Please let me know what other info I need to provide.

Thanks,

[ALL-SPACE-Anas]

It was an internal issue of a config change that didn't persist across reboot in the nfs config.