Operators Tab missing & AAP PermissionError on PV (macOS/vfkit)

[rbsq1312]

General information

I am attempting to run the Ansible Automation Platform (AAP) Operator 2.6 on a fresh install of CRC 2.57.0 (OpenShift 4.20.5) on macOS (Apple Silicon). I encountered two significant blocking issues that required manual workarounds to proceed.

Issue 1: OperatorHub/Marketplace Disabled by Default Upon starting the cluster (crc start), the Operators -> OperatorHub menu item is missing from the OpenShift Console sidebar.

Investigation: The marketplace-operator deployment in the openshift-marketplace namespace is scaled to 0 or unmanaged by default in ClusterVersion overrides.

Impact: A user cannot install Operators via the UI without manually patching the ClusterVersion or manually scaling the marketplace deployment (oc scale deployment marketplace-operator --replicas=1).

Issue 2: AAP Controller UI fails (PermissionError on PV) After installing the AAP Operator and creating an AutomationController instance (with projects_persistence: true), the pods start, but the web UI returns a 500 Server Error.

Investigation: The example-controller-task pod fails to populate the static assets during migration.

Root Cause: Attempting to manually run collectstatic reveals a Permission Error on the mounted volume. The pod user (UID 1000) cannot write to the storage volume provided by the default CRC storage class.

Operating System

macOS

Hypervisor

vfkit

Did you run crc setup before crc start?

yes

Running on

Laptop

Steps to reproduce

1. Start CRC on macOS with increased memory: crc config set memory 24576, crc start.
2. Open the OpenShift Console (crc console).
3. Observe Issue 1: The "Operators" -> "OperatorHub" menu item is missing from the sidebar.
4. Attempted Workaround: Manually scaled the marketplace operator (oc scale deployment marketplace-operator -n openshift-marketplace --replicas=1).
5. Result: The pods started, but the OperatorHub menu still did not appear in the console.
6. Installed the "Ansible Automation Platform" Operator (stable-2.6) via CLI (oc apply -f subscription.yaml) since the UI was inaccessible.
7. Created an AutomationController instance in a new namespace (e.g., aap) with projects_persistence: true.
8. Waited for pods to reach Running state.
9. Observe Issue 2: Accessing the Route URL for the controller returns a 500 Error.

CRC version

CRC version: 2.57.0+ae41f6
OpenShift version: 4.20.5
MicroShift version: 4.20.0

CRC status

CRC VM:          Running
OpenShift:       Running (v4.20.5)
RAM Usage:       10.81GB of 25.15GB
Disk Usage:      35.71GB of 53.08GB (Inside the CRC VM)
Cache Usage:     40.22GB
Cache Directory: /Users/ted/.crc/cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 50
- memory                                : 24576
- pull-secret-file                      : /Users/ted/Downloads/pull-secret.txt

Host Operating System

ProductName:            macOS
ProductVersion:         26.1
BuildVersion:           25B78

Expected behavior

1. The "OperatorHub" menu should be visible by default after a fresh crc start.
2. The AAP Automation Controller login page should load successfully without a 500 Server Error.

Actual behavior

1. The "OperatorHub" menu is missing from the sidebar. Even after manually enabling and scaling the marketplace-operator deployment, the menu remains invisible, forcing a CLI installation.
2. The AAP Controller UI crashes with a 500 Server Error. The web container logs show TemplateDoesNotExist: index.html. Manual investigation confirms a PermissionError [Errno 13] when the pod (UID 1000) tries to write to the PVC storage, preventing static assets from being generated.

CRC Logs

# Log 1: Web Container crashing (from `oc logs -n aap deployment/example-controller -c web`)
django.template.exceptions.TemplateDoesNotExist: index.html

# Log 2: Manual collectstatic attempt failing (from `oc exec`)
ted@MacBook-Pro ~ % oc exec -n aap -it example-controller-task-646fcfb5c8-v22km -- awx-manage collectstatic --noinput --clear
...
Deleting 'api/api.css'
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/files/storage/filesystem.py", line 158, in delete
    os.remove(name)
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/public/static/api/api.css'
command terminated with exit code 1

Additional context

No response

[rbsq1312]

Update: Deep Dive & Workaround Findings

I have performed further debugging to isolate the root cause and found that the permission issue is strictly enforced by the underlying storage driver/mount, even against Root.

1. Confirmation of Permission Lock (Root Access Failed)

I attempted to fix the permissions manually by opening a root shell on the node and pod. Even as root (UID 0), the filesystem rejected modification attempts on the persistent volume.

oc debug pod/example-controller-task-xxxxx -n aap --as-user=0 -c example-controller-task
sh-5.1# chmod -R 777 /var/lib/awx/public/static 

Result:
chmod: changing permissions of '/var/lib/awx/public/static': Operation not permitted

This confirms that the volume mount provided by CRC on macOS is strictly locked preventing the application (UID 1000) from initializing the UI files.

2. Partial Workaround (Ephemeral Mode)

I attempted to bypass the PV issue by disabling persistence in the AutomationController spec:
spec:
projects_persistence: false

Result: This allowed the migration pod to successfully complete (database initialized), confirming the engine works.

New Issue: The Web pod failed to load the UI (500 Server Error) because the static assets were generated on the migration pod's ephemeral drive and were lost upon completion.

3. Manual Restoration of UI

To get the UI working, I had to manually "teleport" the static assets from the Task pod to the Web pod and fix a broken symbolic link.

Findings:

The collectstatic command eventually ran but resulted in index.html being a broken symlink pointing to ../../testpage/index.html.
The actual UI source files exist in /var/lib/awx/venv/awx/lib/python3.11/site-packages/awx/ui.

Successful Workaround Steps:

Deleted the broken index.html symlink.
Manually injected a basic index.html with a meta-refresh redirect to /api/login/.

Confirmed: The backend API (/api/v2/) is fully functional, and logging in via the manual redirect works.

Conclusion: The default storage configuration on macOS CRC prevents the standard installation logic from writing UI assets to the shared volume, breaking the out-of-the-box experience.

[praveenkumar]

@rbsq1312 Thank you for detail issue and workaround.

Operator tab is missing issue is something we are already aware of and openshift/console#15803 suppose to resolve it once backported to release-4.20 branch and after bundle update. It is mostly because depend on extra capability which is not part of default crc.

About storage issue we are using https://github.com/crc-org/snc/tree/release-4.20/kubevirt-hostpath-provisioner-csi so we need to figure out if there is some missing config on that resource which causing this issue.

[anjannath]

hey @rbsq1312 i tried to reproduce the issue you faced, but looking in the docs for AAP operator installation it doesn't mention about creating a AutomationController manually

instead it asks to create a AnsibleAutomationPlatform object, so i followed those steps and later modified the AutomationController object the operator created using oc edit AutomationController aap-platform-controller to set projects_persistence: true

% oc get AutomationController
NAME                      AGE
aap-platform-controller   3m1s
% oc edit AutomationController aap-platform-controller
automationcontroller.automationcontroller.ansible.com/aap-platform-controller edited

after this i was able to get to the login page for the ansible hub, do you have a specific requirement where you need to manually create the AutomationController?

about the CSI driver included in OpenShift Local, the ansible operator docs mention that the storage class should support the RWM (read-write-many) access mode, but the CSI driver used indeed doesn't support that access mode, so there are limitations, ref: kubevirt/hostpath-provisioner#72

[rbsq1312]

Hello gents,

Thanks for the suggestion. I applied the fix, and I can confirm that the Web Interface is now loading and accessible, so that part is resolved.

However, I am now completely locked out of the system due to an authentication/migration deadlock:

Default Credentials Fail: The standard admin credentials are not working on the login page.

Cannot Reset/Create User: I attempted to create a new superuser (or reset the admin password) directly via the awx-manage shell in the pod, but it fails because the service thinks it is still migrating.

It seems the system is stuck waiting for migrate_service_data to complete, which locks the authentication service.

Steps attempted: I identified the web pod and tried running the standard superuser creation command:

WEB_POD=$(oc get pods -n aap | grep "web" | awk '{print $1}' | head -n 1)

oc exec -it $WEB_POD -n aap -- awx-manage createsuperuser

Resulting Error: It returns a 423 Client Error: Locked, preventing any user changes.

2025-12-12 20:13:59,546 ERROR    [-] ansible_base.resource_registry.utils.sync_to_resource_server Failed to sync create of resource emergency (dc8c4fdd-4281-4132-aa19-5ba3460e97a7) to resource server: 423 Client Error: Locked for url: http://aap/api/gateway/v1/service-index/resources/
Response content: {"detail":"Service authentication is locked until migrate_service_data is complete."}

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/utils/sync_to_resource_server.py", line 111, in sync_to_resource_server
    response = client.create_resource(body)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/rest_client.py", line 121, in _make_request
    raise requests.exceptions.HTTPError(f"{e}\nResponse content: {content}", response=resp) from None
requests.exceptions.HTTPError: 423 Client Error: Locked for url: http://aap/api/gateway/v1/service-index/resources/
Response content: {"detail":"Service authentication is locked until migrate_service_data is complete."}

Has anyone seen this state where the UI works, but the backend is indefinitely locked in migrate_service_data? Is there a flag to force-mark the migration as complete?