[c9s/c10s][s390x] test ext.config.shared.secex.ensure & reboot failed

[HuijingHei]

Both failed with cpi.sevice failed

harness.go:1872: mach.Start() failed: machine "ee487572-5fef-4e7e-9471-b387566c0bbb" failed basic checks: detected failed or stuck systemd units: some systemd units failed: cpi.service; <nil>

More info:

Sep  1 10:49:16.066321 cpi.service[3029]: Sending CPI data from secure execution Linux guests is disabled. Use --permit-cpi to enable CPI data.
Sep  1 10:49:16.076263 init.scope[1]: cpi.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Sep  1 10:49:16.076372 init.scope[1]: cpi.service: Failed with result 'exit-code'.
Sep  1 10:49:16.076561 init.scope[1]: Failed to start cpi.service - Apply Control Program Identification (CPI).

[dustymabe]

Transferred this issue to the rhel-coreos-config repo.

[HuijingHei]

Can reproduce on s390x with latest build, if add --permit-cpi, then command works, but s390utils-core-2.38.0-2.el9.s390x does not change since the last good 9.0.20250828-0, is there other thing changed?

cosa fetch && cosa build && cosa osbuild ostree
cosa buildextend-secex
cosa kola run --build=latest --arch=s390x --qemu-secex --tag secex --debug --ssh-on-test-failure


[systemd]
Failed Units: 1
  cpi.service
[root@qemu0 core]# systemctl status cpi
_ cpi.service - Apply Control Program Identification (CPI)
     Loaded: loaded (/usr/lib/systemd/system/cpi.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2025-09-03 08:11:59 UTC; 50s ago
    Process: 2711 ExecStart=/lib/s390-tools/cpictl -e (code=exited, status=6)
   Main PID: 2711 (code=exited, status=6)
        CPU: 10ms

Sep 03 08:11:59 localhost systemd[1]: Starting Apply Control Program Identification (CPI)...
Sep 03 08:11:59 localhost cpictl[2711]: Sending CPI data from secure execution Linux guests is disabled. Use --permit-cpi to enable CPI data.
Sep 03 08:11:59 localhost systemd[1]: cpi.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Sep 03 08:11:59 localhost systemd[1]: cpi.service: Failed with result 'exit-code'.
Sep 03 08:11:59 localhost systemd[1]: Failed to start Apply Control Program Identification (CPI).

[root@qemu0 core]# rpm -qf /usr/lib/systemd/system/cpi.service /lib/s390-tools/cpictl
s390utils-core-2.38.0-2.el9.s390x
s390utils-core-2.38.0-2.el9.s390x

[root@qemu0 core]# /lib/s390-tools/cpictl -e
Sending CPI data from secure execution Linux guests is disabled. Use --permit-cpi to enable CPI data.
[root@qemu0 core]# /lib/s390-tools/cpictl -e --permit-cpi

With help of @PeaceRebel , find that it is because of s390utils-core-2.38.0-1.el9.s390x -> s390utils-core-2.38.0-2.el9.s390x, the patch is https://gitlab.com/redhat/centos-stream/rpms/s390utils/-/commit/ee1fce706328281a58cb658d8277ae49b1d907c3, will add comment in the bug https://issues.redhat.com/browse/RHEL-76931

[jcapiitao]

As you mentioned in the JIRA, I think we need to opt-in back to send CPI data (i.e by adding CPI_PERMIT_ON_PVGUEST=1 to /etc/sysconfig/cpi) if we still want to test by interacting with the VM. But that would mean permitting it in the image itself. And if needed, the end user would have to disable it to mitigate the risk addressed by the JIRA.

[HuijingHei]

That makes sense, but now cpi.service is enabled by default and would lead to failed after booted.

[HuijingHei]

To make our tests passed, maybe add a drop-in under /etc/systemd/system/cpi.service.d/setenv.conf:

[Service]
Environment="CPI_PERMIT_ON_PVGUEST=1"

But should we care about the default cpi.service failed scenario?

[jcapiitao]

But that would mean permitting it in the image itself. And if needed, the end user would have to disable it to mitigate the risk addressed by the JIRA.

I amend this statement, we can use ignition during the test to enable CPI_PERMIT_ON_PVGUEST. Thanks @jbtrystram

[jcapiitao]

@HuijingHei I left a comment on your 191f273 commit.
Also, if you plan to test it locally, note that there is another s390x issue currently c.f openshift/os#1731 (comment)

[marmijo]

This was observed today in rhel-10.1 as well after an upgrade to s390utils-base-2.38.0-2.el10.s390x