leapp upgrade fails with AttributeError: can't set attribute

[beepmode]

I keep running into the same error when running leapp upgrade. As far as I can tell this happens right after a dnf update command has completed.

Known blockers check

Centos version (minimum is 7.9)

# cat /etc/*-release
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)

Available disk space:

# df -Th
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  3.8G     0  3.8G   0% /dev
tmpfs                   tmpfs     3.9G  8.0K  3.9G   1% /dev/shm
tmpfs                   tmpfs     3.9G   20M  3.8G   1% /run
tmpfs                   tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root ext4       37G  8.2G   29G  23% /
/dev/sda1               ext4      477M  259M  189M  58% /boot
/dev/loop0              ext4      1.1G  1.5M  979M   1% /tmp
tmpfs                   tmpfs     781M     0  781M   0% /run/user/0

cPanel version (minimum is 102)

# /usr/local/cpanel/cpanel -V
106.0 (build 11)

Make sure system is up to date:

# yum check-update
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
 * EA4: 185.15.22.168
 * cpanel-addons-production-feed: 185.15.22.168
 * cpanel-plugins: 185.15.22.168
 * base: uk.mirrors.clouvider.net
 * extras: mirrors.vinters.com
 * updates: uk.mirrors.clouvider.net
 
# /usr/local/cpanel/scripts/upcp
[2022-12-30 16:13:52 +0000]   Detected cron=0 (Terminal detected)
----------------------------------------------------------------------------------------------------
=> Log opened from cPanel Update (upcp) - Slave (59716) at Fri Dec 30 16:13:52 2022
[2022-12-30 16:13:52 +0000]   Detected cron=0 (Terminal detected)
[2022-12-30 16:13:52 +0000]   1% complete
[2022-12-30 16:13:52 +0000]   Running Standardized hooks
[2022-12-30 16:13:52 +0000]   2% complete
[2022-12-30 16:13:52 +0000]   mtime on upcp is 1672050540 (Mon Dec 26 10:29:00 2022)
...
[2022-12-30 16:14:41 +0000]   Pre Maintenance completed successfully
[2022-12-30 16:14:41 +0000]   95% complete
[2022-12-30 16:14:41 +0000]   Running Standardized hooks
[2022-12-30 16:14:41 +0000]   100% complete
[2022-12-30 16:14:41 +0000]
[2022-12-30 16:14:41 +0000]     cPanel update completed
[2022-12-30 16:14:41 +0000]   A log of this update is available at /var/cpanel/updatelogs/update.59716.6579294.1672351039.log
[2022-12-30 16:14:41 +0000]   Removing upcp pidfile
[2022-12-30 16:14:41 +0000]
[2022-12-30 16:14:41 +0000] Completed all updates
 

Current nameserver software (PowerDNS is recommended)

# /scripts/setupnameserver --current
Current nameserver type: powerdns

MariaDB version (minimum is 10.3)

# mysql -V
mysql  Ver 15.1 Distrib 10.3.37-MariaDB, for Linux (x86_64) using readline 5.1

PHP version(s):

# whmapi1 php_get_installed_versions
---
data:
  versions:
    - ea-php74

After reboot

# yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el7.noarch.rpm
# yum install -y leapp-upgrade leapp-data-almalinux
# leapp preupgrade
# leapp upgrade

leapp upgrade then errors out after DNF updates have completed:

Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Process Process-476:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 290, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/dnfpackagedownload/actor.py", line 51, in process
    xfs_info=xfs_info, storage_info=storage_info, plugin_info=plugin_info, on_aws=on_aws
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/dnfplugin.py", line 385, in perform_rpm_download
    test=True, on_aws=on_aws
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/dnfplugin.py", line 219, in _transaction
    e.stdout = e.stdout.encode('utf-8', 'xmlcharrefreplace')
AttributeError: can't set attribute

=====================================================================================================
Actor dnf_package_download unexpectedly terminated with exit code: 1 - Please check the above details
=====================================================================================================

Debug output written to /var/log/leapp/leapp-upgrade.log

============================================================
                           REPORT
============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                       END OF REPORT
============================================================

I've attached the /var/log/leapp/leapp-report.txt file (the preflight check) and /var/log/leapp/leapp-upgrade.log. I can provide more information - just let me know what other info might be useful.
leapp-report.txt
leapp-upgrade.log

[danFbach]

Hello,

If you review the leapp-upgrade.log, you can see that there is a package conflict with python36. A few others and myself have had this same problem. leapp does not do a good job of bringing this conflict to the users attention.

Anyway, this is how i fixed the same issue.

1. yum remove python36*
2. reboot server
3. /scripts/elevate-cpanel --continue

[beepmode]

Thanks for your quick response. Running /scripts/elevate-cpanel --continue didn't work for me (I guess because the process had been killed). I instead restored the VM I'm testing this on. The upgrade does complete when I remove the python36* packages before I run leapp preupgrade.

I should probably have package conflict in the log. In my defence though, the file has over 24k lines!

[danFbach]

I wouldn't really blame yourself. It happened to me while upgrading a production server lol
I've looked at the various leapp github repo issue pages and it looks like they are aware of the bug and the plan is to try to remove the conflicting packages prior to upgrade, but it doesn't appear any progress has been made as the repo issues are months old. 🤷‍♂️
Glad you were able to complete the upgrade.

[atoomic]

@beepmode you have two issues tagged as high as reported by the leapp report

1. kernel-uek package

Risk Factor: high
Title: Packages from unknown repositories may not be installed
Summary: 1 packages may not be installed or upgraded due to repositories unknown to leapp:
- kernel-uek (repoid: ol8-uek)
Remediation: [hint] Please file a bug in http://bugzilla.redhat.com/ for leapp-repository component of the Red Hat Enterprise Linux product.
Key: 9a2b05abf8f45fd7915e52542887bb334bb218ea

2. packages signed

Risk Factor: high
Title: Packages not signed by Red Hat found on the system
Summary: The following packages have not been signed by Red Hat and may be removed during the upgrade process in case Red Hat-signed packages to be removed during the upgrade depend on them:
- MariaDB-client
- MariaDB-common
- MariaDB-compat
- MariaDB-devel
- MariaDB-server
- MariaDB-shared
- cpanel-ace-editor
- cpanel-analog
- cpanel-analytics
- cpanel-angular-chosen
- cpanel-angular-growl-2
- cpanel-angular-minicolors
- cpanel-angular-ui-bootstrap
…

---

The kernel-uekis similar to the issue reported there #148 (comment)
From the leapp-upgrade.log it claims that it cannot find an a8 repo for that kernel-uek package

2022-12-30 16:46:59.106 DEBUG    PID: 22519 leapp.workflow.FactsCollection.pes_events_scanner: REMOVE  libssh2:almalinux8-appstream
2022-12-30 16:46:59.118 WARNING  PID: 22519 leapp.workflow.FactsCollection.pes_events_scanner: Cannot determine the representative target base repository.
2022-12-30 16:46:59.121 INFO     PID: 22519 leapp.workflow.FactsCollection.pes_events_scanner: Fallback: Create an artificial representative PESIDRepositoryEntry for the repository mapping
2022-12-30 16:46:59.125 WARNING  PID: 22519 leapp.workflow.FactsCollection.pes_events_scanner: Cannot find suitable repository for PES ID: ol8-uek
2022-12-30 16:46:59.129 WARNING  PID: 22519 leapp.reporting: Stable Key report entry not provided, dynamically generating one - 9a2b05abf8f45fd7915e52542887bb334bb218ea
2022-12-30 16:46:59.143 WARNING  PID: 22519 leapp.workflow.FactsCollection.pes_events_scanner: Cannot determine the representative target base repository.
2022-12-30 16:46:59.145 INFO     PID: 22519 leapp.workflow.FactsCollection.pes_events_scanner: Fallback: Create an artificial representative PESIDRepositoryEntry for the repository mapping
2022-12-30 16:46:59.183 INFO     PID: 19120 leapp.workflow.FactsCollection: Executing actor setuptargetrepos 

I think you can fix it by removing your ol8-uek repo and probably switching to a different kernel package

---

I think the second issue should not appear if you use the cpanel elevate script, if it does then please open a support ticket so we can access your server and investigate

thanks

[danFbach]

@atoomic Just wanted to make you aware that the ticket i submitted to redhat bugzilla regarding the kernel-uek error did finally get a reply.

They essentially just kicked it back to cpanel / elevate / leapp, stating:

"the reported problems are not present in our component in RHEL, so I am closing the ticket - this bugzilla is supposed to be used only for problems that are related to the component in the RHEL system (or CentOS stream, however, leapp & leapp-repository components are not provided in CentOS, so in this case only RHEL is relevant). In this case, the vendor of the package in your OS is supposed to change the report to point to the right web."

you can view the ticket here if you're so inclined. https://bugzilla.redhat.com/show_bug.cgi?id=2153539

as I also previously stated, it looks like the official leapp repo maintainers are aware of the conflict issues (and have been for awhile) as referenced in AlmaLinux/leapp-repository#22 and AlmaLinux/leapp-repository#15 but haven't taken any steps to fix it yet.

[atoomic]

thanks, I ve adjusted the link above.
Running: yum remove python3* to remove the python3 package seem to have solve the issue in many cases

[Cassius242]

I am having the same issue with the same error message, but I do not have python36 installed on this system.

[danFbach]

It could be a different conflicting package.
Try running less /var/log/leapp/leapp-upgrade.log | grep 'conflicts with file' and post the results back here, if any.

[Cassius242]

@danFbach thanks for the quick reply. My issue ended up being two items.

The first one was several package conflicts not involving python36. Once those were removed then the Leapp upgrade finished without errors.

The second issue was once the Leapp upgrade finished it only installed the kernel boot menu entry in the grub2-efi.cfg version. However, this VM didn't have UEFI enabled so it was using the Legacy BIOS grub2.cfg file during boot time. Once this entry was added it booted into the upgrade kernel and everything finished successfully.

[danFbach]

@Cassius242 you're welcome. Glad I could help and that you were able to complete the upgrade!

[ShinyDhas]

Is there any possibility to stop leapp from removing 3rd party applications during OS upgrade. I want my kubelet, kubectl, crio and kubeadm services not to be removed.

[toddr]

ossibility to stop leapp from removing 3rd party applications during OS upgrade. I want my kubelet, kubectl, crio and kubeadm services not to be removed.

If you have installed those packages from a repo, elevate-cpanel should be detecting those additional repositories and stopping. If you installed them by hand, we don't have a good way to detect them at this time.

[toddr]

Glad your upgrade completed. We have created an internal case to block on python36 early so people do not experience this problem in the future. It should be released in a coming version of elevate-cpanel soon!

[toddr]

#250 added to block on the presence of python36 which confuses leapp