These days we are doing some maintenance tasks on hundreds of RHEL servers and we update them using a shell script. We failed to update part of them and after I fixed the failure I would like to share the solution here.
Usually after the update script was run, we would reboot the servers to use the latest kernel and we found the failure was caused by the terminated yum update command abnormally. In other words, we rebooted the servers too early.
Run the command 'package-cleanup --dupes' and we would get the rpm packages with multiple versions:
- [root@xxxxxx ~]# package-cleanup --dupes
- Loaded plugins: fastestmirror
- firewalld-filesystem-0.5.3-5.el7.noarch
- firewalld-filesystem-0.6.3-2.el7_7.1.noarch
- grub2-common-2.02-0.76.el7.1.noarch
- grub2-common-2.02-0.80.el7.noarch
- libipa_hbac-1.16.2-13.el7_6.8.x86_64
- libipa_hbac-1.16.4-21.el7.x86_64
- audit-libs-2.8.5-4.el7.x86_64
- audit-libs-2.8.4-4.el7.x86_64
- libsss_autofs-1.16.2-13.el7_6.8.x86_64
- libsss_autofs-1.16.4-21.el7.x86_64
- python-firewall-0.6.3-2.el7_7.1.noarch
- python-firewall-0.5.3-5.el7.noarch
- grub2-tools-2.02-0.76.el7.1.x86_64
- grub2-tools-2.02-0.80.el7.x86_64
- bind-license-9.11.4-9.P2.el7.noarch
- bind-license-9.9.4-74.el7_6.1.noarch
- samba-common-4.8.3-4.el7.noarch
- samba-common-4.9.1-6.el7.noarch
- libsss_sudo-1.16.4-21.el7.x86_64
- libsss_sudo-1.16.2-13.el7_6.8.x86_64
- nss-3.36.0-7.1.el7_6.x86_64
- nss-3.44.0-4.el7.x86_64
- python-sssdconfig-1.16.2-13.el7_6.8.noarch
- python-sssdconfig-1.16.4-21.el7.noarch
- libsss_idmap-1.16.4-21.el7.x86_64
- libsss_idmap-1.16.2-13.el7_6.8.x86_64
- NetworkManager-libnm-1.12.0-10.el7_6.x86_64
- NetworkManager-libnm-1.18.0-5.el7_7.1.x86_64
- grub2-pc-modules-2.02-0.80.el7.noarch
- grub2-pc-modules-2.02-0.76.el7.1.noarch
- dracut-033-554.el7.x86_64
- dracut-033-564.el7.x86_64
- sssd-client-1.16.2-13.el7_6.8.x86_64
- sssd-client-1.16.4-21.el7.x86_64
- plymouth-core-libs-0.8.9-0.31.20140113.el7.x86_64
- plymouth-core-libs-0.8.9-0.32.20140113.el7.x86_64
- libcom_err-1.42.9-13.el7.x86_64
- libcom_err-1.42.9-16.el7.x86_64
- dhcp-libs-4.2.5-77.el7.x86_64
- dhcp-libs-4.2.5-68.el7_5.1.x86_64
- openssl-libs-1.0.2k-16.el7_6.1.x86_64
- openssl-libs-1.0.2k-19.el7.x86_64
- selinux-policy-3.13.1-229.el7_6.12.noarch
- selinux-policy-3.13.1-252.el7.1.noarch
- libss-1.42.9-16.el7.x86_64
- libss-1.42.9-13.el7.x86_64
- openssh-7.4p1-21.el7.x86_64
- openssh-7.4p1-16.el7.x86_64
- dhcp-common-4.2.5-77.el7.x86_64
- dhcp-common-4.2.5-68.el7_5.1.x86_64
- grub2-tools-minimal-2.02-0.80.el7.x86_64
- grub2-tools-minimal-2.02-0.76.el7.1.x86_64
- e2fsprogs-libs-1.42.9-13.el7.x86_64
- e2fsprogs-libs-1.42.9-16.el7.x86_64
We could remove some of them using 'package-cleanup --cleandupes' command and would get errors for others:
- [root@xxxxxx ~]# package-cleanup --cleandupes
- Loaded plugins: fastestmirror
- Not removing bind-license-9.9.4-74.el7_6.1.noarch because it is required by 2 installed packages
- Not removing libsss_idmap-1.16.2-13.el7_6.8.x86_64 because it is required by 1 installed package
- Not removing dracut-033-554.el7.x86_64 because it is required by 2 installed packages
- Not removing plymouth-core-libs-0.8.9-0.31.20140113.el7.x86_64 because it is required by 1 installed package
- Not removing libsss_autofs-1.16.2-13.el7_6.8.x86_64 because it is required by 1 installed package
- Not removing libss-1.42.9-13.el7.x86_64 because it is required by 1 installed package
- Not removing grub2-pc-modules-2.02-0.76.el7.1.noarch because it is required by 1 installed package
- Not removing libipa_hbac-1.16.2-13.el7_6.8.x86_64 because it is required by 1 installed package
- Not removing sssd-client-1.16.2-13.el7_6.8.x86_64 because it is required by 1 installed package
- Not removing selinux-policy-3.13.1-229.el7_6.12.noarch because it is required by 2 installed packages
- Not removing grub2-common-2.02-0.76.el7.1.noarch because it is required by 2 installed packages
- Not removing grub2-tools-2.02-0.76.el7.1.x86_64 because it is required by 2 installed packages
- Not removing python-firewall-0.5.3-5.el7.noarch because it is required by 1 installed package
- Not removing dhcp-common-4.2.5-68.el7_5.1.x86_64 because it is required by 1 installed package
- Not removing openssl-libs-1.0.2k-16.el7_6.1.x86_64 because it is required by 1 installed package
- Not removing firewalld-filesystem-0.5.3-5.el7.noarch because it is required by 1 installed package
- Not removing libsss_sudo-1.16.2-13.el7_6.8.x86_64 because it is required by 1 installed package
- Not removing nss-3.36.0-7.1.el7_6.x86_64 because it is required by 1 installed package
- Not removing e2fsprogs-libs-1.42.9-13.el7.x86_64 because it is required by 1 installed package
- Not removing openssh-7.4p1-16.el7.x86_64 because it is required by 2 installed packages
- Not removing dhcp-libs-4.2.5-68.el7_5.1.x86_64 because it is required by 1 installed package
- Not removing NetworkManager-libnm-1.12.0-10.el7_6.x86_64 because it is required by 2 installed packages
- Not removing grub2-tools-minimal-2.02-0.76.el7.1.x86_64 because it is required by 2 installed packages
- Not removing samba-common-4.8.3-4.el7.noarch because it is required by 6 installed packages
- Not removing libcom_err-1.42.9-13.el7.x86_64 because it is required by 1 installed package
- Not removing python-sssdconfig-1.16.2-13.el7_6.8.noarch because it is required by 1 installed package
- ......................
At the first time I tried to resolve these conflict one by one. For instance, the package dhcp-common-4.2.5-68.el7_5.1.x86_64 could not be removed and I tried to know what package blocked this action. I run 'yum remove dhcp-common-4.2.5-68.el7_5.1.x86_64' but input 'n' to interrupt such removal and found package dhclient was the reason.
In fact, if the update script run well, there should be just one version, the newer one, packages on the system. But it was terminated, so not all the old packages were updated and removed. For this reason, I updated the dependent package dhclient to latest version, then run 'package-cleanup --cleandupes' command again and it let me to remove the old dhcp-common-4.2.5-68.el7_5.1.x86_64.
Using the above way most conflicts could be resolved except the glibc-common package.
I run below commands to confirm the newer version was installed correctly, then removed the old one and updated the dependent glibc.i686.
- [root@xxxxxx yum.repos.d]# rpm -V glibc-common-2.17-260.el7_6.6.x86_64
- S.5...... /usr/bin/gencat
- S.5...... /usr/bin/getconf
- S.5...... /usr/bin/getent
- S.5...... /usr/bin/iconv
- S.5...... /usr/bin/locale
- S.5...... /usr/bin/localedef
- S.5...... /usr/bin/makedb
- S.5...... /usr/bin/pldd
- S.5...... /usr/bin/rpcgen
- S.5...... /usr/bin/sprof
- S.5...... /usr/sbin/build-locale-archive
- S.5...... /usr/sbin/zdump
- S.5...... /usr/sbin/zic
- S.5...... d /usr/share/doc/glibc-common-2.17/ChangeLog.bz2
- S.5...... /usr/share/i18n/charmaps/IBM273.gz
- [root@xxxxxx yum.repos.d]# rpm -V glibc-common-2.17-292.el7.x86_64
- [root@xxxxxx yum.repos.d]# rpm -e --justdb glibc-common-2.17-260.el7_6.6.x86_64
- error: Failed dependencies:
- glibc-common = 2.17-260.el7_6.6 is needed by (installed) glibc-2.17-260.el7_6.6.i686
- [root@xxxxxx yum.repos.d]# rpm -e --justdb --nodeps glibc-common-2.17-260.el7_6.6.x86_64
- [root@xxxxxx yum.repos.d]# yum update glibc.i686
Finally I could run our update script successfully.
