Linux patch management: How to back out a failed patch
Photo by Nathan Hilton from Pexels
Keeping servers up to date is one of a system administrator's primary responsibilities. However, updates do not always work the way you expect, so it's equally important that you know how to a) revert a patch to get the server back to the previous state and b) apply patches in subsets to get more flexibility.
This article focuses on operating system level patching done with the help of the package management tools YUM/DNF on Red Hat Enterprise Linux (RHEL). For this demonstration, a running RHEL 7.7 server is updated to RHEL 7.9 using yum, and then reverted back to the RHEL 7.7 version. It's always recommended to do periodic backups of your servers and the demonstrations provided in this article are for educational purposes only.
Check the server for pending patches/updates
Unless you're using a systems management tool like Red Hat Satellite, you need to test the servers for any updates manually.
For this article, consider a RHEL 7.7 server:
[root@rhel77 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
Use the check-update command to non-interactively check for outstanding updates on your server:
[root@rhel77 ~]# yum check-update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
NetworkManager.x86_64 1:1.18.4-3.el7 rhel-7-server-rpms
NetworkManager-config-server.noarch 1:1.18.4-3.el7 rhel-7-server-rpms
NetworkManager-libnm.x86_64 1:1.18.4-3.el7 rhel-7-server-rpms
NetworkManager-team.x86_64 1:1.18.4-3.el7 rhel-7-server-rpms
kernel-tools-libs.x86_64 3.10.0-1127.19.1.el7 rhel-7-server-rpms
kexec-tools.x86_64 2.0.15-43.el7 rhel-7-server-rpms
kmod.x86_64 20-28.el7 rhel-7-server-rpms
kmod-libs.x86_64 20-28.el7 rhel-7-server-rpms
kpartx.x86_64 0.4.9-131.el7 rhel-7-server-rpms
krb5-libs.x86_64 1.15.1-46.el7 rhel-7-server-rpms
….
….
….
systemd.x86_64 219-73.el7_8.9 rhel-7-server-rpms
systemd-libs.x86_64 219-73.el7_8.9 rhel-7-server-rpms
systemd-sysv.x86_64 219-73.el7_8.9 rhel-7-server-rpms
tcpdump.x86_64 14:4.9.2-4.el7_7.1 rhel-7-server-rpms
teamd.x86_64 1.29-1.el7 rhel-7-server-rpms
tuned.noarch 2.11.0-8.el7 rhel-7-server-rpms
tzdata.noarch 2020a-1.el7 rhel-7-server-rpms
util-linux.x86_64 2.23.2-63.el7 rhel-7-server-rpms
yum.noarch 3.4.3-167.el7 rhel-7-server-rpms
yum-utils.noarch 1.1.31-54.el7_8 rhel-7-server-rpms
Obsoleting Packages
iwl7260-firmware.noarch 25.30.13.0-76.el7 rhel-7-server-rpms
iwl7265-firmware.noarch 22.0.7.0-72.el7 installed
[ You might also like: Red Hat Insights: Vulnerability management ]
When this article was written, RHEL 7.9 was the most recent version of RHEL 7 and doing a yum update would take this RHEL 7.7 server to RHEL 7.9:
[root@rhel77 ~]# yum -y update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package NetworkManager.x86_64 1:1.18.0-5.el7 will be updated
---> Package NetworkManager.x86_64 1:1.18.8-1.el7 will be an update
---
Complete!
After a successful reboot, check the RHEL version to confirm that the server is updated to 7.9:
[root@rhel77 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
Reverting the updates
First, check the history of the transaction with the yum command:
[root@rhel77 ~]# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
8 | root <root> | 2020-11-01 23:10 | I, O, U | 157 EE
The Action(s) and Altered columns give information about what changes occurred with this transaction.
Action(s):
- I - New package Installed
- O - Package is Obsoleted
- U - Package is Updated
Altered:
- 157 packages were altered
- EE - There were some errors/warnings in the transaction
Using yum history packages-list shows the changes that happened from that package's point of view.
As systemd is the first process started in RHEL versions 7 and above, that package is protected with /etc/yum/protected.d/systemd.conf:
[root@rhel77 ~]# cat /etc/yum/protected.d/systemd.conf
systemd
Revert the last transaction with yum history undo:
[root@rhel77 ~]# yum history undo last
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Undoing transaction 8, from Sun Nov 1 23:10:56 2020
Updated NetworkManager-1:1.18.0-5.el7.x86_64 @?rhel-7-server-rpms
Update 1:1.18.8-1.el7.x86_64 @rhel-7-server-rpms
Updated NetworkManager-config-server-1:1.18.0-5.el7.noarch @?rhel-7-server-rpms
Update 1:1.18.8-1.el7.noarch @rhel-7-server-rpms
Updated NetworkManager-libnm-1:1.18.0-5.el7.x86_64 @?rhel-7-server-rpms
....
Resolving Dependencies
--> Running transaction check
…
--> Finished Dependency Resolution
Error: Trying to remove "systemd", which is protected
Caution: The following process is unsupported and untested. Resolve any issues that arise from the downgrade by recovering from a backup or reinstallation.
Move the yum configuration file, which protects systemd:
[root@rhel77 ~]# mv /etc/yum/protected.d/systemd.conf /etc/yum/protected.d/systemd.conf.bak
Re-run the yum history undo last command:
[root@rhel77 ~]# yum history undo last
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Undoing transaction 8, from Sun Nov 1 23:10:56 2020
....
Skipping the running kernel: kernel-3.10.0-1160.2.2.el7.x86_64
....
Dependencies Resolved
=======================================================================================================================
Package Arch Version Repository Size
=======================================================================================================================
Installing:
iwl7265-firmware noarch 22.0.7.0-72.el7 rhel-7-server-rpms 7.3 M
Removing:
bc x86_64 1.06.95-13.el7 @rhel-7-server-rpms 215 k
iwl7260-firmware noarch 25.30.13.0-79.el7 @rhel-7-server-rpms 87 M
linux-firmware noarch 20200421-79.git78c0348.el7 @rhel-7-server-rpms 394 M
Downgrading:
NetworkManager x86_64 1:1.18.0-5.el7 rhel-7-server-rpms 1.9 M
NetworkManager-config-server noarch 1:1.18.0-5.el7 rhel-7-server-rpms 149 k
NetworkManager-libnm
…
Transaction Summary
=======================================================================================================================
Install 1 Package
Remove 3 Packages
Downgrade 152 Packages
…
Running transaction
Installing : libgcc-4.8.5-39.el7.x86_64 1/308
Installing : 1:grub2-common-2.02-0.80.el7.noarch 2/308
Installing : redhat-release-server-7.7-10.el7.x86_64 3/308
warning: /etc/os-release saved as /etc/os-release.rpmsave
Installing : setup-2.8.71-10.el7.noarch 4/308
warning: /etc/shadow created as /etc/shadow.rpmnew
Installing : 1:grub2-pc-modules-2.02-0.80.el7.noarch
....
yum-utils.noarch 0:1.1.31-52.el7
Complete!
Reboot the server
Check the version of RHEL and the kernel:
[root@rhel77 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[root@rhel77 ~]# rpm -q kernel
kernel-3.10.0-1062.el7.x86_64
kernel-3.10.0-1160.2.2.el7.x86_64
[root@rhel77 ~]# uname -r
3.10.0-1160.2.2.el7.x86_64
The server is using the latest kernel version from RHEL 7.9. You can change the default kernel using grub2-set-default command.
Apply the patch in subsets to get more flexibility of rollback
Pinning RHEL to a specific version is one way you can prevent the system from being updated to the latest minor version. Subscription Manager can be used to set the version of RHEL:
[root@rhel77 ~]# subscription-manager release --list
+-------------------------------------------+
Available Releases
+-------------------------------------------+
7.0
7.1
7.2
7.3
7.4
7.5
7.6
7.7
7.8
7.9
7Server
Check the available updates and repo status before pinning the version:
[root@rhel77 ~]# yum update
...
Transaction Summary
=======================================================================================================================
Install 2 Packages (+2 Dependent packages)
Upgrade 152 Packages
[root@rhel77 ~]# subscription-manager repos --list-enabled
Repo ID: rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL: https://cdn.redhat.com/content/dist/rhel/server/7/latest/$basearch/os
Enabled: 1
Pin the version to RHEL 7.8, clear the yum cache, and check the enabled repos:
[root@rhel77 ~]# subscription-manager release --set=7.8
[root@rhel77 ~]# yum clean all
[root@rhel77 ~]# subscription-manager repos --list-enabled
Repo ID: rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL: https://cdn.redhat.com/content/dist/rhel/server/7/7.8/$basearch/os
Enabled: 1
It's evident from the Repo URL output above that it's now using 7.8 and not latest.
Run yum update and check whether there are changes to the updates available:
[root@rhel77 ~]# yum update
....
Transaction Summary
=======================================================================================================================
Install 2 Packages (+1 Dependent package)
Upgrade 136 Packages
Patching in subsets
The history option with yum uses the transaction to rollback/undo the operation. So, updating the packages as subsets gives the flexibility to roll them back. This is a better approach than updating the system as a whole.
Update the 'device-mapper' package with update device-mapper.
[root@rhel77 ~]# yum update device-mapper
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be updated
--> Processing Dependency: device-mapper = 7:1.02.158-2.el7 for package: 7:device-mapper-libs-1.02.158-2.el7.x86_64
---> Package device-mapper.x86_64 7:1.02.170-6.el7 will be an update
--> Running transaction check
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.170-6.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================================================================
Package Arch Version Repository Size
=======================================================================================================================
Updating:
device-mapper x86_64 7:1.02.170-6.el7 rhel-7-server-rpms 297 k
Updating for dependencies:
device-mapper-libs x86_64 7:1.02.170-6.el7 rhel-7-server-rpms 325 k
Transaction Summary
=======================================================================================================================
Upgrade 1 Package (+1 Dependent package)
Total download size: 621 k
Is this ok [y/d/N]:
Check the history with yum history.
[root@rhel77 ~]# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
2 | root <root> | 2020-11-02 15:25 | Update | 2
1 | System <unset> | 2020-09-14 04:25 | Erase | 1 EE
history list
Get more specific details about transaction ID 2 using yum history info 2.
[root@rhel77 ~]# yum history info 2
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Transaction ID : 2
Begin time : Mon Nov 2 15:25:12 2020
Begin rpmdb : 345:fc299a705f95203fda639ed1153803ad6771d7fb
End time : 15:25:14 2020 (2 seconds)
End rpmdb : 345:f8faef254573c0653a624c30ee393b08bb365ab2
User : root <root>
Return-Code : Success
Command Line : update device-mapper
Transaction performed with:
Installed rpm-4.11.3-40.el7.x86_64 installed
Installed subscription-manager-1.24.13-1.el7.x86_64 installed
Installed yum-3.4.3-163.el7.noarch installed
Packages Altered:
Updated device-mapper-7:1.02.158-2.el7.x86_64 @?rhel-7-server-rpms
Update 7:1.02.170-6.el7.x86_64 @rhel-7-server-rpms
Updated device-mapper-libs-7:1.02.158-2.el7.x86_64 @?rhel-7-server-rpms
Update 7:1.02.170-6.el7.x86_64 @rhel-7-server-rpms
history info
Finally, undo the last transaction using yum history undo last.
[root@rhel77 ~]# yum history undo last
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Undoing transaction 2, from Mon Nov 2 15:25:12 2020
Updated device-mapper-7:1.02.158-2.el7.x86_64 @?rhel-7-server-rpms
Update 7:1.02.170-6.el7.x86_64 @rhel-7-server-rpms
Updated device-mapper-libs-7:1.02.158-2.el7.x86_64 @?rhel-7-server-rpms
Update 7:1.02.170-6.el7.x86_64 @rhel-7-server-rpms
Resolving Dependencies
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be a downgrade
---> Package device-mapper.x86_64 7:1.02.170-6.el7 will be erased
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be a downgrade
---> Package device-mapper-libs.x86_64 7:1.02.170-6.el7 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================================================================
Package Arch Version Repository Size
=======================================================================================================================
Downgrading:
device-mapper x86_64 7:1.02.158-2.el7 rhel-7-server-rpms 294 k
device-mapper-libs x86_64 7:1.02.158-2.el7 rhel-7-server-rpms 322 k
Transaction Summary
=======================================================================================================================
Downgrade 2 Packages
Total download size: 616 k
Is this ok [y/d/N]:
[ A free guide from Red Hat: 5 steps to automate your business. ]
It isn't the end
Being proactive is the best way to avoid rolling back a failed patch. Taking periodic operating system level and application-level backups is the recommended way to roll back the changes confidently. Read my Enable Sysadmin article on how to use ReaR for image-based backup and restore.
For more information on rolling back updates in Red Hat Enterprise Linux, be sure to check out the Red Hat Customer Portal.
Sreejith Anujan
Sreejith is a cloud technology professional with more than 14 years of experience in on-premise and public cloud providers. He enjoys working with customers on their enablement plans to upskill the technical team on container and automation tooling. More about me
Try Red Hat Enterprise Linux
Download it at no charge from the Red Hat Developer program.
Related Content
