Continuing my previous post about installation of additional packages on Exadata compute node – here I’m going to check what happens if I upgrade some system packages beyond version expected by my particular Exadata firmware.
I repeat previous exercise and delete exadata-sun-vm-computenode-exact package following by switching to Latest Oracle Linux 6 repository and installation of newest available ipa-client, which – as a result – upgrades dbus-libs and krb5-libs packages to the versions newer then those delivered with 12.1.2.3.3 firmware:
[root@exa01vm03 ~]# vi /etc/yum.repos.d/exadata.repo # disabled [ol6_u8_x86_64_base] and enabled [public_ol6_latest] [root@exa01vm03 ~]# yum remove exadata-sun-vm-computenode-exact Setting up Remove Process Resolving Dependencies --> Running transaction check ---> Package exadata-sun-vm-computenode-exact.noarch 0:12.1.2.3.3.161013-1 will be erased --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================== Package Arch Version Repository Size ==================================================================================================================== Removing: exadata-sun-vm-computenode-exact noarch 12.1.2.3.3.161013-1 @exadata_ol6_base_repo_12.1.2.3.3 0.0 Transaction Summary ==================================================================================================================== Remove 1 Package(s) Installed size: 0 Is this ok [y/N]: y Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Erasing : exadata-sun-vm-computenode-exact-12.1.2.3.3.161013-1.noarch 1/1 Verifying : exadata-sun-vm-computenode-exact-12.1.2.3.3.161013-1.noarch 1/1 Removed: exadata-sun-vm-computenode-exact.noarch 0:12.1.2.3.3.161013-1 Complete! [root@exa01vm03 ~]# yum install ipa-client Setting up Install Process Resolving Dependencies ... Dependencies Resolved ==================================================================================================================== Package Arch Version Repository Size ==================================================================================================================== Installing: ipa-client x86_64 3.0.0-51.0.1.el6 public_ol6_latest 152 k Installing for dependencies: ... dbus x86_64 1:1.2.24-9.0.1.el6 public_ol6_latest 211 k ... Transaction Summary ==================================================================================================================== Install 57 Package(s) Upgrade 2 Package(s) Total download size: 17 M Is this ok [y/N]: y Downloading Packages: ... Installed: ipa-client.x86_64 0:3.0.0-51.0.1.el6 ... Dependency Updated: dbus-libs.x86_64 1:1.2.24-9.0.1.el6 krb5-libs.x86_64 0:1.10.3-65.el6 Complete!
As you can see above 57 new packages have been installed and 2 upgraded. Please notice that among new packages there is dbus which is not part of Exadata braded packages. Now, my node no longer strictly matches requirements of 12.1.2.3.3 firmware – but still fulfills dependencies defined by exadata-sun-vm-computenode-minimum.
I’m going to check what happen when I try to upgrade my Exadata to newer version. Because I don’t want to upgrade my system at the moment – I will simulate this using dbnodeupdate.sh script (now it is integrated with patchmgr). For this purpose I downloaded patch 21634633 (see Doc ID 1553103.1) directly to my node and unpacked it to /u01/tmp/dbserver_patch_19.190411. I create subdirectory called dbnodeupdate and unpack file dbnodeupdate.zip into it (the file is delivered by the patch and used on the target node when you patch the system using patchmgr). The script has following parameters:
Usage: dbnodeupdate.sh [ -u | -r | -c | -b | -C ] [ -l <baseurl|zip file> ] [-p] <phase> [-n] [-s] [-i] [-k] [-A] [-G] [-K] [-w] [-q] [-S] [-v] [-t] [-R] [-b] [-m] [-M] [-N] [-a] | [-V] | [-h]
-u Upgrade
-r Rollback
-c Complete post actions (verify image status, cleanup, apply fixes, relink all homes, enable GI to start/start all domU's)
-l <baseurl|zip file> Baseurl (http or zipped iso file for the repository)
-s Shutdown stack (domU's for VM) before upgrading/rolling back
-p Bootstrap phase (1 or 2) only to be used when instructed by dbnodeupdate.sh
-q Quiet mode (no prompting) only be used in combination with -t
-n No backup will be created (Option disabled for systems being updated from Oracle Linux 5 to Oracle Linux 6)
-M Allow rpms be removed or pre-updated at prereq check time (-v) - (default is not allowed)
-N (default) No rpms will be removed or pre-updated at prereq check time (-v) - this will cause most prereq checks to fail.
-a Allow dbnodeupdate.sh to run with active NFS/SMB mounts. Requires compatible dbserver_backup.sh
Active NFS/SMB mounts supported starting release 12.1.2.1.1
-t 'to release' - used when in quiet mode or used when updating to one-offs/releases via 'latest' channel (requires 11.2.3.2.1)
-v Verify prereqs only. Only to be used with -u and -l option
-b Perform backup only
-R Remove any custom rpms found
-m Install / update-to exadata-sun/hp-computenode-minimum only (11.2.3.3.0 and later)
-i When specified during completion steps (-c), relinking of stack will be disabled and stack will not be not started.
-S When specified during the Oracle Linux 6 - Oracle Linux 7 upgrades: Skips certification of Grid Infrastructure/Database home compatibility with Oracle Linux 7.
-D Relink database homes only (skip Grid Infrastructure). Only possible in combination with -c
-k Keep EM-agents running
-C Cleanup mode. Cleans up all temporary content on the database node
-K Keep rpms and config the same while running complete post actions (-c) (default when rolling back)
-w Ignore open hardware alerts
-d Disable application database connection draining - (default is ON but only possible if supported by Oracle Grid Infrastructure).
-V Print version
-G Allow for non-signed repository metadata (starting 19.2 ony)
-h Print usage
As you can see there are lot of options to run this script. I’ve marked these which are relevant to my example. I’m going to run it with option -v nad without option -M. Why such combination? Because even in prereq mode dbnodeupgrade.sh is able to remove some packages (the same applies to patchmgr) to satisfy further dependencies (mainly packages related to OFED stack). Below command verifies requirements before upgrading from 12.1.2.3.3 to 18.1.12.0.0:
[root@exa01vm03 dbnodeupdate]# ./dbnodeupdate.sh -u -l http://yum-mirror/EngineeredSystems/exadata/dbserver/18.1.12.0.0/base/x86_64/ -v
(*) 2019-05-05 00:09:00: Initializing logfile /var/log/cellos/dbnodeupdate.log
##########################################################################################################################
# #
# Guidelines for using dbnodeupdate.sh (rel. 19.190411):
...
##########################################################################################################################
Continue ? [y/n] y
(*) 2019-05-05 00:09:03: Unzipping helpers (/u01/tmp/dbserver_patch_19.190411/dbnodeupdate/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
(*) 2019-05-05 00:09:07: Collecting system configuration settings. This may take a while...
(*) 2019-05-05 00:09:57: Validating system settings for known issues and best practices. This may take a while...
(*) 2019-05-05 00:09:57: Analyzing yum repository. This may take a while...
(*) 2019-05-05 00:09:57: Original /etc/yum.conf moved to /etc/yum.conf.050519000900, generating new yum.conf
(*) 2019-05-05 00:09:58: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo
WARNING: Potential conflicting rpms(s) found: dbus. These custom rpm(s) may fail the update and are recommended to be removed and reinstalled post-update.
(*) 2019-05-12 15:14:43: Validating the specified source location.
(*) 2019-05-12 15:14:44: Cleaning up the yum cache.
(*) 2019-05-12 15:14:46: Performing yum package dependency check for 'exact' dependencies. This may take a while...
(*) 2019-05-12 15:14:52: 'Exact' package dependencies cannot be applied
(*) 2019-05-12 15:14:52: Performing yum package dependency check for 'minimum' dependencies. This may take a while...
(*) 2019-05-12 15:14:57: 'Minimum' dependency check failed.
-----------------------------------------------------------------------------------------------------------------------------
Running in prereq check mode. Flag -M was not specified this means NO rpms will be pre-updated or removed to make the prereq check work.
-----------------------------------------------------------------------------------------------------------------------------
Active Image version : 12.1.2.3.3.161013
Active Kernel version : 2.6.39-400.284.1.el6uek
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Inactive Image version : n/a
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Current user id : root
Action : upgrade
Upgrading to : 18.1.12.0.0.190111 (to exadata-sun-vm-computenode-minimum)
Baseurl : http://yum-mirror/EngineeredSystems/exadata/dbserver/18.1.12.0.0/base/x86_64/ (http)
Create a backup : Yes
Shutdown stack : No (Currently stack is up)
Missing package files : Not tested.
RPM exclusion list : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete lists : /etc/exadata/yum/obsolete_nodeps.lst, /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
: RPM obsolete list is extracted from exadata-sun-vm-computenode-18.1.12.0.0.190111-1.x86_64.rpm
Exact dependencies : Will fail on a next update. Update to 'exact' will be not possible. Falling back to 'minimum'
: Update target switched to 'minimum'
Minimum dependencies : Will fail on a next update. Update not possible without solving package problems first.
: Or run prereq check with -M but this may remove packages at prereq time.
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 120519151303)
Diagfile : /var/log/cellos/dbnodeupdate.120519151303.diag
Server model : Exadata Virtual Machine (domU)
dbnodeupdate.sh rel. : 19.190411 (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.
ERROR: Found dependency issues during pre-check. Packages failing:
ERROR: Package: oracle-ofed-release-guest-1.0.0-31.el6.x86_64 (Fails because of required removal of Exadata rpms)
ERROR: Consult file exa01vm03:/var/log/cellos/minimum_conflict_report.120519151303.txt for more information on the dependencies failing and for next steps.
The following known issues will be checked for but require manual follow-up:
(*) - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
Prereq check failed, check the above report for the details. It's also possible this is because of -M flag not being specified.
When needed: run prereq check with -M to remove known rpm dependency failures or execute the commands in exa01vm03:/var/log/cellos/nomodify_results.120519151303.sh.
What we can conclude from above? First of all prereq failed – but this is almost always expected if you run script without -M option. The reason for that are OFED packages – line 56 is giving a hint about this. If we look into /var/log/cellos/minimum_conflict_report.120519151303.txt as advice in line 57 then we see what packages are making it failed:
Error: Package: oracle-ofed-release-guest-1.0.0-31.el6.x86_64 (exadata_generated_120519151303)
Requires: libibcm = 1.0.5-1.0.2.el6
Installed: libibcm-1.0.5-3.el6.x86_64 (installed)
libibcm = 1.0.5-3.el6
Available: libibcm-1.0.5-1.0.2.el6.x86_64 (exadata_generated_120519151303)
This dependency conflict will be resolved automatically by dbnodeupdate.sh (or patchmgr) – when run in modification prereq mode or during actual upgrade. The script just removes some packages related to OFED including libibcm. You can find what is removed in this mode by looking into file /var/log/cellos/nomodify_results.120519151303.sh.
More interesting are the other lines (22,37,45) – telling us that exact dependencies could not be applied and upgrade will use minimal dependencies. This means that even after upgrade to 18.1.12.0.0 our node will have some Exadata branded packages newer that those included in this firmware. This is result of using Latest repository – this should not happen if ipa-client has been installed from Base OL 6.8 repository.
To have better information about what causes exact dependencies to be not met – you can check this file /var/log/cellos/exact_conflict_report.120519151303.txt. You will find long list of conflicts – similar to what I’ve shown to you in my previous blog post.
There is also warning concerning dbus package in line 17. Script recommends that this package should be removed before an upgrade and reinstalled again after. Not following this advice might result in failure of upgrade. This is however not conclusive and might also be ignored (but it is better to test this on similar test environment and with good backup). Oracle just warns you that there are known issues when upgrading with this package installed. There is a file ./os_upgrades_common/known_exadata_packages.lis in the subdirectory where you unpack dbnodeupdate.zip, which contains list of packages for which script displays such warning.
The proper way to proceed
I always try to bring the the system as much close to baseline as possible by removing/downgrading conflicting packages. In this case I can just run yum history undo <step> – to rollback all changes done during installation of our ipa-client. In reality this might not be as easy as it sounds – if there were subsequent package installations – then I prefer rolling back all the changes done by customer and reinstalling necessary packages after finishing upgrade process. Remember that you will need to prepare YUM repository for current version of your firmware – to accomplish this task, as undo or rollback will need to downgrade packages to their original version.