Updating Exadata Database Server Software

Document 1553103.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-stat...
dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the
DBNodeUpdate Utility and patchmgr (Doc ID 1553103.1)
In this Document
Main Content
Quick link to the download
Requirements and Support
Running the dbnodeupdate.sh/patchmgr Utility
Running dbnodeupdate.sh via sudo
Information to collect for RCA
Collecting RCA for Oracle Linux 5 to Oracle Linux 5 or Oracle Linux 6 to Oracle Linux 6 updates
For regular prereq checks or updates failing (when the system is still bootable)
For updates or rollbacks that caused the system to become non-bootable
Collecting RCA for Oracle Linux 5 to Oracle Linux 6 updates
System rolled back automatically
System where auto rollback did not kick in automatically
OL5-OL6 phases
Troubleshooting
References
APPLIES TO:
Oracle Exadata Storage Server Software - Version 11.2.2.4.2 and later

Linux x86-64
MAIN CONTENT
README for DB Node Update Utility and dbserver.patch.zip (v5.170420)

=========================
NOTE: Starting Exadata 12.2.1.1.0 dbnodeupdate.zip is no more available as a separate download. Exadata Database Node
Updates should be applied using patchmgr. The dbserver.patch.zip file contains a special version of patchmgr used to orchestrate
dbnodeupdate in updating Exadata Databases Nodes. dbnodeupdate.zip is still available in the dbserver.patch.zip file for those who
prefer manual execution the conventional way.
IMPORTANT: Starting release 5.170209 by default the patchmgr/dbnodeupdate.sh script does not remove Exadata rpms that are
known to fail the yum dryrun during pre-req check time. If you want to be sure to rely on precheck output then run precheck with
the additional flag to allow rpm modifications but do understand that this is might be making changes to your system that are not
expected at this time. This new default behavior makes sure no rpms are removed at precheck time but will rather generate a script
with rpms that would have been removed. The operator can then decide if removing any of these rpms at prereq check would
impact the system and optionally post-pone removal of these rpms until the actual planned maintenance window. It is
recommended to run the prereq check with the flags to allow making changes within a maintenance window right before the actual
updating. Known (not custom) conflicting rpms will be always removed when running the update regardless.
Flags:
For running patchmgr and allowing it to make changes at prereq time, add flag: -modify_at_prereq
For running patchmgr and not allowing it to make changes at prereq time, add flag: -nomodify_at_prereq (default)
1 of 15 5/4/17, 9:05 AM
For running dbnodeupdate and allowing it to make changes at prereq time, add flag: -M
For running dbnodeupdate and not allowing it to make changes at prereq time, add flag: -N (default)
This README covers the following :
1. Quick link to the dbserver.patch.zip file

2. Introduction
3. Requirements
4. Downloads and installation.
5. Running the patchmgr Utility - Usage
6. Running dbnodeupdate.sh via sudo
7. Information to collect when things fail
8. Troubleshooting
9. Orchestrating Database Servers Updates (dbnodeupdate) with patchmgr (using dbserver.patch.zip)
10. Revision table
Quick link to the download
dbnodeupdate.zip standalone (patch 16486998) cannot be downloaded anymore but can still be found inside dbserver.patch.zip
Download dbserver.patch.zip as p21634633_12*_Linux-x86-64.zip, which contains dbnodeupdate.zip and patchmgr for dbnodeupdate
orchestration via patch 21634633
Find the Oracle Exadata update bits for the release you need in note 888828.1
NOTE: dbnodeupdate.sh/dbserver.patch.zip support ANY Exadata hardware version (v1, v2, X2-*, X3-*, X4-*, X5-*). See the
specific Exadata Storage Server release README to see if the update you want to apply is supported for your hardware
version. The dbnodeupdate.sh utility requires you to be on Exadata release 11.2.2.4.2 (running on OL 5.5 or later)
NOTE: In case of an issue with the environment being updated, it is advised to collect var/log/cellos/dbnodeupdate.log and the
diag file that corresponds with your run (for example: var/log/cellos/dbnodeupdate.060114093556.diag, where 060114093556 is
your runid) - then upload the files to your SR or bug.
Introduction
The 'DB Node Update Utility' (dbnodeupdate.sh) (and also dbnodeupdate orchestrated by patchmgr in dbserver.patch.zip) automates all the
steps and checks to update Oracle Exadata database servers to a new Exadata release and replaces manual steps. The dbnodeupdate.sh
utility includes the latest best practices and workarounds for known issues.
The major use-cases for the dbnodeupdate.sh/dbserver.patch utility are :
Precheck + Updating database servers running Exadata releases later than 11.2.2.4.2 on Oracle Linux 5.5 or later
Rolling back updates
Post-Patching (or Post-Rollback steps) (relinking the Oracle homes, enabling Grid Infrastructure to start)
Addressing, fixing or working around known issues
A shortened list of some of the functionality provided by the dbnodeupdate.sh utility as follows:
Creates log file to track script execution and changes

Creates diag file with 'before patching' situation
Provides 'next steps' in case of any error
Includes checks and workarounds for known issues
Creates and runs the backup utility
Checks space requirements of /boot filesystem
Checks for left-over snapshots
Checks for required files and utilities
Includes 'check-only' option
2 of 15 5/4/17, 9:05 AM
Relinks all Database and Grid Infrastructure (GI) Homes

Enables / disables GI to stop/start
Enables / disables domU to stop/start
Checks for nfs/smbfs/cifs mounts and unmounts when possible
Validates provided media (zip, http)
Validates user input
Provides rollback option
Generates 'yum repo'-file
Validates yum.conf-file
Logs console history
Stops OS/Exawatcher before patching
Unpacks media
Stops db console and em agent
Checks for failed 'pvmove' actions
Checks for left over snapshots and failed backups
Supports multiple em agents
Checks for free space in "/"
Requirements and Support
A list of the requirements for running the dbnodeupdate.sh utility:
The 'dbserver.patch.zip' file - See Quick links for downloading the 'DB Node Update Utility'
Any upgrade either via dbnodeupdate or patchmgr requires a repository with the Exadata rpms for the release being updated to. This
repository can be one of the following:
1. A web based repository. Example: http://yum-repo.us.oracle.com/yum/unknown/EXADATA/dbserver/11.2_save/latest/x86_64/
2. 'a file based repository' (zip file with Exadata ISO in it)
NOTE: When using an http location as yum repository, it is expected that the repository is populated before starting the actual upgrade.
See My Oracle Support note 1473002.1 for instructions on building a repository manually (when not using the zip file or ISO image
within the zip file)
Downloads and installation
Download the dbserver.patch.zip file (see 'Quick link to the download')
Determine the required Oracle Exadata release listed in My Oracle Support note 888828.1
1. Unzip the dbserver.zip file:
When running in orchestration mode (patchmgr) you can unzip and run patchmgr as root or non root from any Linux system.
When running dbnodeupdate, unzip the dbnodeupdate.zip within the dbserver.patch.zip on the target node
2. Use a zip-file with an ISO image (not relevant when using a yum repository via http)
When using a zip file with an ISO image in it, it can be picked up by the dbnodeupdate.sh/patchmgr utility specifying the path. This is
the alternative approach when not using http.
When using patchmgr the zip file should be stored on the driving node
When using dbnodeupdate the zip file should be stored on the target node
NOTE: By default dbnodeupdate.sh/patchmgr makes backups (provided the database servers have the standard lvm's in use and free
space available in the default volume group).
Because running backups takes time, it is possible to make the backup before the outage. When this is done or when backups are made
manually with another tool, use the a flag to be sure no new backups are made at update time. For updates from Oracle Linux 5 to
Oracle Linux 6, the backup is mandatory and cannot be skipped.
3 of 15 5/4/17, 9:05 AM
Running the dbnodeupdate.sh/patchmgr Utility
The dbnodeupdate.sh utility must be run as the root user on the target server. It's recommended to run the dbnodeupdate.sh/patchmgr
session using the Linux 'screen' or 'vnc' utility such that when a network problem breaks the connection to the server the patching session
continues. The patchmgr utility can be run as root or non-root user from any Linux server other than the node that will be updated.
Note: The 'screen utility' should be used by JAWS, WindowEyes and other screen reader users.
The Linux screen utility can be downloaded via Oracle Public-yum
Note: for instructions how to run the patchmgr/dbnodeupdate see the tool's help (-h) or refer to the Oracle Exadata Maintenance Guide.
Running patchmgr (in dbserver.patch.zip) to orchestrate dbnodeupdate.sh
See the Exadata maintenance guide or patchmgr online help for usage.
Examples:
Read only prereq check:
# ./patchmgr -dbnodes dbs_group -precheck -nomodify_at_prereq -log_dir auto -target_version

12.1.2.3.4.170111 -yum_repo http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.3.4
/base/x86_64/
Prereq check (may remove or pre-update rpms we know of that need to be removed):
# ./patchmgr -dbnodes dbs_group -precheck -modify_at_prereq -log_dir auto -target_version

12.1.2.3.4.170111 -yum_repo http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.3.4
/base/x86_64/
Updating:
# ./patchmgr -dbnodes dbs_group -upgrade -log_dir auto -target_version 12.1.2.3.4.170111 -yum_repo

http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.3.4/base/x86_64/
Rollback:
# ./patchmgr -dbnodes dbs_group -rollback -log_dir auto
NOTE: dbgroup specifies a file with all dbnodes to be updated. The node driving the update should not be added to the the file.
Also in the above example the http address can be replaced by iso.zip file location using -iso_repo instead of -yum_repo
NOTE:
Regular Oracle Exadata Database Server and Cell updates such as described in document 888828.1 can only be applied when the
date-stamp of the target release is newer than the date-stamp of the release currently running. Updates from images running a release
with a date-stamp newer than the target release will be blocked.
For example: Updates from 12.1.1.1.2.150411 to 12.1.2.1.0.141206.1 are blocked because 12.1.1.1.2 (with timestamp 150411) has a
date-stamp newer than 12.1.2.1.0 (141206), even while Exadata release 12.1.2.1.0 seems to be more recent than 12.1.1.1.2. When
situations like this happen, it's recommended to find a maintenance release with a newer date-stamp for that same major release.
4 of 15 5/4/17, 9:05 AM
dbnodeupdate.sh can be executed via sudo. Perform the following steps to setup sudoers:
Login as root and edit /etc/sudoers by typing the command 'visudo'.
# visudo
Add the following entry to the bottom of the sudoers file to allow the oracle user to run dbnodeupdate as root :
oracle ALL=(ALL) NOPASSWD:SETENV: /u01/stage/patch/dbnodeupdate/dbnodeupdate.sh
As root create the /u01/stage/patch/dbnodeupdate/ directory as and unzip dbnodeupdate.zip:
# mkdir -p /u01/stage/patch/dbnodeupdate
# cp dbnodeupdate.zip /u01/stage/patch/dbnodeupdate
# cd /u01/stage/patch/dbnodeupdate
# unzip dbnodeupdate.zip
To verify the correct setup run dbnodeupdate in prereq check mode as oracle:
$(oracle) cd /u01/stage/patch/dbnodeupdate
$(oracle) sudo ./dbnodeupdate.sh -u -l http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3
/base/x86_64/ -v
dbnodeupdate would exit if the script runs without root privileges.
NOTE: The above setup requires the entire content of /u01/stage/patch/dbnodeupdate to be owned by root.
A new version of dbnodeupdate would need to be placed in the same location as specified by sudoers.
Information to collect for RCA
Collecting RCA for Oracle Linux 5 to Oracle Linux 5 or Oracle Linux 6 to Oracle Linux 6 updates
Oracle Linux 5 to Oracle Linux 5 are updates from releases < 12.1.2.1.0 to < 12.1.2.1.0
Oracle Linux 6 to Oracle Linux 6 are updates from releases 12.1.2.1.0 to > 12.1.2.1.0
For regular prereq checks or updates failing (when the system is still bootable)
Collect the following information:
1. /var/log/cellos/dbnodeupdate.log
2. /var/log/cellos/dbnodeupdate.<runid>.diag (where runid is the run-id f your failing dbnodeupdate session)
These are also listed in the on-screen dbnodeupdate dialog:
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 051015022126)

Diagfile : /var/log/cellos/dbnodeupdate.051015022126.diag
For updates or rollbacks that caused the system to become non-bootable
See section 'System where auto rollback did not kick in automatically'
Collecting RCA for Oracle Linux 5 to Oracle Linux 6 updates
This affects updates from releases < 12.1.2.1.0 to 12.1.2.1.0 or later.
5 of 15 5/4/17, 9:05 AM
System rolled back automatically
When an error occurs at update time auto-rollback should kick in. When this happens console shows the following message on screen (this
is only a snippet):
ATTENTION:
The update to 12.1.2.1.0 has failed. Auto rollback has prepared and completed rollback preparations
After rollback completes the database node should reboot and when logging back on the the system the 'imageinfo' command should show
the release as before the failed update. Now before re-attempting the update first do an RCA.
Kernel version: 2.6.39-400.243.1.el6uek.x86_64 #1 SMP Wed Nov 26 09:15:35 PST 2014 x86_64
Image version: 11.2.3.3.1.140529.1
Image activated: 2015-01-13 09:26:19 -0800
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys2
Collect all logging from the active and inactive system partition:
If after auto rollback the system partitions is /dev/mapper/VGExaDb-LVDbSys2 the inactive system partition is /dev/mapper
/VGExaDb-LVDbSys1
If after auto rollback the system partitions is /dev/mapper/VGExaDb-LVDbSys1 the inactive system partition is /dev/mapper
/VGExaDb-LVDbSys2
In the following example, the inactive system partition is LVDbSys2 (change LVDbSys2 for LVDbSy1 if your system has a different inactive
partition)
1. Dump console history (as soon as possible after the failing update): ipmitool -H <ilom-hostname> -U root -P password sunoem cli
'show /SP/console/history' > /var/log/cellos/serialconsole.<hostname>.txt
2. # mkdir -p /mnt/LVDbSys2
3. # mount /dev/mapper/VGExaDb-LVDbSys2 /mnt/LVDbSys2
4. # tar cvf /root/var_log_LVDbSys1_LVDbSys2.tar /var/log /mnt/LVDbSys2/var/log /.ol6_upgrade* /mnt/LVDbSys2/.ol6_upgrade*
5. # gzip -f /root/var_log_LVDbSys1_LVDbSys2.tar
6. # umount /mnt/LVDbSys2
In addition also collect*:
1. sundiag
/opt/oracle.SupportTools/sundiag.sh (see document 761868.1)
2. sosreport
See How To Collect an Sosreport on Oracle Linux document 1500235.1
See How to Collect sosreport under Rescue Mode document 1928852.1
3. ILOM snapshot
/opt/oracle.SupportTools/sundiag.sh snapshot (see document 761868.1)
*Note: before re-attempting the update re-collect these diag files again.
System where auto rollback did not kick in automatically
When auto rollback didn't not kick in and the system is not booting properly, use the steps in "How to recover from a failed Exadata DB
Server update or rollback document 1952372.1" then manually collect the files as follows:
1. Dump console history (as soon as possible after the failing update): ipmitool -H <ilom-hostname> -U root -P password sunoem cli
'show /SP/console/history' > /var/log/cellos/serialconsole.<hostname>.txt (note, if not possible to run on the problematic host, run
the command from another host)
2. # mkdir /mnt/LVDbSys1
3. # mkdir /mnt/LVDbSys2
6. # tar cvf /root/var_log_LVDbSys1_LVDbSys2.tar /mnt/LVDbSys1/var/log /mnt/LVDbSys2/var/log /mnt/LVDbSys1/var/log
/.ol6_upgrade* /mnt/LVDbSys2/.ol6_upgrade*
7. # gzip -f /root/var_log_LVDbSys1_LVDbSys2.tar
6 of 15 5/4/17, 9:05 AM
In addition also collect*:
1. sundiag
/opt/oracle.SupportTools/sundiag.sh (see document 761868.1)
2. sosreport
See How To Collect an Sosreport on Oracle Linux document 1500235.1
See How to Collect sosreport under Rescue Mode document 1928852.1
3. ILOM snapshot
/opt/oracle.SupportTools/sundiag.sh snapshot (see document 761868.1)
*Note: before re-attempting the update re-collect these diag files again.
OL5-OL6 phases
the OL5-OL6 update has 6 different phases:
1. Phase Initialize: PREREQ CHECKS, CAPTURE SYSTEM SETTINGS, on OL5 kernel, First reboot
2. Phase Prep: UPGRADE RPM DATADASE, PRESERVE KEY CONFIGURATIONS, on custom OL6 kernel
3. Phase 1: REMOVE CONFLICTING PACKAGES, INITIAL PACKAGE UPGRADES, on custom OL6 kernel, Second reboot
4. Phase 2: REINSTALL INITIAL PACKAGE UPGRADES, UPGRADE KERNEL and DEPS, on OL5 kernel, Third reboot
5. Phase 3: INSTALL EXADATA PACKAGES, on new OL6 kernel
6. Phase 4: EXADATA CONFIGURATIONS, RESTORE PRE-UPGRADE CUSTOMIZATIONS, REMOVE KNOWN OL5 PKGS, Fourth reboot
The different phases also provide reports the user might want to review in case of failure. These are displayed at the console and are also
found in the /var/log/cellos/dbupgrade.log:
In case of failure:
"======== NOTICE ========"

"Copying relevant diagnostic files to the inactive volume: ${v_inactive_lvm_name}"
"Once recovery from backup is complete see the following files for diagnostic information:"
"${EXADATA_IMG_LOG}/dbupgrade_ol6_upgrade_marker"
"${EXADATA_IMG_LOG}/dbupgrade_ol6_upgrade_state_file"
"${EXADATA_IMG_LOG}/dbupgrade.log and ${EXADATA_IMG_LOG}/dbupgrade.trc"
"======== NOTICE ========"
The different phases also provide reports the user might want to review after a succesfull update. These are displayed at the console and
are also found in the /var/log/cellos/dbupgrade.log:
At the end of the OL5-OL6 upgrade:
"NOTE: Configuration files saved by this procedure (.pre_OL6_upgrade) have been preserved in:"
"NOTE: Logfiles for this procedure are located at ${v_dbupgrade_logfile}{.log, .trc}"
"NOTE: The /etc/inittab file is mostly deprecated (only the default run level remains)."
"The Exadata specific entries, pre upgrade OL5 /etc/inittab entries have been converted to OL6 configurations."
"The pre upgrade, /etc/inittab is saved here: ${v_active_vol_mount}${v_dbupgrade_upgrade_save_files}/etc
/inittab.pre_OL6_upgrade."
"Non-converted, pre upgrade, /etc/inittab entries, if any, are:"
"--- begin: non-converted inittab entries ---"
"--- end: non-converted inittab entries ---"
"NOTE: The /etc/modprobe.conf file is deprecated."
"The pre upgrade modprobe.conf entries have been migrated to /etc/modprobe.d/exadata.conf."
"NOTE: Files in /etc/modprobe.d/ must have a .conf file extension."
"Any non .conf files in /etc/modprobe.d/ prior to the upgrade have been moved to .config files if possible (no
file over-writes)."
"Non-converted, pre upgrade, /etc/modprobe.d files not moved to a .conf equivalent if any, are:"
"--- begin: non .conf files in /etc/modprobe.d/ ---"
"--- end: non .conf files in /etc/modprobe.d/ ---"
Troubleshooting
7 of 15 5/4/17, 9:05 AM
When hitting any issue:
Check you are on the latest release of the dbnodeupdate or dbserver.patch.zip package
There are two possible error situations when running this script:
1. Errors due to a unexpected configuration or problem on the db server - most often due to customizations
The type of failure seen most often are dependency issues introduced by installation of custom rpms - see the next section.
2. Errors due to bugs in the dbnodeupdate.sh script or the Exadata image
When filing a SR the minimum amount of information required is:
the dbnodeupdate.sh runid (as seen in the confirmation screen)

The dbnodeupdate.log file
The dbnodeupdate diag file (as mentioned in the confirmation screen, i.e.: /var/log/cellos/dbnodeupdate.<runid>.diag)
Troubleshooting flow:
The tool prevents against proceeding the update when a certain check, software or firmware version is incorrect. This is to prevent
against failing updates and prevent against unplanned downtime. In such cases it is recommended to fix the issue and re-run the
script.
When it is unclear what needs to be done, when the script fails or when no reboot is initiated:
In case the script stopped before the 'confirmation screen'
At this point no changed are made yet - communicate this to the parties involved.
Did the prereq run succeed ? Research screen output and /var/log/cellos/dbnodeupdate.log for the prereq
See /var/log/cellos/dbnodeupdate.log where the script got stuck
What is the last successful action ? Use screen and /var/log/cellos/dbnodeupdate.log output
Possible unresponsive ILOMS may require a reset when collect of diag does not respond within 10 minutes
Correct possible config issues and re-run the script
Was the right repository used ?
for http repositories check for the right url
for iso repositories check the right iso zip
In case the script stopped after the 'confirmation screen'
Did the backup complete ?
If the backup did complete it is possible to rollback now to reduce downtime ?
if the backup did not complete, was this the cause of the backup script failing ?
see /var/log/cellos/dbnodeupdate.log
see /var/log/cellos/dbserver_backup.log
What did the screen state when it failed, what was the last successful action ? match this with the output in /var/log
/cellos/dbnodeupdate.log
For updates from 11.2.3.1.0 and later:
check if the 'yum update' part did complete also see /var/log/yum.log
see /var/log/cellos/exadata.computenode.post.log, /var/log/messages and /var/log/cellos/exachkcfg.log if no
reboot was initiated
/var/log/cellos/exadata.computenode.post.log file has the output from the /opt/oracle.cellos
/exadata.computenode.post.sh script which is started as post scriptlet from the exadata-
sun-computenode rpm. When there is no timestamp in it that matches your update the script has not
been started. Investigate why via the dbnodeupdate.log file
For updates from 11.2.2.4.2:
see /var/log/cellos/bootstrap.uln.log
Do not reboot a node when the update did not reboot itself
Don't do any manual rollback - follow the dbnodeupdate.sh rollback procedure
When a node is rebooted:
dbnodeupdate.sh completed its work starting the yum update. The yum update did complete.
Nodes not booting may need to open ILOM to see what is going on.
When nothing happends and the ILOM is not in the process of being updated - reset the ILOM and monitor the
boot process
When the node boots in the new image continue with the 'dbnodeupdate.sh -c' steps
Always collect and analyze all logs before re-attempting the update.
When updates or (auto)rollbacks (possibly via dbnodeupdate.sh) fail, an Exadata database server may become unbootable. It can also
happen a database server does boot but fails to load the right modules or libraries after an broken or interrupted update, effectively
making the system unusable. As a result it can happens users and administrators are unable to login to the system, even via the console.
8 of 15 5/4/17, 9:05 AM
MOS document 1952372.1 describes how to recover from a failing update or rollback on an Exadata DB server. This document only
applies to X2 and later DB servers where logical volume management (lvm) is in use and where a backup was made to the inactive
system partition prior the update. This is especially important for updates to Oracle Linux 6 / 12.1.2.1.0.
Dependencies, custom RPM packages and regular Oracle Linux RPM updates
dbnodeupdate.sh only installs Exadata channel updates.
NOTE: The dbnodeupdate.sh utility is a wrapper around the 'yum update' procedure. This means, YUM dependency issues can occur
when custom rpms are installed. If such errors will occur, dbnodeupdate.sh will detect this during prereq. When the dependency check
(yum dryrun) doesn't pass the update will be blocked and the system cannot be updated until the dependency issue is resolved. This
prevents against updates failing half way. The purpose of the dbnodeupdate.sh script is to automate and validate to minimize human
error. When errors happen due to the system (mis)configuration or dependency issues detected, the dbnodeupdate.sh utility can always
be run again after the problem is fixed or the dependency issue is resolved.
PREPARATION AND DEPENDENCY ISSUES
Operators are allowed to customize Exadata database servers by updating supplied packages or installing new packages as long as it does
not interfere with Exadata functionality, provided the kernel and InfiniBand packages are not altered. Although customizations are allowed,
they sometimes cause dbnodeupdate.sh to fail during the update because of package dependency problems. Because the failure occurred,
without warning, during update while the system was offline for maintenance, it typically led to extended downtime while the package
dependency problems were understood and rectified. it is recommended to remove custom packages before the update if they are known to
cause dependency failures. The packages can be reinstalled post update.
Each run of the dbnodeupdate.sh utility appends logging to the /var/log/cellos/dbnodeupdate.log file. Each run has a unique 'runid' and this
runid can be found within the dbnodeupdate.log file
The runid can be used to find the starting point in the logfile or to find the related diag file (/var/log/cellos/dbnodeupdate.<runid>.diag.
The diag file is created before the actual updating occurs. This diag file collects the situation 'as is' before the patching. It should provide all
the required information to expidite correction of database server patching problems. Some of the information that is collected :
File : /var/log/cellos/bootstrap.uln.log
File : /var/log/cellos/dbserver_backup.sh.log
File : /var/log/cellos/exadata.computenode.post.log
File : /etc/oratab
File : /etc/sysctl.conf
File : /etc/yum.conf
File : /var/log/cellos/validations.log
File : /boot/grub/grub.conf
File : /var/log/yum.log
File : /var/log/cellos/exachkcfg.log
File : /etc/enterprise-release
File : /etc/exadata/yum/exclusion.lst (post 11.2.3.2.1 only)
File : /etc/exadata/yum/obsolete.lst (post 11.2.3.2.1 only)
File : images.properties
File : issues.properties Output for : /usr/local/bin/imageinfo
Output for : /usr/local/bin/imagehistory
Output for : /usr/sbin/vgdisplay
Output for : /usr/sbin/lvdisplay
Output for : /sbin/vgscan
Output for : df -h
Output for : rpm -qa --qf "%{n}-%{v}-%{r}.%{arch}\n
Output for : ps -ef
Output for : mount
Output for : /opt/oracle.SupportTools/reclaimdisks.sh -check
Output for : /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL
Output for : ipmitool sunoem cli 'show /SP/console/history'
More is added in later releases.
When filing an SR, collect (and upload) the following information:
1. The 'runid' of the specific action.

2. The dbnodeupdate.log.
3. The corresponding diag file (dbnodeupdate.<runid>.diag).
9 of 15 5/4/17, 9:05 AM
The above items are minimum requirements for support engineers to diagnose a patching issue.
Revision History
Change
Date
April 25 2017 Released dbserver.patch.zip 5.170420

Adds support for future releases
April 20 2017 Released dbserver.patch.zip 5.170419

Adds fix for unpublished bug 25910229, bug 25884664, bug 25875907,
bug 25842271, bug 25753833, bug 25710548, bug 25643221, bug 25611800,
bug 25541830
Feb 17 2017 Released dbserver.patch.zip 5.170214

Adds fix for unpublished bug 25523387, bug 25541830
Feb 10 2017 Released dbserver.patch.zip 5.170209

At precheck by default no rpms known to cause dependency issues will be removed.
Starting this release operator needs to specify flag to allow removal/pre-update of
rpms known to cause dependency issues at precheck time.
Adds fix for unpublished bug 25469336, bug 25477198, bug 25490919, bug
23697239, bug 25504902, bug 25504886, bug 25504139
Feb 3 2017 Dbserver patch supports updates starting release 11.2.2.4.2 and later
Jan 31 2017 Released 5.170131

Starting Exadata 12.2.1.1.0 dbnodeupdate.zip is no more available as a separate
download. Exadata Database Node Updates should be applied using
dbserver.patch.zip. The dbserver.patch.zip file contains a special version of
patchmgr used to orchestrate dbnodeupdate in updating Exadata Databases Nodes.
dbnodeupdate.zip is still available in the dbserver.patch.zip file
Jan 30 2017 Released 5.170126

Adds support for future releases.
Jan 25 2017 Released 5.170123

Add fix for unpublished bug 25384342
Jan 13 2017 Released 5.170112

Adds fix for unpublished bug 25342527, bug 25328498
Dec 22 2016 Released 5.161221

Adds fix for unpublished bug 25255382, bug 25106611, bug 25250671
Dec 15 2016 Released 5.161214

10 of 15 5/4/17, 9:05 AM
Change
Date
Dec 11 2016 Released 5.161207

Adds support for future releases.
Nov 28 2016 Released 5.161126

bug 25108341, < bug 25108686>, bug 25106746, bug 24625614, bug 25081959
Nov 11 2016 Released 5.161110

bug 24934782, bug 24963916, bug 24933927
Oct 21 2016 Released 5.161020

Adds support for unpublished <bug24927574>
Oct 14 2016 Released 5.161014

Oct 12 2016 Released 5.161012

bug 24743555, bug 24321439 and bug 24663044
Sept 7 2016 Released 5.160907

Adds fix for unpublished bug 24594560

Adds fix for unpublished bug 24492131, bug 24455645> and bug 24529319
Aug 10 2016 Released 5.160809

July 22 2016 Released 5.160722

July 6 2016 Released 5.160706

Adds fix for unpublished bugs:
bug 23727288, bug 2066282, bug 2065181, bug 23604154, bug 23593827,
bug 23589251 bug 23589251, bug 23592335, bug 23577929, bug 23640301
June 11 2016 Released 5.160611

Add fix for unpublished bugs:
bug 23568450
June 9 2016 Released 5.160608
11 of 15 5/4/17, 9:05 AM
Change
Date

bug 23543534, bug 23301145
May 21 2016 Released 5.160521

bug 23177655, bug 23301145, bug 23175360, bug 23233160,
bug 23211144, bug 23211144, bug 23180964
April 26 2016 Released 5.160426

Add fix for unpublished bug:
bug 23170654

Add fix for unpublished bug:
bug 23124968


March 29 2016 Released 5.160328

bug 22988819, bug 22818033, bug 22954014

bug 22903561, bug 22914123 and bug 22933744

Adds OL5 fix for Adds fix for CVE-2015-3197 CVE-2016-0702 CVE-2016-0705
CVE-2016-0797 CVE-2016-0800

Adds fix for CVE-2015-1798 CVE-2015-1799 CVE-2015-5300 CVE-2015-7704
CVE-2015-8138
Adds fix for CVE-2015-3197 CVE-2016-0702 CVE-2016-0705 CVE-2016-0797
CVE-2016-0800
Feb 25 2016 Released 5.160225

Adds fixes for unpublished bug 22817311 and CVE-2015-7547
Feb 18 2016 Released 5.160217

12 of 15 5/4/17, 9:05 AM
Change
Date
Jan 26 2016 Released 5.160126

Adds fixes for unpublished bugs:
bug 22558388, bug 22588882
Jan 22 2016 Released 5.160121

Dec 25 2015 Released 5.151224

Adds fixes and enhancements for unpublished bugs:
bug 22385393, bug 22391793, bug 22391819, bug 20629098, bug 22444382
Dec 17 2015 Released 5.151217

bug 22377699, bug 22345914, bug 22081369, bug 22295206, bug 21789515
Adds option for additional -N flag during prereq check.
Using the -N flag will make sure no rpms are removed at prereq check time.
Using the -N flag will generate a script of rpms that would otherwise have
been removed
When using -N most yum dependency prereq checks will typically fail
Operators can then decide to either postpone the removal of the
(known) conflicting rpms and remove them during at update time or
remove the rpms right away if this has no impact to their applications.
This removal can be done by re-running the script in prereq check
mode without specifying -N or by running the generated script
manually.
Add -a option for allowing active network mounts
Starting release 12.1.2.1.1 active network mounts are allowed. This can be
done by specifying the -a flag.
Dec 11 2015 Released 5.151210

bug 22343472, bug 21891550, bug 22304886
Dec 3 2015 Released 5.151203

bug 21978213,bug 22026321, bug 21872009, bug 21535908, bug 22089101,
bug 22178948 (blocking function made inactive for field), bug 21891550,
bug 22172439, bug 22217097, bug 22230825, bug
22263541, bug 22286402, bug 22296092, bug 22301859
Nov 12 2015 Added full support and instructions for using sudoers
Oct 22 2015 Released 5.151022

Oct 21 2015 Released 5.151021

bug 22070309,bug 22026321,bug 22026321,bug 21943139,bug
21943542,bug 21908779,bug 21908561,bug 21911526
13 of 15 5/4/17, 9:05 AM
Change
Date
Oct 12 2015 Added information for RCA

Adds support for 12.1.2.2.0.150917 (regular, dom0 and domU)
Sept 16 2015 Released 5.150916 of dbnodeupdate.zip and dbserver.patch.zip

dbserver.patch.zip contains dbnodeupdate.zip and patchmgr for dbnodeupdate
orchestration. Download via patch 21634633
Adds support for 12.1.2.2.0 (regular, dom0 and domU)
bug 21829218,bug 21839470,bug 21450925,bug 21325251,bug
21536482,bug 21558122,bug 21555899,bug 21632956,bug 21644609,bug
21659467,bug 21684595,bug 21768122,bug 21781895,bug 21798983,bug
21770760,bug 21781989,bug 21837708, bug 21261160

Jul 1 2015 Released 5.150701

Jun 26 2015 Released 5.150625

Jun 20 2015 Released 5.150619

Jun 15 2015 Released 4.62

Adds fix for GI start retry when first start attempt fails via unpublished
bug 20407400

Add fix for unpublished bug 21214404 (grub not updated)

Disables tcp segment offloading (via unpublished bug 21190996)
Applies fixes for CVE-2015-3456 (VENOM) (via unpublished bug 21114760)
Adds support for ib_set_node_desc.sh to support bond0 (via unpublished bug
21081046)
Searches for free space for local yum repo on domU systems (via unpublished
bug 21112161)
Adds flag to not stop EM Agents (via unpublished bug 21112761)
April 14 2015 Released 4.46 - No functional changes, adds support for future releases
14 of 15 5/4/17, 9:05 AM
Change
Date
April 11 2015 Released 4.45a - Adds fix for unpublished bug 20859967
April 8 2015 Released 4.45 - Adds fix for unpublished bug 20708183 and unpublished bug 20812128
Mar 24 2015 Released 4.44 - Adds fix for leap second, unpublished bug 20706647 and bug 20597282.
Supports up to release 12.1.2.1.1
Mar 9 2015 Released 4.40 - Adds fix for unpublished bug 20637355 and pre-req checks for SELinux
Mar 6 2015 Released 4.29 - no functional changes
Feb 24 2015 Released 4.28 and added -i flag to the documentation which skips relinking and starting of
the stack when in '-c' mode
Feb 11 2015 Released 4.26 - Add fixes for CVE-2015-0235
END
Didn't find what you are looking for?
15 of 15 5/4/17, 9:05 AM

Updating Exadata Database Server Software

Uploaded by

Copyright:

Available Formats

Updating Exadata Database Server Software

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Updating Exadata Database Server Software

Uploaded by

Copyright:

Available Formats

Document 1553103.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-stat...

Oracle Exadata Storage Server Software - Version 11.2.2.4.2 and later

README for DB Node Update Utility and dbserver.patch.zip (v5.170420)

This README covers the following :

1. Quick link to the dbserver.patch.zip file

Quick link to the download

The major use-cases for the dbnodeupdate.sh/dbserver.patch utility are :

Creates log file to track script execution and changes

Relinks all Database and Grid Infrastructure (GI) Homes

Requirements and Support

A list of the requirements for running the dbnodeupdate.sh utility:

Downloads and installation

Download the dbserver.patch.zip file (see 'Quick link to the download')

1. Unzip the dbserver.zip file:

Running the dbnodeupdate.sh/patchmgr Utility

The Linux screen utility can be downloaded via Oracle Public-yum

Running patchmgr (in dbserver.patch.zip) to orchestrate dbnodeupdate.sh

Read only prereq check:

# ./patchmgr -dbnodes dbs_group -precheck -nomodify_at_prereq -log_dir auto -target_version

# ./patchmgr -dbnodes dbs_group -precheck -modify_at_prereq -log_dir auto -target_version

# ./patchmgr -dbnodes dbs_group -upgrade -log_dir auto -target_version 12.1.2.3.4.170111 -yum_repo

# ./patchmgr -dbnodes dbs_group -rollback -log_dir auto

Running dbnodeupdate.sh via sudo

Running dbnodeupdate.sh via sudo

Login as root and edit /etc/sudoers by typing the command 'visudo'.

oracle ALL=(ALL) NOPASSWD:SETENV: /u01/stage/patch/dbnodeupdate/dbnodeupdate.sh

As root create the /u01/stage/patch/dbnodeupdate/ directory as and unzip dbnodeupdate.zip:

dbnodeupdate would exit if the script runs without root privileges.

Information to collect for RCA

Collect the following information:

These are also listed in the on-screen dbnodeupdate dialog:

Logfile : /var/log/cellos/dbnodeupdate.log (runid: 051015022126)

For updates or rollbacks that caused the system to become non-bootable

Collecting RCA for Oracle Linux 5 to Oracle Linux 6 updates

This affects updates from releases < 12.1.2.1.0 to 12.1.2.1.0 or later.

System rolled back automatically

In addition also collect*:

System where auto rollback did not kick in automatically

In addition also collect*:

the OL5-OL6 update has 6 different phases:

"======== NOTICE ========"

At the end of the OL5-OL6 upgrade:

When hitting any issue:

When filing a SR the minimum amount of information required is:

the dbnodeupdate.sh runid (as seen in the confirmation screen)

dbnodeupdate.sh only installs Exadata channel updates.

PREPARATION AND DEPENDENCY ISSUES

When filing an SR, collect (and upload) the following information:

1. The 'runid' of the specific action.

April 25 2017 Released dbserver.patch.zip 5.170420

April 20 2017 Released dbserver.patch.zip 5.170419

Feb 17 2017 Released dbserver.patch.zip 5.170214

Feb 10 2017 Released dbserver.patch.zip 5.170209

Jan 31 2017 Released 5.170131

Jan 30 2017 Released 5.170126

Jan 25 2017 Released 5.170123

Jan 13 2017 Released 5.170112

Dec 22 2016 Released 5.161221

Dec 15 2016 Released 5.161214