UP006174 7 5to8 0

Policy 7.5 to 8.
0 - Upgrade Procedure
Software Upgrade Procedure
Corporate Headquarters 5200 Paramount Parkway Morrisville, NC 27560 USA Phone +1.888.628.5521 +1.919.468.5500 E-mail: info@tekelec.com Copyright TEKELEC 2013. All Rights Reserved
Broadband Network Solutions Software Upgrade Procedure
Policy 7.5 to 8.0 - Upgrade Procedure
CAUTION: Use only the Upgrade procedure included in the Upgrade Kit. Before upgrading any system, please access Tekelecs Customer Support site and review any Technical Service Bulletins (TSBs) that relate to this upgrade. Refer to Appendix E for instructions on accessing this site. Contact the Tekelec Customer Care Center and inform them of your upgrade plans prior to beginning this or any upgrade procedure. Phone: 1-888-FOR-TKLC (1-888-367-8552) or 919-460-2150 (international) FAX: 919-460-2126 EMAIL: support@tekelec.com
UP006174
Version 1.4
1 of 136
CHANGE HISTORY
Date 5/30/12 6/08/12 6/25/12 7/2/12 7/23/12 7/27/12 7/31/12 8/2/12 8/9/12 8/22/12 8/29/12 9/20/12 10/02/12 10/03/12 Version 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.10 0.11 1.0 1.1 1.2 Author NPx NPx NPx NPx NPx NPx NPx NPx NPx NPx NPx NPx NPx NPx Description Initial Draft Upgrade Manager Procedures added Updates from reviews Additional Updates from testing Additional Updates from testing Added key exchange tool. Added iso Verify steps before site upgrade. Added Work around for setting Force Standby on 7.5.x servers Added specific procedures for backout Improvements from 8.0.0_24.1.0 load Changes from latest 8.0.0 RC load Update for TSB Change to exclusions steps, Add inetstat verify steps A note added to firmware upgrade section (CS) Added Firewall detail Added Known Limitations Added Replication Activation backout Add updates from FOA Add workaround for 8.0.0_29.1.0 Close Upgrade Manager during upgrade Add screen tool for upgrade, when using ssh Add TN003516 steps to clean up /var/ directory Add icmp blocking scenario among servers Approved* (Yes/No) No No No No No No No No No No No Yes Yes Yes
10/16/12 10/16/12 10/30/12 12/03/12
1.3 1.4 1.5 1.6
NPx NPx NPx CS
Yes Yes Yes Yes Yes
3/07/13 1.7 CS *Through Formal Peer Review
UP006174
Version 1.4
2 of 136
TABLE OF CONTENTS
1. INTRODUCTION ................................................................................................................... 6 1.1 Purpose and Scope ......................................................................................................... 6 1.2 References ...................................................................................................................... 6 1.3 Software Release Numbering .......................................................................................... 6 1.4 Acronyms ........................................................................................................................ 6 1.5 Terminology..................................................................................................................... 7 2. UPGRADE OVERVIEW ........................................................................................................ 9 2.1 Upgrade Path .................................................................................................................. 9 2.2 -- New for 8.0 -- ............................................................................................................... 9 2.3 SPR Upgrade to Rel 8.0 .................................................................................................. 9 2.4 Upgrade Sequence........................................................................................................ 10 2.5 Customer Impacts ......................................................................................................... 10 2.6 Rollback (Backout) ........................................................................................................ 10 2.7 TPD Version .................................................................................................................. 10 2.8 Server Hardware Platforms ........................................................................................... 10 2.9 Loading Application software ......................................................................................... 10 2.10 Required Materials and Access ................................................................................... 11
2.10.1 Upgrade Media.................................................................................................................. 11 2.10.2 Logins, Passwords and Server IP Addresses ................................................................... 12
2.11 Upgrade Manager Process ......................................................................................... 13

2.11.1 Overview ........................................................................................................................... 13 2.11.2 Operation sequence for a site upgrade............................................................................. 15 2.11.3 Operation sequence for 8.0 Replication feature activation ............................................... 16
2.12 Firewall ....................................................................................................................... 17 2.13 Known Limitations ....................................................................................................... 17 3. UPGRADE PREPARATION ............................................................................................... 19 3.1 Prerequisites ................................................................................................................. 19 3.2 PMAC Upgrade ............................................................................................................. 19 3.3 Plan and Track Upgrades .............................................................................................. 19 3.4 Perform System Health Check (Upgrade Preparation) .................................................. 23 3.5 Firmware Upgrades ....................................................................................................... 24 3.6 Deploy Software (Upgrade Preparation) ........................................................................ 24
3.6.1 3.6.2 3.6.3 3.6.4 Deploying Upgrade Software to servers ........................................................................... 24 Copy ISO image files to Management Server (PMAC) ..................................................... 24 Copy ISO image files to servers {PP5160, DL360, c-Class} ............................................ 25 Verify CMP Software Images ............................................................................................ 30
3.7 Verify Network Firewall connectivity .............................................................................. 32 3.8 Backups and Backup Locations ..................................................................................... 32 4. SOFTWARE UPGRADE CAUTIONS.................................................................................. 34 5. UPGRADE CMP CLUSTERS ............................................................................................. 35 5.1 Upgrade Active Site CMP Cluster .................................................................................. 35
5.1.1 Make Upgraded Server Active .......................................................................................... 44 5.1.2 Upgrade Second CMP at Primary Site ............................................................................. 49
5.2 Upgrade Secondary Site CMP Cluster (if deployed by Operator)................................... 56 6. UPGRADE SITE _____________________ ....................................................................... 67 6.1 Site Upgrade Preparations ............................................................................................ 67
UP006174 Version 1.4 3 of 136
Policy 7.5 to 8.0 - Upgrade Procedure 6.1.1 6.1.2 6.1.3 6.1.4
Configuration Preparations ............................................................................................... 67 Key Exchanges from CMPs .............................................................................................. 68 Key Exchanges between servers of MPE/MRA cluster at a site ...................................... 70 Verify Deployed software images at site MPE/MRA server .............................................. 72
6.2 Upgrade MPEs - Site ________________ .................................................................... 74 6.3 Upgrade Site MRA Cluster Site ____________ .......................................................... 83 7. ACTIVATE 8.0 REPLICATION FEATURE .......................................................................... 94 8. POST UPGRADE ACTIVITIES ......................................................................................... 101 8.1 Configuration settings .................................................................................................. 101 8.2 Verify System Upgrade................................................................................................ 101 8.3 Additional Instructions ................................................................................................. 102 9. BACKOUT (ROLLBACK) ................................................................................................. 103 9.1 Backout of Replication Activation................................................................................. 104 9.2 Backout of Partial-upgraded Cluster ............................................................................ 108 9.3 Backout of Fully Upgraded Cluster .............................................................................. 112 9.4 Recovery of Server (from Backup or Initial Config) ...................................................... 116 APPENDIX A. MANAGING HA STATUS OF SERVERS ....................................................... 118 APPENDIX B. METHODS OF DELIVERING SOFTWARE UPGRADE ISO ........................... 120 APPENDIX C. UPGRADE PMAC ON A C-CLASS SYSTEM ................................................. 123 APPENDIX D. USING ILO (OR RMM) TO REMOTELY ACCESS A SERVER ....................... 130 APPENDIX E. ACCESSING TEKELECS CUSTOMER SUPPORT SITE .............................. 132 APPENDIX F. USING SCREEN SHELL TOOL ................................................................... 133 APPENDIX G. TN003516 TO CLEAN UP FILES ................................................................... 134 APPENDIX H. HANDLING ICMP BLOCKING SCENARIO .................................................... 135 APPENDIX I. REVIEW COMMENTS ...................................................................................... 136
UP006174
Version 1.4
4 of 136
List of Figures
Figure 1. Example procedure steps used in this document ........................................................................... 8
List of Tables
Table 1. Acronyms ........................................................................................................................................ 6 Table 2. Terminology.................................................................................................................................... 7 Table 3: Logins, Passwords and Server IP Addresses ................................................................................ 12 Table 4: Upgrade Manager Operations ....................................................................................................... 13 Table 5: Upgrade Manager Firewall requirements ..................................................................................... 17
List of Procedures
Procedure 1: Prerequisites........................................................................................................................... 19 Procedure 2: Plan and Track Cluster Upgrades .......................................................................................... 21 Procedure 3: Perform System Health Check (Upgrade Preparation) .......................................................... 23 Procedure 4: Copy ISO images to Management Servers (PMAC) ............................................................. 25 Procedure 5: Copy ISO images to target system......................................................................................... 27 Procedure 6: Verify CMP Software Images ............................................................................................... 30 Procedure 7: Backups (System and server) location and access ................................................................ 32 Procedure 8: Upgrade Standby server at Primary CMP Site __________________ ................................. 36 Procedure 9: Make Upgraded Server Active .............................................................................................. 44 Procedure 10: Upgrade Second CMP Server and Primary Site, and restore cluster ................................... 49 Procedure 11: Upgrade Secondary-Site CMPs ........................................................................................... 56 Procedure 12: Configuration Preparations Procedure ................................................................................. 67 Procedure 13: Key Exchanges from CMPs to MPE/MRA ......................................................................... 68 Procedure 14: Key exchanges between servers of MPE/MRA clusters...................................................... 70 Procedure 15: Verify deployed software image .......................................................................................... 72 Procedure 16: Upgrade MPEs Site........................................................................................................... 74 Procedure 17: Upgrade Site MRA .............................................................................................................. 83 Procedure 18: 8.0 Replication Activation ................................................................................................... 94 Procedure 19: Remove Replication Exclusions .......................................................................................... 98 Procedure 20: Configuration Settings ....................................................................................................... 101 Procedure 21: Verify System Upgrade ..................................................................................................... 101 Procedure 22: Backout of Replication Activation .................................................................................... 104 Procedure 23: Backout Partial-upgraded Cluster ...................................................................................... 108 Procedure 24: Backout Fully Upgraded Cluster ....................................................................................... 112 Procedure 25: Recovery of Server from Backup ...................................................................................... 116 Procedure 26: Upgrade from physical CD media {PP5160, DL360} ..................................................... 120
UP006174
Version 1.4
5 of 136
1. INTRODUCTION 1.1 Purpose and Scope

This document describes methods utilized and procedures executed to perform a Software upgrade to Release 8.0 on in-service Policy 7.5.x servers. The audience for this document includes Tekelec customers as well as these Tekelec groups: Software Development, Product Verification, Technical Communications, and Customer Service including Software Operations and New Product Introduction (NPX). The execution of this procedure assumes that the CMP, MRA and MPE Application media (ISO file, CDROM or other form of media) has already been delivered to the customers premises and delivered to the local workstation being used to perform this upgrade. The distribution of the software load is outside the scope of this procedure. The SDM-SPR Upgrade is not included in this Document.
1.2 References
[1] Policy 8.0 Release Notes [2] Feature Notice Release 8.0; 910-6405-001 Revision A; March 2012 [3] PM&C 3.x/4.x Incremental Upgrade; 909-1619-001; Rev 4.4; 02/01/12 [4] Policy 8.0 Network Architecture Planning Document, SS005938
1.3 Software Release Numbering

The Policy 8.0 is comprised of three software components, the CMP, MPE and the MRA ISOs, each versioned separately. Refer to reference [1.] Policy 8.0 Release Notes for the target release in order to identify the separate software components included in the release and their version numbers.
1.4 Acronyms
Table 1. Acronyms BIOS BMC CD-ROM FRUSDR GPS HSC IP IPM IPMI ISO MOP NEBS RMM SUP TPD UI Basic Input Output System Baseboard Management Controller Compact Disc Read-only Media Field Replaceable Unit Sensor Data Record Global Product Solutions Hot Swap Controller Internet Protocol Initial Product Manufacture Intelligent Platform Management Interface ISO 9660 file system (when used in the context of this document) Method of Procedure Network Equipment-Building System Remote Management Module System Update Package Tekelec Platform Distribution User Interface
UP006174
Version 1.4
6 of 136
1.5 Terminology
This section describes terminology as it is used within this document. Table 2. Terminology Firmware Platcfg Runlevel Coded instructions and data programmed directly into the circuitry of read-only memory for controlling the operation of the server or one of its devices. Refers specifically to the Platform Configuration Utility User Interface which is a text-based user interface. A preset TPD operating state represented by a single-digit integer which designates a different system configuration and allows access to a different combination of processes. Procedure used to determine the health and status of the server, typically performed using the TPD syscheck utility. The Tekelec PP5160 Application Server. An Intel based, 1U NEBS compliant rack-mount server. The process of converting a TPD based Policy 7.5.x server from its current software release to a newer release. State that allows for a successful software upgrade of a server. This requires bringing the server out of service and disabling certain processes. A hardware timing device that triggers the server to reset if the OS, due to some fault condition, such as a hang, neglects to regularly service the watchdog.
System Health Check PP5160 Upgrade Upgrade Ready Watchdog
The figure below shows an example of a procedural step used in this document. Each step has a checkbox that the user should check-off to keep track of the progress of the procedure. Any sub-steps within a step are referred to as Step X.Y. The example in Figure 1 shows Step 1 and Step 2.1 to Step 2.6. The title box describes the operations to be performed during that step GUI menu items, action links and buttons to be clicked on are in bold Arial font. GUI fields and values to take note of during a step are in bold Arial font. Each command that the user enters is formatted in 10-point bold Courier font. Command output is formatted in normal 8 to 10-point Courier font. Variable user-entered command line input is surrounded by angled brackets and formatted in <bold, italicized 10-point Courier> font.
UP006174
Version 1.4
7 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Figure 1. Example procedure steps used in this document
Step Procedure Change to upgrade directory Verify IPMI services have stopped Verify all modules are not loaded and that /dev/ipmi0 does not exist $ cd /var/TKLC/upgrade
1. 2.
# service ipmi status-all ipmi_msghandler module not loaded. ipmi_si module not loaded. ipmi_devintf module not loaded. /dev/ipmi0 does not exist.
3.
Mount the Update ISO
# mount -o loop /var/TKLC/upgrade/<Flash /mnt/upgrade
ISO Name>.iso
Example: # mount -o loop /var/TKLC/upgrade/872-2068-01-1.0.0_70.28.0.iso /mnt/upgrade
UP006174
Version 1.4
8 of 136
2. UPGRADE OVERVIEW
This section lists the required materials and information needed to execute a Policy 8.0 software upgrade.
2.1 Upgrade Path

The upgrade is supported from the Policy 7.5.x GA software releases.
7.5.x
8.0
2.2 -- New for 8.0 -In Release 8.0, a new Upgrade Manager function is provided to improve the upgrade process. This function is built into the 8.0 CMP. It is used to upgrade the MRE/MRAs, after the initial upgrade of the CMPs to the new release. I.e. the upgrade will be done for the CMPs first, and then the new Upgrade Manager is used to upgrade the remaining servers. This tool provides the option to upgrade multiple MPE clusters at a single site in parallel. There is also an improved data replication service implemented in Release 8.0. Because the replication service extends between servers, it is necessary to upgrade all servers in the Policy system to 8.0, and then activate the new replication service between the servers. This adds new steps to the Upgrade activities, but is also simplified by the Upgrade Manager tool. It also has a specific roll back procedure in case of a problem with the new replication service. NOTE: the new Replication service requires that certain additional tcp ports are open in the customer network between the CMP and MPE/MRAs. The upgrade to 8.0 does not require changes to existing Policies, call flows, or other design 1 activities. New Features provided in Policy 8.0 can be activated after the upgrade, on a schedule and plan that is separate from the upgrade. These new feature activations may require planning, but are outside the scope of this document. For a full list of new features in Policy Release 8.0, see the Policy 8.0 Feature release notice.
2.3 SPR Upgrade to Rel 8.0

[Note: Subscriber Profile Repository (SPR) is an optional component of the Tekelec Policy Solution. This section only applies to customers that use the Tekelec SPR.] It is recommended to upgrade to SPR 8.0 (if SPR is used in the deployment) before the upgrade to 2 Policy 8.0, but this is not required . Certain Policy 8.0 features depend on SPR 8.0 functionality and these features may not be activated till the SPR upgrade to 8.0 is completed. For a full list of new features in SPR Release 8.0, see the SPR 8.0 Feature release notice.
This is the design intent of the release. The user should confirm in the Release notes if there might be exceptions to this that need to be managed before upgrade. 2 This is the design intent of the release. The user should confirm in the Release notes if there might be exceptions to this that need to be managed before upgrade. UP006174 Version 1.4 9 of 136
2.4 Upgrade Sequence

This procedure applies to an Active/Standby pair of servers (or a single server, if it is not configured with high availability). This pair of servers will be referred to as a cluster or ha cluster. The cluster type may be CMP, MRA or MPE. For CMP cluster, the cluster status may also be Primary site or Secondary site. The customer deployment may consist of multiple clusters. Required Cluster Upgrade Sequence: 1. CMP Primary site cluster 2. CMP Secondary site cluster 3. MRA and MPE clusters 4. New Replication activation Note: There may be limitations to the CMP management functions during the period when the CMP Active site cluster is on release 8.0 and one or more MRA/MPE clusters are on release 7.5.x. For this reason, it is recommended that the deployed policies are not changed during the upgrade period.
3
2.5 Customer Impacts

The cluster upgrade proceeds by upgrading the standby server, and then switching over from the Active to the Standby, and upgrading the second server. A server boot is part of the Upgrade action.
2.6 Rollback (Backout)

Rollback is the reverse of the upgrade. The full pre-upgrade image is stored on the server during the upgrade, and can be restored from a command line.
2.7 TPD Version

The Tekelec Product Distribution (TPD) version needed for this release is included in the Policy Application Software Upgrade iso, and is upgraded also as part of this procedure.
2.8 Server Hardware Platforms

The Policy 8.0 software upgrade can be used on any server that previously had the Policy 7.5.x release. This includes the PP5160, DL360, and BL460.
2.9 Loading Application software

For upgrade of server Application software, the recommended method is to copy the Application iso to the servers using scp/ftp. If the system is C-class, the Application software must also be loaded into the PMAC software management library to support new installs and FRU activities (PMAC is not used for Upgrade). It is also possible to load software from a CD/DVD. The PP5160 and DL360 are Rack Mount servers, and have a front panel CD/DVD Drive. This can be used for the upgrade. The BL460 is a blade server and does not have a CD/DVD Drive. However, the PMAC server (provided with C-Class solutions) has CD/DVD drive that is used to load and manage Application software (and TPD) versions. Software may be copied from PMAC to a blade server.
Specific limitations are to be determined. Version 1.4 10 of 136
UP006174
2.10 Required Materials and Access

The following materials and access are needed to execute an upgrade: Required Item Policy 8.0 Software Media and TPD Notes See section 2.10.1. Either as an ISO image file or in physical CD media format.
Policy 8.0 Software Release Notes. Policy 7.x Software Media and TPD This may be needed for recovery of a server that has failed upgrade, and cannot be backed out. Note: The login may be through ssh, local console, or iLo/RMM maintenance port.
The ability to log into the target CMP servers as root. The ability to secure copy (scp) from the local workstation being used to perform this upgrade to the target CMP servers, or otherwise be able to transfer binary files to the target server. User logins, passwords, IP addresses and other administration information. VPN access to the customers network is required if that is the only method to log into the target servers.
See Section 2.10.2 It must be also possible to access the Policy Manager GUI, and the PMAC GUI (for a BL460 system). The GUIs may be tunneled via VPN for remote console access.
2.10.1 Upgrade Media

You must obtain a copy of the Policy 8.0 software target release media. The media can be in either ISO image file format or physical DVD media. It is best to have both formats available before going to site. The Policy 8.0 software ISO image files will be in the following format: 872-241z-102-8.0.0_x.y.0-mpe-x86_64.iso Where z is: 2-CMP, 3-MPE. 4-MRA, 5-MPE-LI The Upgrade Media must be also delivered to the customer site prior to the execution of this upgrade procedure, in case recovery actions from the customer site become necessary. The distribution of media is outside the scope of this procedure. If using ISO image files, it is assumed that the ISO image files have been delivered to a local workstation being used to perform this upgrade and any user performing the upgrade will have access to the ISO image files. If the user performing the upgrade is at a remote location, it is assumed that the ISO files are already available to them before starting the upgrade procedure.
UP006174
Version 1.4
11 of 136
2.10.2 Logins, Passwords and Server IP Addresses

The IP Address assignments for each site, from the appropriate Tekelec Network IP Site Survey (example: SS005938), must be available. This ensures that the necessary administration information is available prior to an upgrade. Further, need to confirm login information for key interfaces, and document in table below. [It is assumed that the logins may be common among the customer sites. If not, record for each site.] Note: Consider the sensitivity of the information recorded in this table . While all of the information in the table is required to complete the upgrade, there may be security policies in place that prevent the actual recording of this information in permanent form. Table 3: Logins, Passwords and Server IP Addresses
Item Value
CMP servers (each CMP server)
GUI Administrator Login User/Password: root password: Note: This is the password for the root login on the servers. This is not the same login as the GUI or Application Administrator.
MRE/MPA servers (each server) Target RMM/iLo (each server) Target OA (each C-class enclosure) PMAC server (each C-class site)
root password: RMM Administrator Login: User/Password OA Administrator Login: User/Password GUI Administrator Login User/Password: root password: Note: This is the password for the root login on the servers. This is not the same login as the GUI or Application Administrator.
Software Upgrade Pack Target Release 4
Target Release Number: Policy 8.0 software ISO Image (.iso) file names:
The ISO image filenames should match those referenced in the Release Notes for the target release. If using physical CD media these ISO images will be extracted from the physical media during the upgrade process. UP006174 Version 1.4 12 of 136
2.11 Upgrade Manager Process

The 8.0 CMP supports an Upgrade Manager feature which is used to upgrade MRAs and MPEs.
2.11.1 Overview
The Upgrade Manager collects and displays the Upgrade related status of the MPE and MRA servers, and provides a menu of operations for executing the required upgrade steps. Like other CMP forms, the user must have privileges to use this tool. The following operations are supported:
Table 4: Upgrade Manager Operations Name Push Tool Description Scp the upgrade script from CMP to the selected remote server(s) Expected outcomes Push Tool - Command returns Successful. Script policyUpgrade.pl is delivered to /opt/camiant/bin Force Standby Command executed successfully. Status on the Upgrade Manager shows Forced Standby. Cancel Force Standby Command executed successfully. Status on the Upgrade Manager shows Standby. Turn on Replication Command executed successfully. Upgrade Manager shows new Replication status. Turn off Replication Command executed successfully. Upgrade Manager shows new Replication status. Turn on Legacy Replication - Command executed successfully. Upgrade Manager shows new Replication status. Turn off Legacy Replication - Command executed successfully. Upgrade Manager shows new Replication status. Start Upgrade - Command executed successfully. Upgrade Manager shows status of upgrade. Version 1.4 The command call in remote server policyUpgrade.pl --pushTool
Force Standby
Force the selected server(s) to standby
SOAP (HTTP/HTTPS)
Cancel Force Standby
Cancel the selected server(s) force standby
SOAP (HTTP/HTTPS)
Turn on Replication
Turn on the selected server(s) Replication mode.
SOAP (HTTP/HTTPS)
Turn off Replication
Turn off the selected server(s) Replication mode.
SOAP (HTTP/HTTPS)
Turn on Legacy Replication
Turn on the selected server(s) Legacy Replication mode
policyUpgrade.pl setVal Cm5Compat=1 SyncMastering=1 policyUpgrade.pl setVal Cm5Compat=0 SyncMastering=0
Turn off Legacy Replication
Turn off the selected server(s) Legecy Replication mode
Start Upgrade
Kick off an upgrade on the selected server(s).
policyUpgrade.pl -startUpgrade
UP006174
13 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Backout Initiate a backout on the selected server(s). Backout - Command executed successfully. Upgrade Manager shows status of upgrade.
Software Upgrade Procedure policyUpgrade.pl --backOut
Example View of Upgrade Manager Form:
UP006174
Version 1.4
14 of 136
2.11.2 Operation sequence for a site upgrade

The following is the normal sequence for upgrade at a site (MPEs and MRAs), using the Upgrade Manager: Pre-requisites: All CMPs in Topology are already upgraded to 8.0. Application iso is deployed to the server /var/TKLC/Upgrade directory. Ssh key exchange is completed between CMP and every MPE/MRA
st
Steps upgrade 1 of 2 servers in a cluster: 1. Push the Upgrade script from CMP to all MPE/MRAs at a site 2. Select some or all standby servers and execute Forced Standby 3. Select forced standby servers from previous step, and execute Upgrade 4. Confirm Upgrade completions for these servers 5. Execute ivi NodeInfo to force SwitchOver 6. Confirm 8.0 servers become Active Steps upgrade 2
nd 5
of 2 servers in a cluster:
This is a command line activity from the Active CMP server. This step is required for 7.5.x 8.0 Upgrade. Future upgrades to maintenance releases of 8.0 are not expected to need this command line method, and will use an appropriate Upgrade Manager operation instead. UP006174 Version 1.4 15 of 136
Policy 7.5 to 8.0 - Upgrade Procedure 1. Select some or all 7.5.x standby servers and execute Forced Standby
2. Select forced standby servers from previous step, and execute Upgrade 3. Confirm Upgrade completions for these servers 4. Cancel Forced Standby on these servers 5. Confirm 8.0 servers Standby Post-requisites: 1. Remove Application iso from the servers /var/TKLC/Upgrade directory.
2. Activate 8.0 Replication (see following section)
2.11.3 Operation sequence for 8.0 Replication feature activation

After an Upgrade to 8.0 from 7.5.x, the Replication mode will initially be set to Legacy Replication (pre-8.0 mode). It is important to then activate the 8.0 Replication feature. The Upgrade Manager supports the activation of the 8.0 Replication feature, and rollback of this feature. This is performed after all servers in the network are upgraded to Rel 8.0, and should be performed in a Maintenance window (period of low traffic). [Roll back of this feature activation is supported, but may have impacts on subscribers.] Below is a summary of this feature activation sequence. A detailed procedure is provided later in this document. Summary of 8.0 Replication feature activation: Pre-requisites: All servers in Topology are upgraded to Release 8.0 All servers are currently using Legacy Replication All steps must be executed in a single Maintenance Window
Steps Activate 8.0 Replication feature (cluster at a time) Note: Each operation must be acknowledged or completed before proceeding to the next step. 1. At Upgrade Manager, Select standby server of a MPE/MRA cluster, and execute: a. Replication Off b. Legacy Replication Off c. Replication On
2. Select active server of cluster, and execute: a. Replication Off b. Legacy Replication Off c. Replication On
3. -- Repeat 1,2 for every MPE/MRA Cluster in the Network -4. At Upgrade Manager, Select standby server of CMP Active site cluster, and execute: a. Replication Off b. Legacy Replication Off c. Replication On
5. Select active server of CMP Active site cluster, and execute: a. Replication Off UP006174 Version 1.4 16 of 136
Policy 7.5 to 8.0 - Upgrade Procedure b. Legacy Replication Off c. Replication On 6. Repeat 4,5 for CMP Secondary Cluster --
2.12 Firewall
The following protocol ports need to be open on the MPE/MRAs for the Upgrade Manager. Table 5: Upgrade Manager Firewall requirements Component Tomcat SSH Comcol Port/Protocol 80/HTTP 8443/HTTPS 22/TCP 15360/TCP (cmsoapa) 16810/TCP (inetsync) 17398/TCP (inetrep) 17400/TCP (inetrep) 17401/TCP (cmha) 16878/TCP (inetmerge) 15616/TCP (Imysqld)
For Policy 8.0, the inetrep ports are new, and are used by the new Replication software. The 17400 port is opened for listening on each MPE/MRA blade, and the Active CMP connects to this port for replication. Active CMP connects to -- MPE/MRA Port 17400 Primary Site Active CMP connects to Secondary Site Active CMP Port 17400 The 17400 port is also opened on the Standby MPE/MRA/CMP and for used for replication from Active to Standby servers in a cluster. Active MPE connects to -- Standby MPE Port 17400 Active MRA connects to -- Standby MRA Port 17400 Active CMP connects to -- Standby CMP Port 17400 The same Firewall rules should also be applied to Port 17389, but this port may not be actively used unless the MPE/MRA Geo-Redundancy feature is enabled.
2.13 Known Limitations

The following is a list of Known Limitations for the Operations of the system, during the period that the system is being upgraded (when there is a mix of 9.0 and 7.5 servers in the network).
The Backout of a fully upgraded MPE/MRA cluster will result is the loss of all state data (call sessions) for existing calls in-progress. The Backout of a partially upgraded MPE/MRA cluster will result in the resumed use of state data that will be partially out-of-date. o The 7.5 server will retain its state data after traffic is switched to the 9.0 server of the cluster, but this data will no longer be updated from traffic activity at the 9.0 server. If
Version 1.4 17 of 136
UP006174
the traffic is switched back to the 7.5 server, it will resume traffic handling with the state data it has, which will be partially out-of-date depending on how long the 9.0 server was Active. The Mode settings must not be changed during upgrade interval o After upgrading the CMP, if you change modes, then 7.5 MPEs are unable to process quota correctly (because UseLocalQuota gets set to true) o After upgrading both the CMP and the MPE, then we can no longer terminate Gy sessions (because the quota is all messed up) The Topology settings (Platform Settings Topology) must not be changed during upgrade interval. o Replication of the Topology settings is disabled during the upgrade interval. Changes will not be handled correctly at the CMP or target servers. If the icmp is blocked between the servers o Check Appendix H: Handling ICMP Blocking scenario.
UP006174
Version 1.4
18 of 136
3. UPGRADE PREPARATION
This section provides detailed procedures to prepare a system for upgrade execution. These procedures are executed outside a maintenance window.
3.1 Prerequisites
This procedure verifies that all required prerequisite steps needed to perform an upgrade have been completed.
Procedure 1: Prerequisites
Step Procedure Verify all required materials are present Review Release Notes
1.
Materials are listed in Section 2.10: Required Materials. Verify required materials are present. Review Policy 8.0 software Upgrade Release Notes for the target release for the following information: Individual Software components and versions included in target release Issues (Tekelec PRs) resolved in target release Known Issues with target release Any further instructions that may be required to complete the Software Upgrade for the target release
2.
3.
4.
Verify all administration data needed during upgrade Contact Tekelec Customer Care Center
Double-check that all information in Section 2.10.2 is filled-in and accurate.
Contact the Tekelec Customer Care Center and inform them of your plans to upgrade this system.
3.2 PMAC Upgrade

Policy Release 8.0 includes an upgrade to the Management Server (PMAC) for C-class installations. The PMAC version is a major upgrade, from release 3 to release 4, and includes changes to the look and feel of the GUI, better reliability and improved Software Inventory function. Functionality remains the similar to previous release, and changes are easy to learn. This upgrade is backwards compatible to Policy 7.5 installations, and can be performed without any risk of network impacts. See Appendix C for instructions to Upgrade PMAC.
3.3 Plan and Track Upgrades

The procedures in this document divide the Upgrade into 3 steps: UP006174 Upgrade CMP clusters Upgrade MPE and MRA clusters, 1 site at a time Activate the 8.0 Replication feature Version 1.4 19 of 136
The following Procedure must be completed before the Upgrade begins, to identify the Clusters to be upgraded and plan the work. It can also be used to track the completion of the upgrades, and assign work to different engineers. The MPE Upgrades can be done in parallel. Notes: No Policy Changes or Configuration change may be made while the system is in mixedmode. Time estimates are for upgrade activity without roll back. Roll back time is typically same or less than upgrade time. On C-class systems, the PMAC server must be upgraded before upgrading of the Policy application servers. There is a separate procedure for PMAC Upgrade.
UP006174
Version 1.4
20 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 2: Plan and Track Cluster Upgrades
Step Procedure Use the following Checklist to plan the Cluster upgrades for the entire system. PRIMARY Site CMP cluster Upgrade Result Maintenance Windows are planned
Engineer
Time
1.
2.
1 hr Site Name ________________
3.
SECONDARY- Site CMP cluster Upgrade
30 min Site Name _________________
4.
MPE/MRA clusters Upgraded at Site 1
2 hrs Site Name __________________ Cluster List: Cluster Name Hostname 1 Hostname 2 Completed?
5.
MPE/MRA clusters Upgraded at Site 2
2 hrs Site Name _________________ Cluster List:
Cluster Name
Hostname 1
Hostname 2
Completed?
6.
MPE/MRA clusters Upgraded Site 3
2 hrs Site Name _________________ Cluster List:
Cluster Name
Hostname 1
Hostname 2
Completed?
UP006174
Version 1.4
21 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 2: Plan and Track Cluster Upgrades
Step Procedure MPE/MRA clusters Upgraded Site 4 Result Site Name _________________ Cluster List:
Engineer
Time 2 hrs
7.
Cluster Name
Hostname 1
Hostname 2
Completed?
UP006174
Version 1.4
22 of 136
3.4 Perform System Health Check (Upgrade Preparation)

This procedure is part of Software Upgrade Preparation and is used to determine the health and status of the servers to be upgraded. This must be executed at least once within the time frame of 24-36 hours prior to the start of a maintenance window.
Procedure 3: Perform System Health Check (Upgrade Preparation)

Step Procedure Login to Manager (CMP) GUI View Active Alarms Identify the cause of any active alarms, and determine if these may have impact on the upgrade. Export current Alarms to file, and save. Verify that the system is running within expected parameters. Export current KPIs to file, and save. Confirm that any needed Technical Service Bulletins (TSB) have been applied. Specifically: TN003475 Procedure to reset upsynclog configuration parameters back to designed values must be applied. This TSB requires that a command is run on each server in the system to confirm that the configuration is set correctly. If is it not, a procedure must be applied (in a maintenance window) to set the correct configuration value. Verify Command (run on each server): Result
1.
2. 3. 4.
View KPI Dashboards Confirm TSB have been applied (as needed)
iqt -p PartAttrDef where "partDefRecNum='UpSyncLog' and attr='KeepCount'"

Correct Result: recNum partDefRecNum attr val 135 UpSyncLog KeepCount 0 See the TSB for the procedure to repair, if needed. On 7.5 MPE/MRA servers, the following file may be missing, and this will cause a Upgrade failure.
5.
Fix for Missing file
Check if this file exists before upgrade. # ls /opt/camiant/smsr/smscfg/log4j.properties If not: # touch /opt/camiant/smsr/smscfg/log4j.properties 6.
Check for Custom settings, Advanced Settings record these At some sites, there may have been customizations applied, for various reasons. Need to identify these settings, and consult with Tekelec support if these settings will need to be re-applied after Upgrade to 8.0.
UP006174
Version 1.4
23 of 136
3.5 Firmware Upgrades

Tekelec notifies customers when critical Firmware upgrades are needed. However, other non-critical firmware updates may be available, and recommended. In general, the upgrade of the Policy application does not depend on these firmware upgrades. However, it is strongly recommended deploy any recommended firmware upgrades. This is typically deployed before Policy 9.0 upgrade. Current Recommended HP Firmware delivery: HP Service Pack for ProLiant 2.2.1 ISO HP Misc Firmware 2.2.1 ISO Upgrade Procedures Document Release Notes P/N: 875-1124-101 P/N: 875-0903-212 P/N: 909-2234-001 Rev A P/N: 910-6585-001 Rev A ISO: 872-2488-101-2.2.1-10.26.0.iso ISO: 872-2161-114-2.2.1_10.27.0.iso
This Tekelec HP Firmware rev includes Release notes that identify the latest firmware revs for each of the system components. Firmware upgrade procedures are not included in this document. Note: no Firmware upgrade is currently required for the PP5160 server.
3.6 Deploy Software (Upgrade Preparation)

This procedure is intended for remote execution of the Upgrade. Software should be deployed to each Policy server upgrade directory, before the actual upgrade activities. This will typically be done with scp, wget or ftp. Because of the large size of the software isos, sufficient time should be planned to accomplish this step. It is recommended to copy the iso images to a server at the site, and then re-distribute the iso images to the other servers at the site. This allows faster transfer times, and allows the host names to be used during the transfer (which reduces the possibility of the wrong image being deployed to a server).
3.6.1 Deploying Upgrade Software to servers

There are three Software Images in this upgrade (CMP, MRA or MPE/MPE-LI). A single image must be deployed to the Upgrade directory of each server to be upgraded, where the image is the correct type for that server. i.e. the New CMP software image must be deployed to the CMP servers, the new MRA image deployed to the MRA servers, and the MPE image deployed to the MPE servers. IMPORTANT: If the deployed image type (CMP, MRA, MPE) does not match the existing installed software type, the upgrade will fail. Example: an attempt to Upgrade a CMP with a MPE software image will fail during the Upgrade action. [Note: To change a server from one application type to another, the server must first be cleaned of all application software by an Install OS acti on, and then the new Application type installed.]
3.6.2 Copy ISO image files to Management Server (PMAC)

If the system has a Management Server (PMAC), then the following procedure must be applied for each PMAC server in the system. IMPORTANT: PMAC should be upgraded from 3.0 to 4.0 release prior to this step. See Appendix C. This procedure transfers software Upgrade isos to the PMAC, and loads isos into the PMAC Software Image repository. UP006174 Version 1.4 24 of 136
Note: ISO transfers to the target systems may require a significant amount of time depending on the number of systems and the speed of the network. The ISO transfers to the target systems should be performed prior to, outside of, the scheduled maintenance window. Schedule the required maintenance windows accordingly before proceeding. Note: Because the iso images are large, the procedure includes instructions to check space available in the /var/TKLC/upgrade directory before copying the isos to this directory. After the Add Image action on the PMAC, the iso images are registered in PMAC, and stored in the /var/TKLC/smac/image/ directory which is very large. After this step, the added images can then be removed from the /var/TKLC/upgrade directory.
Procedure 4: Copy ISO images to Management Servers (PMAC) Pre-requisite: PMAC is upgraded to 4.0 release.
Step Procedure PMAC GUI: login to pmac as pmacadmin
Open the PMAC GUI, select Software Manage Software Images Determine what Images are installed, and confirm that new images are not yet installed.
Result
1.
2.
WinSCP to PMAC server, login as root
From workstation with ISO Images, open WinSCP (or similar tool), ad login as root.
3.
WinSCP: Change Target Directory to /var/TKLC/upgrade WinSCP: Remove existing ISOs from /var/TKLC/upgrade WinSCP: Copy ISO image to PMAC
Change Target Directory to /var/TKLC/upgrade
4.
To keep this directory space free, remove any existing ISOs
5.
Copy a ISO image to PMAC /var/TKLC/Upgrade directory
6.
PMAC GUI: Add Image
Software Manage Software Images Execute Add Image Select the ISO that was just copied to the PMAC server.
7.
PMAC GUI: Verify Image is added
Software Manage Software Images The just added image will show in this view after about 1 minute. Note: added images are stored in /var/TKLC/smac/image
8.
Repeat above steps for all images
Repeat above steps for all images

- TPD Software ISO for Policy - Policy CMP Iso - Policy MPE (or MPE-LI) Iso - Policy MRA Iso
3.6.3 Copy ISO image files to servers {PP5160, DL360, c-Class}

This procedure applies to all Server types. For c-Class installations, the previous procedure must also be executed. It is assumed that there is scp access to each server to be upgraded, and that the Application software images are available in a .iso format that can be copied to the servers. UP006174 Version 1.4 25 of 136
For PP5160 or DL360, it is also possible to use the CD drive on the server to perform the upgrade, as an alternative to copying the iso file to the server. For C-class servers, the PMAC server may be used to load the isos from the PMAC CD drive. Note: ISO transfers to the target systems may require a significant amount of time depending on the number of systems and the speed of the network. The ISO transfers to the target systems should be performed prior to, outside of, the scheduled maintenance window. Schedule the required maintenance windows accordingly before proceeding. The iso images are put in the /var/TKLC/upgrade directory on each server. Because the iso images are large, the following procedure includes instructions to check space available before copying the iso to this directory. The Upgrade command, used later in the procedure, will look in this directory for available upgrades, and present a list to the user.
UP006174
Version 1.4
26 of 136
Procedure 5: Copy ISO images to target system

Step Procedure Select Server at the upgrade site to use as Distribution point Result
1.
This procedure will select a server at the site to copy the iso images to, and then this server will be used to re-distribute iso images to the other servers at the site. Distribution Server at site (CMP, MPE, MRA): Ex: -cmp_ip_address
2.
Ssh to Distribution Server: 1) Access the login prompt. 2) Log into the server as the root user on the iLO or RMM.
login as: root password: <enter password>
3.
Verify enough space exists for ISO Verify that there is at least 1G in the Avail column. If not, clean up files until there is space available. Make sure you know what files you can remove safely before cleaning up. It is recommended that you only clean up files in the /var/TKLC/upgrade directory as this is a platform owned directory that should only contain ISO images. This directory should not be expected to contain images for any length of time as they can get purged. Removing files other than those in directory /var/TKLC/upgrade is potentially dangerous.
Cleanup un-needed iso files in upgrade directory. # ls /var/TKLC/upgrade If needed: # rm * /var/TKLC/upgrade/*.iso Check disk space available # df -h /var/TKLC
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vgroot-plat_var_tklc 3.9G 174M 3.6G 5% /var/TKLC
4.
Copy a Policy 8.0 software ISO image file from the local workstation to the target server upgrade directory. Image will be one of: CMP, MRA or MPE.
From the local workstation: (use WinSCP, or equivalent) Copy Policy 8.0 software ISO to target server
# scp <ISO Name> root@<server IP>:/var/TKLC/upgrade

Example for CMP iso: # scp 872-2472-101-8.0.0_18.2.0-cmp-x86_64.iso root@xx.xx.xx.xx:/var/TKLC/upgrade
UP006174
Version 1.4
27 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 5: Copy ISO images to target system
Step Procedure Verify ISO image file is copied to correct location. Examine output of the command and verify that the ISO file is present and that file size appears correct. Validate the Policy 8.0 software ISO
Rel 8.0 Application Part Numbers: cmp mpe mpe-li mra 872-2472-101 872-2473-101 872-2474-101 872-2475-101 From the Distribution server:
Result
5.
# ls l /var/TKLC/upgrade-rw-r--r-- 1 root root 863408128

Jul 24 14:27 872-2472-101-8.0.0_18.2.0-cmp-x86_64.iso
6.
# su platcfg
Maintenance Upgrade Validate Media
Note: the iso type may not be shown in the iso name, as in the example above. In this case, the following part numbers identify the applications types for Rel 8.0:
Choose Iso to validate, and enter

Expected Result: UMVT Validate Utility v2.2.1, (c)Tekelec, June 2010 Validating /var/TKLC/upgrade/cmp_8.0_18.2.0.iso Date&Time: 2011-11-01 09:24:39 Volume ID: tklc_872-2472-101_Rev_A_18.2.0 Part Number: 872-2472-101 Version: 18.2.0 Disc Label: cmp Disc description: cmp The media validation is complete, the result is: PASS CDROM is Valid
Note: Do not continue if ISO image validation reports any errors or is invalid. Instead remove the ISO file and either re-copy it to the target system or regenerate it from physical media.
UP006174
Version 1.4
28 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 5: Copy ISO images to target system
Step Procedure Re-Distribute iso to other servers of this type at the site This step will depend on the iso type.
From the Distribution server:
Result
7.
# cat /etc/hosts | grep <cmp, mpe, mra>

Example : [root@slak-cmp-a upgrade]# cat /etc/hosts | grep cmp 10.250.85.25 slak-cmp-a 10.250.84.25 brbg-cmp-a 10.250.84.26 brbg-cmp-b 10.250.85.25 slak-cmp-a 10.250.85.26 slak-cmp-b For the servers of the type for this iso:
# ssh <hostname> # ls -l /var/TKLC/upgrade

[make sure this directory is empty. If not, remove any existing iso files]
# exit Copy software to CMPs: # scp 872-2472* <cmp_hostname>:/var/TKLC/upgrade Copy software to MPEs: # scp 872-2473* (or 872-2474*) <mpe_hostname>:/var/TKLC/upgrade Copy software to MRAs: # scp 872-2475* <mra_hostname>:/var/TKLC/upgrade
Remove iso from Distribution server (if it is not needed for this server)
If the iso file is not needed at the Distribution server, delete it. [Example: Distribution server is CMP, and iso is MPE.] From the Distribution server:
8.
# ls /var/TKLC/upgrade Remove CMP iso from non-CMP distribution server: # rm /var/TKLC/upgrade/872-2472* Remove MPE iso from non-MPE distribution server: # rm /var/TKLC/upgrade /872-2473* (or 872-2474*) Remove MRA iso from non-MRA distribution server: # rm /var/TKLC/upgrade /872-2475*
Repeat steps 4 9 for each server type (CMP, MPE, MRA) at the upgrade site. This procedure needs to be repeated for each site to be upgraded. Steps 4 9 must be repeated for each server type at the target site to be upgraded.
9.
10.
This procedure needs to be repeated for each site to be upgraded.
Procedure is completed
UP006174
Version 1.4
29 of 136
3.6.4 Verify CMP Software Images

Detailed steps are shown in the procedure below to verify that the image files are correctly deployed and ready for upgrade activity on the CMPs. [A similar step will be done later for the upgrade of the MPE/MRA servers.]
Procedure 6: Verify CMP Software Images

Step Procedure SSH: Primary Active CMP Log into the server as the root user login: root Password: <root_password> Result
1.
2.
SSH: Verify Image is deployed at Primary CMP cluster
Rel 8.0 Application Part Numbers: cmp mpe mpe-li mra 872-2472-101 872-2473-101 872-2474-101 872-2475-101
Verify that the correct software iso is loaded on the server. IMPORTANT: if the iso is the wrong type (example: cmp iso loaded on a current mra configured server), the upgrade step for this server will fail and the server will need to be re-installed from the Install OS step. # getPolicyRev 7.5.x_x.x.x # getPolicyRev p cmp # ls -l /var/TKLC/upgrade total 706236 -rw-r--r-- 1 root root 863408128 Jul 3 03:04 cmp-8.0.0_18.2.0--872-2472-101--x86_64.iso Verify that the iso matches the correct part number for this server function (cmp), and Verify there is only one iso in this directory.
UP006174
Version 1.4
30 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 6: Verify CMP Software Images
Step Procedure Validate iso image Result
3.
This step will validate the iso image: # su platcfg

Maintenance Upgrade Validate Media
Choose Iso to validate, and enter

Expected Result: UMVT Validate Utility v2.2.1, (c)Tekelec, June 2010 Validating /var/TKLC/upgrade/cmp_8.0_18.2.0.iso Date&Time: 2011-11-01 09:24:39 Volume ID: tklc_872-2472-101_Rev_A_18.2.0 Part Number: 872-2472-101 Version: 18.2.0 Disc Label: cmp Disc description: cmp The media validation is complete, the result is: PASS CDROM is Valid
Note: Do not continue if ISO image validation reports any errors or is invalid. Instead remove the ISO file and either re-copy it to the target system or regenerate it from physical medi
# exit 4.
If iso image is not found, or not valid
If iso image is not found, or not valid, re-deploy the correct iso image.
5.
Repeat steps 2-4 for each of the remaining CMP servers in the network.
Primary Standby CMP _________________ Secondary Active CMP _________________ Secondary Standby CMP ________________
UP006174
Version 1.4
31 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 6: Verify CMP Software Images
Step Procedure SSH: Primary Active CMP Verify SSH Key Exchange for CMP cluster Result
6.
# ssh <Primary Standby CMP> Confirm that there is no password prompt. If needed, perform ssh Key Exchange: # su platcfg Camiant Configuration Exchange SSH keys OK # ssh <Secondary Standby CMP> Confirm that there is no password prompt. If needed, perform ssh Key Exchange: # su platcfg Camiant Configuration Exchange SSH keys OK Procedure is completed
7.
SSH: Secondary Active CMP Verify SSH Key Exchange for CMP cluster
3.7 Verify Network Firewall connectivity

Verify that the additional firewall connectivity needed for Rel 8.0 is implemented in the network. Specifically: See detail from Policy 8.0 Network Architecture Planning Document (NAPD) (see References)
3.8 Backups and Backup Locations

IMPORTANT: Backups for servers, and the system, must be collected and readily accessible for recovery operations. Consider doing this before each major activity. Backup collection should be automated for a customer deployment, so identify the location and access method for these backups. If needed, perform manual backups.
Procedure 7: Backups (System and server) location and access

Step Procedure Result
UP006174
Version 1.4
32 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 7: Backups (System and server) location and access
Step Procedure Identify Backups Location Backup location is: Result
1.
____________________________________________________ Instructions to access to backups are as follows:
______________________________________________ ______________________________________________ ______________________________________________
UP006174
Version 1.4
33 of 136
4. SOFTWARE UPGRADE CAUTIONS
Call the Tekelec Customer Care Center at 1-888-FOR-TKLC (1-888-367-8552); or 1-919-460-2150 (international) prior to executing this upgrade to ensure that the proper media are available for use. Before upgrade, users must perform the system health check Section 0. This check ensures that the system to be upgraded is in an upgrade-ready state. Performing the system health check determines which alarms are present in the system and if upgrade can proceed with alarms.
**** WARNING *****

If the server being upgraded is not in a Normal state, the server should be brought to the Normal state before the upgrade process is started. [Normal state is generally determined by lack of alarms.]
**** WARNING *****

Please read the following notes on upgrade procedures: Where possible, command response outputs are shown as accurately as possible. EXCEPTIONS are as follows: Session banner information such as time and date. System-specific configuration information such as hardware locations, IP addresses and hostnames. ANY information marked with XXXX or YYYY. Where appropriate, instructions are provided to determine what output should be expected in place of XXXX or YYYY Aesthetic differences unrelated to functionality such as browser attributes: window size, colors, toolbars and button layouts. After completing each step and at each point where data is recorded from the screen, the technician performing the upgrade must initial each step. A check box should be provided. For procedures which are executed multiple times, the check box can be skipped, but the technician must initial each iteration the step is executed. The space on either side of the step number can be used (margin on left side or column on right side). Captured data is required for future support reference if Tekelec Technical Services is not present during the upgrade. Any CLI level windows should be logged.
UP006174
Version 1.4
34 of 136
5. UPGRADE CMP CLUSTERS

This procedure will upgrade the Primary site CMP cluster, and then upgrade the optional Secondary site CMP cluster, in a single maintenance window. Notes: Once the first CMP at the Active site is upgraded and made active, the other CMPs will report an alarm condition to indicate that they are not able to sync to the Active CMP. As the other CMPs are upgraded, they will re-sync. The Upgraded CMPs can perform basic monitoring of the 7.5.x MRAs/MPEs, to allow the migration of these elements to be spaced over several maintenance windows. However, configuration changes must not be performed for these elements. New Policies should not be introduced/deployed during the period when the Policy elements are being upgraded. Once all elements are on the new release, Policy activities can resume. Rollback option is supported at each step of the upgrade
5.1 Upgrade Active Site CMP Cluster

This procedure should be executed inside of a Maintenance window. It is assumed that the CMPs may be deployed as 2 geo-redundant clusters, identified as Primary (active) Site and Secondary (non-active) Site. [However, a geo-redundant CMP configuration is not required.] This section will upgrade the Primary site CMP Cluster, and the next section will upgrade the Secondary Site CMP Cluster. Both may be completed in a single Maintenance window.
Identify the CMPs Sites to be upgraded here, and verify which site is Primary and Secondary: CMP Site Geo Status Primary Site Secondary Site Operator Site Name Site Designation from Topology Form (aka; Site 1 or Site 2)
Note the Information on this CMP cluster: Cluster Name________________________ Server-A Hostname ___________________ Server-A IP _________________________ Server-A Status ______________________ Server-B Hostname ___________________ Server-B IP _________________________ Server-B Status ______________________
IMPORTANT: CMP servers MUST be upgraded before the MRA or MPE servers.
UP006174
Version 1.4
35 of 136
WARNING: Known Problem in 8.0.0_29.1.0 Release: 7.x MPE/MRA may show No Response on 8.0 CMP GUI After Upgrade of the CMP cluster, the CMP may randomly fail to report statistics and configuration data from one or more MPE/MRAs in the network. In the KPI Dashboard, the cluster will show with a RED indicator and No Response. The MPE/MRA cluster will continue to perform as before, but status information is limited. The work around is to continue with the upgrade of the affected MPE/MRAs. Once upgraded to 8.0, this problem is resolved. It is also important, because of this, to confirm status of MPEs/MRAs before upgrade of the Active site CMPs. WARNING: Known Problem in 8.0.0_29.1.0 Release: Upgrade fail with error of missing __db.4 file Upgrade of a server may fail if the Upgrade Manager form is open during the upgrade. This is caused by the Upgrade Manager polling the server for status during upgrade. The work around is to Close the Upgrade Manager form while the server upgrade is in-progress for any server.
Procedure 8: Upgrade Standby server at Primary CMP Site __________________ Pre-requisites: Procedure 6 is completed which copies the newest policyUpgrade.pl script onto the Primary Active CMP server
Step
Procedure
Result
UP006174
Version 1.4
36 of 136
Policy 7.5 to 8.0 - Upgrade Procedure 1.

GUI: Identify the CMP Cluster to be Upgraded From CMP Manager: Platform Setting Topology Setting
Select the (Primary) CMP Cluster to be Upgraded and View Status (Primary CMP Site):
2.
GUI: Identify the first server in cluster to be Upgraded
Record the CMP Server with Status standby that will be upgraded first. Server to Upgrade First ________________________
UP006174
Version 1.4
37 of 136

GUI: Make Standby server Forced Standby
From Manager GUI, Topology view, select Modify for the current Standby CMP server Topology CMP Primary Site Cluster View Modify Server A or B Set the Forced Standby checkbox on Standby Server Save the form.
An Alarm will occur to indicate that the Active CMP server has lost sync with the Standby CMP Server. (this may take a minute to appear in the Active Alarms list)
4.
SSH: Login to Primary Active CMP server via ssh
CentOS release 4.6 (Final) Kernel 2.6.18-1.2849prerel3.3.0_63.1.0 on an i686 localhost login: root Password: <root_password>
Verify that this is the Active CMP: # ha.stat 5.

SSH: Primary Active CMP Extract Upgrade script from CMP iso
# cd /var/TKLC/upgrade # ls 872-2472-102-8.0.0_29.1.0-cmp-x86_64.iso # mount -o loop /var/TKLC/upgrade/872-2472* /mnt/upgrade # cp /mnt/upgrade/upgrade/policyScripts/policyUpgrade.pl /opt/camiant/bin # umount /mnt/upgrade
UP006174
Version 1.4
38 of 136

SSH: Primary Active CMP: Prepare Servers for Upgrade. Disable replication for certain data tables.
Execute command to Disable Replication for certain data tables.
# iqt -p NodeInfo
nodeId nodeName hostName inhibitFlag nodeCap excludeTables A3411.121 slak-cmp-b slak-cmp-b,10.250.85.26 MasterCapable A3411.190 slak-cmp-a slak-cmp-a,10.250.85.25 H MasterCapable C1428.038 slak-mpe-07a slak-mpe-07a,10.250.85.28 MasterCapable C1428.073 slak-mpe-07b slak-mpe-07b,10.250.85.29 MasterCapable C3265.167 slak-mra-b slak-mra-b,10.250.85.5 MasterCapable C3265.212 slak-mra-a slak-mra-a,10.250.85.4 MasterCapable C3573.020 slak-mpe-01a slak-mpe-01a,10.250.85.7 MasterCapable C3573.027 slak-mpe-01b slak-mpe-01b,10.250.85.8 MasterCapable
# policyUpgrade.pl --prepareUpgrade
Verify that file is updated with excludeTables LongParam,AppEventDef,HaCfg
# iqt -p NodeInfo
nodeId nodeName hostName inhibitFlag nodeCap excludeTables A3411.121 slak-cmp-b slak-cmp-b,10.250.85.26 MasterCapable LongParam,AppEventDef,HaCfg A3411.190 slak-cmp-a slak-cmp-a,10.250.85.25 H MasterCapable LongParam,AppEventDef,HaCfg C1428.038 slak-mpe-07a slak-mpe-07a,10.250.85.28 MasterCapable LongParam,AppEventDef,HaCfg C1428.073 slak-mpe-07b slak-mpe-07b,10.250.85.29 MasterCapable LongParam,AppEventDef,HaCfg C3265.167 slak-mra-b slak-mra-b,10.250.85.5 MasterCapable LongParam,AppEventDef,HaCfg C3265.212 slak-mra-a slak-mra-a,10.250.85.4 MasterCapable LongParam,AppEventDef,HaCfg C3573.020 slak-mpe-01a slak-mpe-01a,10.250.85.7 MasterCapable LongParam,AppEventDef,HaCfg C3573.027 slak-mpe-01b slak-mpe-01b,10.250.85.8 MasterCapable LongParam,AppEventDef,HaCfg
Note: this change is automatically replicated to all 7.5.x servers from the Active CMP, and notifies the servers not to process any further updates to these tables. This step is needed since the upgraded CMPs (8.0) may send table updates to the 7.5.x servers that they will not be able to process correctly. Note: This Minor Alarm may be expected from servers
31101 - GN_WARNING/WRN configuration change forcing re-init [SyncMaster.cxx:587], but will clear itself very quickly.
7.
SSH/Console: Login to CMP Force Standby server Either: 1) SSH - Access the login prompt. 2) Log into the server as the root user on the iLO or RMM, and access Remote Console
UP006174
Version 1.4
39 of 136

SSH/Console: Verify ha status at Standby CMP
# ha.stat
Result will show ColdStandby and Offline. 9.

SSH/Console: Verify Application is running
# service qp_procmgr status qp_procmgr (pid 13410) is running...
UP006174
Version 1.4
40 of 136

SSH/Console: Apply Upgrade at Primary Standby CMP Note: the Application is NOT shutdown prior to the upgrade. This will take ~20 minutes
If using ssh, execute screen to prevent hang-ups, and do not exit this screen session till the server reboots. # screen # su - platcfg Maintenance Upgrade Initiate Upgrade
Monitor the ssh window output for the first few minutes, in case the upgrade fails during its pre-upgrade checks. After this, output will then show that software packages are being upgraded. This will take 15 - 20 minutes. The SSH session will close as the server reboots. Re-boot will take several minutes.
Manager GUI Activity: --------------------There will be a Major alarm 70005 for CMP cluster. Also multiple Minor Database replication Alarms: 31101, 31102, 31106, 31107, 31114. KPI Dashboard, and PCRF and MRA reports will show that traffic is proceeding as normal.
UP006174
Version 1.4
41 of 136

SSH/Console: Login again to upgraded server. If login using the Console or Remote Console, verify that server returns to the login prompt after boot. SSH/Console: Verify software versions
12.
# getPlatRev 5.0.1-72.45.0 # getPolicyRev 8.0.0_x.x.x
13.
SSH/Console: Verify success of Upgrade
# tail /var/TKLC/log/upgrade/upgrade.log The following indicates SUCCESS of Upgrade.
IF UPGRADE_STATUS is not equal to SUCCESS, then collect upgrade.log for analysis. See step 18.
14.
SSH/Console: Verify Status of server processes (Server is still Forced Standby)
# ha.mystate
resourceId DbReplication VIP QP DbReplication_old role Stby Stby Stby Stby node A3411.190 A3411.190 A3411.190 A3411.190 lastUpdate 0809:165127.110 0809:165127.123 0809:165148.757 0809:165127.112
NOTE: It may take a few minutes (after the system has re-booted) for all the processes to reach the Stby state. They may initially be shown as OOS. 15.
SSH/Console: Verify replication
# inetstat
16.
SSH: Primary Active CMP Verify HA status
In SSH session with Primary Active CMP server, verify upgraded server is ColdStandby. # ha.stat
node role cs-tb31-cmpa ProvideSvc cs-tb31-cmpb ColdStandby avail Available Offline seqNum 146282 144723 score 146282 100687
UP006174
Version 1.4
42 of 136

GUI: Verify Upgraded Standby Server status from CMP Manager Topology Form GUI: Verify Alarms
From CMP Manager: Topology <CMP Cluster (Primary)> View Upgraded CMP should be in offline (Force Standby = yes) From CMP Manager: System Wide Reports -> Active Alarms: Below is an example of alarms that may be seen. Note: it is recommended to sort this view by Severity, to see the most important alarms at the top of the form.
18.
19.
GUI: Verify System Admin Reports
Upgraded CMP shows No Data
20.
GUI: Verify System Wide Reports KPI Dashboard
System Wide Reports KPI Dashboard
Verify that report shows all normal traffic processing for the MPEs/MRAs. If any of the Verifications above fail, then Roll Back the Upgrade. Refer to procedure for Roll back of a Partial Upgrade cluster
21.
IF UPGRADE Failure ROLL BACK
UP006174
Version 1.4
43 of 136

Proceed to next Procedure
If Verifications are successful, then one-half of the CMP Primary cluster is now upgraded but the 8.0 server not available for service (Forced Standby). Proceed to the next Procedure to complete the CMP Cluster Upgrade. Procedure is completed
5.1.1 Make Upgraded Server Active

In this step, the upgraded server will be made to the Active server. The other server in the cluster will be made Forced Standby till it can be upgraded. IMPORTANT: This step should not be service affecting, but it is recommended to perform this in a Maintenance Window as a precaution.
Procedure 9: Make Upgraded Server Active Pre-requisites: Procedure 6 was completed which copies the newest policyUpgrade.pl script onto the Primary Active CMP server
Step Procedure GUI: Verify Status of Cluster to be Upgraded Result From CMP Manager: Topology Setting View Primary CMP Cluster One Primary Site CMP server will be Active, and the other (Force) Standby
1.
2.
GUI: View KPI Dashboard, and make a snapshot
From CMP Manager: System Wide Reports KPI Dashboard Confirm current status is OK. Take a screen shot.
3.
SSH: Login to the Primary Active CMP Server
Ssh to the current Primary Active CMP Server ____________
4.
SSH: Primary Active CMP - Verify Server is Active
# ha.stat
node role cs-tb31-cmpa ProvideSvc cs-tb31-cmpb ColdStandby avail Available Offline seqNum 146282 144723 score 146282 100687
Cntl-c 5. SSH: Primary Active CMP - Invoke failover of CMP cluster Note: this step will cause the 8.0 CMP to become Active, and the 7.5.x server to become Force Standby # policyUpgrade.pl --failover <CMP_Hostname>
UP006174
Version 1.4
44 of 136
<CMP_Hostname> is current Active CMP hostname Note: any ssh sessions to the current CMP VIP address will close when the switchover occurs, and will need to be re-opened. 6.
SSH: Login to the New Primary Active CMP Server Ssh to the New (8.0) Primary Active CMP Server ____________
7.
SSH: Verify New Primary Active CMP Server is Active, and all resources are Active
GUI: Verify access to 8.0 CMP Manager GUI
# ha.mystate
resourceId DbReplication VIP QP DbReplication_old role Active Active Active Active node A2635.240 A2635.240 A2635.240 A2635.240 subResou 0 0 0 0 lastUpdate 0612:224121.532 0612:224120.872 0612:224418.295 0612:224120.815
8.
Close CMP GUI Browser window, and re-open Access CMP Manager GUI using the VIP address Policy 8.0 Manager login form should be visible. Login credentials are the same as pre-upgrade.
9.
GUI: View KPI Dashboard, and make a snapshot. Compare to prior KPI Dashboard. Verify that service is normal.
From CMP Manager: Topology Setting <Cluster> View
10.
GUI: Verify Active Alarms
Open the Active Alarms view. Wait a few minutes for alarms to clear. Certain Alarms are expected: Specifically, the following Critical Alarms are expected from the Active CMP: 31228 HA Standby Offline 70025 MySQL slave schema mismatch These will clear as the other CMPs are upgraded.
UP006174
Version 1.4
45 of 136
Note: On the Policy 8.0 GUI, there is on-line help to get additional alarm information. Click on the alarm ID in the Active Alarm view to get the Alarm details.
UP006174
Version 1.4
46 of 136

GUI: Verify Individual MRA/MPE Reports (as desired)
Open and review Reports for the MRAs/MPEs. The reports should show traffic behavior comparable to the pre-upgrade. Note: Policy 8.0 Reports are organized differently. The user may need to click on a details link to see the full report.
UP006174
Version 1.4
47 of 136

GUI: Verify System Administration Reports Report status will show with 8.0 CMP Active
13.
GUI: Verify Platform Setting Topology
Platform Setting Topology Setting All Clusters
View CMP (P)
14.
GUI: Verify System Administration Reports
Upgraded CMP will show Active, Cluster will show degraded. Force Standby CMP (old release) will show on-line.
UP006174
Version 1.4
48 of 136
15.
If Verify Steps Fail
If Verify steps fail, do not proceed. Consult with Tekelec Support. If needed, see Rollback Procedure for Partial Upgraded Cluster.
16.
SSH: Key Exchange from Upgraded CMP server to MPE/MRAs
# policySSHKey.pl --command syncSSHKeys

Sync SSH Key with All C level Nodes: Begin to sync SSH key with node:C1975.230 Begin to sync SSH key with node:C1975.137 --------------------------------NodeID IP Result C1975.230 10.240.241.19 exchanged key successfully C1975.137 10.240.241.18 exchanged key successfully
Verify that all key exchanges are successful. Reexecute if needed.

NOTE: this tool expects the MPE/MRAs use the standard root password. It will fail, if this is not true. As an alternative, the keyexchance command can be used for each MPE/MRA: # keyexchange <mpe/mra hostname>
17.
Proceed to next Procedure
One-half of the Primary CMP cluster is now upgraded and Active. The prior-release CMP server is in (Forced Standby). DO NOT REMOVE FORCE STANDBY CONDITION on 7.5 CMP server! Proceed to the next Procedure to complete the Cluster Upgrade.
5.1.2 Upgrade Second CMP at Primary Site

In this step, the second server of the CMP Site 1 cluster will be upgraded, and the cluster returned to Active/Standby normal condition.
Procedure 10: Upgrade Second CMP Server and Primary Site, and restore cluster
UP006174
Version 1.4
49 of 136

GUI: Verify Status of Cluster to be Upgraded Platform Setting Topology Setting All Clusters
View CMP (P)
2.
SSH: Login to Primary CMP Force Standby server Either: 1) SSH - Access the login prompt. 2) Log into the server as the root user on the iLO or RMM, and access Remote Console SSH/Console: Verify that the server is the correct CMP server for Upgrade
3.
# getPolicyRev 7.5.x # ha.stat node role CSLAB-CMP1-A ColdStandby CSLAB-CMP1-B ColdStandby avail Offline Offline seqNum 4573326 4574922 score 110010 80672
4.
SSH/Console: Verify qp_procmgr status is running.
# service qp_procmgr status
Note: Do not continue if qp_procmgr is not running. UP006174 Version 1.4 50 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Contact Policy TAC for Assistance.
5.
SSH/Console: Verify comcol database is running
# prod.state
Note: Current state B means the system is running and synced. Note: If the Current state does not have a state of B. Do not continue if qp_procmgr is not running. Contact Policy TAC for Assistance. # ra.stat
sev ht timeStamp event instance *C 1 09:50:05.796 QP Slave database is a down slave has a different schema version than the master. errInfo The MySQL
6.
SSH/Console: Verify that the system doesnt have any unexpected qp_procmgr or MySQL alarms
Ctrl + c --- to exit command Note: Do not continue if the system has any unexpected qp_procmgr or MySQL alarms. Contact Policy TAC for Assistance. If using ssh, execute screen to prevent hang-ups, and do not exit this screen session till the server reboots. # screen # su - platcfg Maintenance Upgrade Initiate Upgrade
7.
SSH/Console: Apply Upgrade Note: the Application is NOT shutdown prior to the upgrade.
This step will take about 20 Minutes, and the server will boot.
This step will take about 20 Minutes, and the server will boot. 8.
SSH/Console: Login again to upgraded server. Verify that server returns to the login prompt after boot. SSH: Verify software versions
9.
# getPlatRev 5.0.1-72.45.0 # getPolicyRev
UP006174
Version 1.4
51 of 136
Policy 7.5 to 8.0 - Upgrade Procedure 8.0.0_x.x.x 10.

SSH: Verify success of Upgrade
# tail /var/TKLC/log/upgrade/upgrade.log The following indicates SUCCESS of Upgrade.
11.
SSH: Verify that the server processes are running
Verify that all server processes are Stby (Forced Standby)
# ha.mystate
resourceId DbReplication VIP QP DbReplication_old role Stby Stby Stby Stby node A2635.240 A2635.240 A2635.240 A2635.240 subResou 0 0 0 0 lastUpdate 0612:224121.532 0612:224120.872 0612:224418.295 0612:224120.815
Important: It can take several minutes after the server boot for all server processes to start up. If needed, run the command several times till there is the correct result. # while :; do date; ha.mystate; echo; sleep 5; done WAIT till all processes are Stby. 12.
SSH: Verify Replication status
# inetstat <should show Active or Standby>

Expected information for the Manager Reports
13.
GUI: System Administration Reports
14.
IF Verify steps fail
If Verify steps fail, do not proceed. Consult with Tekelec Support. If needed, see Backout Procedure for a fully upgraded cluster.
15.
SSH: Key Exchange from Upgraded Standby
# policySSHKey.pl --command syncSSHKeys Version 1.4 52 of 136
UP006174

CMP server to MPE/MRAs
Sync SSH Key with All C level Nodes: Begin to sync SSH key with node:C1975.230 Begin to sync SSH key with node:C1975.137 --------------------------------NodeID IP Result C1975.230 10.240.241.19 exchanged key successfully C1975.137 10.240.241.18 exchanged key successfully
Verify that all key exchanges are successful. Re-execute if needed. 16.
GUI: Remove Forced Standby Topology Setting <Cluster> View Modify (Primary Site CMP) Remove Forced Standby check mark, and Save.
17.
GUI: Verify Active/Standby Cluster
Topology Setting <CMP Cluster> View CMP Servers will have status of Active and Standby.
18.
SSH: Verify that the standby server is Stby and Active Server is Active.
# ha.mystate
# ha.states
resourceId DbReplication DbReplication VIP VIP QP QP DbReplication_old DbReplication_old role Stby Active Stby Active Stby Active Active Stby node A2635.240 A2635.228 A2635.240 A2635.228 A2635.240 A2635.228 A2635.228 A2635.240 subResou 0 0 0 0 0 0 0 0 lastUpdate 0612:224121.532 0612:224121.247 0612:224120.872 0612:224121.355 0612:224418.295 0612:224121.294 0612:224121.245 0612:224120.815
Note: the assigned node Ids for the two servers will depend on the installation. These Ids are internal to the software. UP006174 Version 1.4 53 of 136

GUI: Verify access to Upgrade Manager System Maintenance View
Open Upgrade Manager System Maintenance view Note status
20.
GUI: Upgrade Manager - Get accurate status from all servers by Push of Upgrade status scripts Repeat for all servers till the Upgrade Status column is completed.
Open the Upgrade Manager System Maintenance view. Wait for it to fully populate. The 7.5.x servers will typically show an Upgrade Status of unknown. Select a 7.5.x server in the Standby state, using the selection checkbox, and select Operation button. It will display Loading, and after a couple of seconds, will display the allowed operations:
Note: The Operations pick list is specific to the current state of the selected server. Select the Push Script action. select Operation -> Push Script
UP006174
Version 1.4
54 of 136
Close Sucess dialog with the x in the upper right corner. Wait a few seconds for the command to complete and the results to show in the form.
Repeat the Push Script action for each server.

When done, Verify that the System Maintenance view shows the status of all the deployed servers, at all sites. It should show Completed: upgrade <from the previous install or upgrade of the server>.
Note: It is expected that the Legacy Replication will show Off for 7.5.x servers, but On for 8.0 servers. 21.
Procedure is complete.
CMP Active Site Cluster Upgrade is complete. If the Operator has Secondary-CMP site, it may be upgraded in the same maintenance window. CAUTION: it is not supported to Demote/Promote from a Primary CMP cluster (8.0) to a Secondary CMP cluster (7.5.x). CAUTION: No Configuration or Topology changes are supported from 8.0 CMPs to 7.5.x MPE/MRA clusters. The GUI does not prevent this, but the actions may not work as expected.
THIS PROCEDURE HAS BEEN COMPLETED
UP006174
Version 1.4
55 of 136
5.2 Upgrade Secondary Site CMP Cluster (if deployed by Operator)

If the Operator deployment includes a CMP Secondary Site, this procedure must be executed. If not, this procedure can be skipped. It is possible to upgrade the Secondary Site CMPs in the same maintenance window as the Active Site CMPs, or in a later maintenance window. However, the Secondary-site CMPs should be upgraded before any of the MRAs and MPEs. For this procedure, CMP Active site (Primary) cluster is already upgraded to 8.0. The CMP Secondary-site servers will be reporting Critical alarms that they are not able to sync with Active site due to version mismatch. This procedure will use the Policy 8.0 Upgrade Manager feature. This procedure should be performed in a maintenance window.
Procedure 11: Upgrade Secondary-Site CMPs

Step
1.
Procedure GUI: Perform Health checks
Result
From the CMP Manager GUI, review the health of the Policy System.
View Active Alarms View KPI forms Reset Counters on network elements to provide a baseline for the upgrade.
If there are issues in the Policy System, consider if it is wise to proceed. 2.

SSH: Open window to Active CMP server (VIP) Login to the server as the root user
login: root Password: <enter password> Confirm that this is active server: # ha.mystate
This session will be used for executing a switchover command later in this procedure. Keep this window open.
UP006174
Version 1.4
56 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 11: Upgrade Secondary-Site CMPs
Step
3.
Procedure st GUI: Confirm Status for 1 Secondary-CMP
Result
Upgrade Manager System Maintenance -> WAIT for the form to fully populate. This may take a few seconds.
Review Software Release status Review Upgrade status
Primary site CMPs should be on 8.0. Secondary site CMPs should be on 7.5.x. IF NEEDED: if the Upgrade Status shows an error, it may be needed to execute the Push Script Action, as follows: Select checkbox for a CMP and select Operation > Push Script
Close sucess dialog using x to return to window. Review Software Release status Review Upgrade status
4.
GUI: Push Script 2 Secondary-CMP
nd
Upgrade Manager System Maintenance Repeat above operation for second CMP at Secondarysite
UP006174
Version 1.4
57 of 136
Step
5.
Procedure GUI: Verify status of selected CMPs
Result
Upgrade Manager System Maintenance At the top of the form, select Application Filter: either Site1 CMP Cluster, or Site2 CMP Cluster. The form will now display only CMPs at the site to be upgraded. (Note the Secondary site may be either Site1 or Site2) Example:
6.
GUI: Force Standby on standby Secondary-CMP
Upgrade Manager System Maintenance Select the check box for the Standby CMP server at the site and select Operation: Force Standby (It takes 15 seconds for the Operation pick list to load.) There will be the following dialogs:
Close sucess dialog using x to return to window. Confirm that the server state is changed to Force Standby in the form (may take several seconds).
This step will prevent the server from becoming Active. The current Active CMP is un-affected. Alarms are expected.
UP006174
Version 1.4
58 of 136
Step
7.
Procedure Option 1: GUI: Upgrade Manager - Upgrade Force Standby SecondaryCMP NOTE: This step requires that the 8.0 CMP Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
Result
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to upgrade GUI: Upgrade Manager System Maintenance Select the checkbox for the Force Standby cmp server, and Select Operation: Start Upgrade
Confirm the operation, and verify that the Upgrade request was executed successfully.
UP006174
Version 1.4
59 of 136
Step Procedure Wait for Upgrade to process Result
Wait for Upgrade to proceed. Monitor Upgrade Progress, if desired:
This step will take 20 minute or more, and the server will boot during this time
1) Follow status on the Upgrade Manager System

Maintenance form: Upgrade Status
The Upgrade status will proceed through several status messages. 2) Optional: ssh to server, run # tail f /var/TKLC/log/upgrade/upgrade.log
NOTE: the following error messages are seen when the server is re-booting:
After the server re-boots, Confirm that status on the GUI form is Completed.
UP006174
Version 1.4
60 of 136
Step
8.
Procedure Option 2: SSH/iLo/Console: Upgrade Force Standby Secondary-CMP NOTE: This step requires that the 8.0 CMP Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
Result
Option 2 Execute Upgrade from iLo/console/ssh for blade Login to Target Server as root: # getPolicyRev 7.x # ha.stat Cold Standby If using ssh, execute screen to prevent hang-ups, and do not exit this screen session till the server reboots. # screen # su - platcfg Maintenance Upgrade Initiate Upgrade
SSH : Upgraded server Verify Upgrade completed successfully Re-Login to blade after re-boot and verify current rev:
# getPlatRev # getPolicyRev 8.0 View Upgrade log from the server: # tail /var/TKLC/log/upgrade.log Look for SUCCESS
10.
SSH: Upgraded server Verify that the server processes are running
# ha.mystate
Note: It takes several minutes after the server boot for all server processes to start up. If needed, run the command several times till there is the correct result. UP006174 Version 1.4 61 of 136
# while :; do date; ha.mystate; echo; sleep 5; done WAIT till all processes are Stby. 11. 12.
SSH: Upgraded server Verify replication IF Verify steps have failed, or the Upgrade has gone more than 25 Minutes.
NOTE: If the upgrade fails, the upgrade software will typically roll back automatically to the prior release and configuration.
# inetstat If the Upgrade does not Complete sucessfully.

Do not proceed.
View/collect Upgrade log from the server, if possible: # tail /var/TKLC/log/upgrade.log

Consult with Tekelec Support.
13.
SSH: Active CMP at Primary Site -- Cause Switchover to Upgraded CMP
IMPORTANT: This step is executed from the Active CMP at the Primary site! This will cause a switchover of the Secondary Site CMP cluster, making the 8.0 server Active, and the 7.5.x server to become Force Standby # policyUpgrade.pl --failover <CMP_Hostname> <CMP_Hostname> is current Active Secondary Site CMP hostname
14.
GUI: Upgrade Manager Verify switchover
Upgrade Manager System Maintenance After a few seconds (perhaps as many as 15 seconds), the System Maintenance form will update to show that the Active/Force Standby roles have changed. The upgraded 8.0 CMP is now Active, and the 7.5.x CMP is Force Standby.
15.
GUI: Verify Alarms
View alarms and confirm status. DB Replication Alarms may take a few minutes to clear after the switchover.
UP006174
Version 1.4
62 of 136
Step
16.
Procedure Option 1: GUI: Upgrade Manager - Upgrade second Secondary-CMP in Cluster
Result
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to upgrade GUI: Upgrade Manager System Maintenance Upgrade Manager System Maintenance Select the checkbox for the current Force Standby CMP server at the site and select Operation: Start Upgrade Monitor Upgrade Progress
This step will take 20 minute or more, and the server will boot during this time
Wait for Upgrade to process
Wait for Upgrade to proceed (up to 25 minutes). Monitor Upgrade Progress, if desired.
1) Follow status on the GUI: Upgrade Manager

System Maintenance: Upgrade Status
UP006174
Version 1.4
63 of 136
Step
17.
Procedure Option 2: SSH/iLo/Console: Upgrade Force Standby Secondary-CMP NOTE: This step requires that the 8.0 CMP Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
Result
SSH : Upgraded server Verify Upgrade completed successfully Re-Login to blade after re-boot and verify current rev:
# getPlatRev # getPolicyRev 8.0 View Upgrade log from the server: # tail /var/TKLC/log/upgrade.log Look for SUCCESS
19.
SSH: Upgraded server Verify that the server processes are running
# ha.mystate
Note: It takes several minutes after the server boot for all server processes to start up. If needed, run the command several times till there is the correct result. UP006174 Version 1.4 64 of 136
# while :; do date; ha.mystate; echo; sleep 5; done WAIT till all processes are Stby. 20.
IF Upgrade does not show Completed, or the Upgrade has gone more than 25 Minutes, or the verify step above fails.
NOTE: If the upgrade fails, the upgrade software will typically roll back automatically to the prior release and configuration.
If the Upgrade does not Complete sucessfully.

Do not proceed. Consult with Tekelec Support.
21.
CMP: Upgrade Manager Remove Forced Standby
Upgrade Manager System Maintenance Select check box for Force Standby CMP server (just upgraded) and select Operation: Release Force Standby Follow the dialogs. Confirm that status on the form is updated from Force Standby to Standby after a few seconds.
22.
CMP: Verify Alarm Status for Upgraded Cluster
SystemWideReports Active Alarms Verify that Alarms for the upgrade cluster all clear.
23.
SSH: Active CMP at Secondary Site verify process status
# ha.mystate
# ha.states
resourceId DbReplication DbReplication VIP VIP QP QP DbReplication_old DbReplication_old role Stby Active Stby Active Stby Active Active Stby node A2635.240 A2635.228 A2635.240 A2635.228 A2635.240 A2635.228 A2635.228 A2635.240 subResou 0 0 0 0 0 0 0 0 lastUpdate 0612:224121.532 0612:224121.247 0612:224120.872 0612:224121.355 0612:224418.295 0612:224121.294 0612:224121.245 0612:224120.815
24.
CMP: IF problems, rollback
If the Upgraded cluster is not in a normal condition, consult with Tekelec Support. Rollback, if needed.
UP006174
Version 1.4
65 of 136
25.
No Active Alarms are expected
UP006174
Version 1.4
66 of 136
6. UPGRADE SITE _____________________

The following procedures will upgrade a site containing one or more MPE clusters, and (optional) MRA cluster.
6.1 Site Upgrade Preparations 6.1.1 Configuration Preparations

Procedure 12: Configuration Preparations Procedure
Step Procedure GUI: Open CMP GUI GUI: Verify Upgrade Manager status display Result
1. 2.
Login to CMP GUI as Administrator (or as Upgrade Engineer, if an account is defined for this). Upgrade Manager System Maintenance Open form wait a few seconds for the Status of the servers in the managed network to be displayed.
Verify that status is shown for all servers. Verify that the CMP clusters are upgraded to release 8.0. See example below.
If Upgrade status is not shown, it may be necessary to execute the operation to Push Script to the servers. To do this: Select each server at the site (one at a time) using the checkbox, and select the Operation: Push Script. Confirm that status information on the form (including Upgrade Status) is updated after a few seconds. This step is not service affecting. It must be done before the Upgrade action is applied.
UP006174
Version 1.4
67 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 12: Configuration Preparations Procedure
Step
3.
Procedure GUI: Configure Network Element Capability (if needed)
Result
GUI: Network Network Elements GGSN For compatibility of Policy 8.0 with ggsn systems that use UsageReport-26, select this option and save.
6.1.2 Key Exchanges from CMPs

Procedure 13: Key Exchanges from CMPs to MPE/MRA
Step
4. 5.
Procedure GUI: Open CMP GUI SSH: Primary Active CMP Verify Key exchanges to MPE/MRA servers
Result
Login to CMP GUI as Administrator (or as Upgrade Engineer, if an account is defined for this). Ssh to Primary Active CMP Verify key exchanges from CMP to the MPE/MRA servers are completed: # policySSHKey.pl --command checkSSHKeys Check output to confirm that key exchanges are completed.
UP006174
Version 1.4
68 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 13: Key Exchanges from CMPs to MPE/MRA
Step
6.
Procedure SSH: Primary Active CMP IF keyExchange need to be updated:
Result
IF the check of Key exchanges (previous step) shows that certain exchanges are not completed, then exexecute Key Exchange tool: # policySSHKey.pl --command syncSSHKeys Example output: Sync SSH Key with All C level Nodes: Begin to sync SSH key with node:C1180.027 Begin to sync SSH key with node:C0682.103 Begin to sync SSH key with node:C3474.104 Begin to sync SSH key with node:C0682.146 Begin to sync SSH key with node:C1180.101 Begin to sync SSH key with node:C3474.070 --------------------------------NodeID IP Result C1180.027 10.240.238.89 exchanged successfully C0682.103 10.240.238.92 exchanged successfully C3474.104 10.240.238.80 exchanged successfully C0682.146 10.240.238.84 exchanged successfully C1180.101 10.240.238.81 exchanged successfully C3474.070 10.240.238.88 exchanged successfully [root@cs-tb31-cmpb ~]#
key key key key key key
If any key exchanges fail, run this command again. 7.

SSH: Primary Standby CMP Verify Key exchange
Ssh to Primary Standby CMP Execute this tool to verify key exchanges from CMP to the MPE/MRA servers: # policySSHKey.pl --command checkSSHKeys If any key Exchanges are incomplete: # policySSHKey.pl --command syncSSHKeys
8.
SSH: Secondary Active CMP Verify Key exchange
Ssh to Secondary Active CMP Execute this tool to verify key exchanges from CMP to the MPE/MRA servers: # policySSHKey.pl --command checkSSHKeys If any key Exchanges are incomplete: # policySSHKey.pl --command syncSSHKeys
UP006174
Version 1.4
69 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 13: Key Exchanges from CMPs to MPE/MRA
Step
9.
Procedure SSH: Secondary Standby CMP Verify Key exchange
Result
Ssh to Secondary Standby CMP Execute this tool to verify key exchanges from CMP to the MPE/MRA servers: # policySSHKey.pl --command checkSSHKeys If any key Exchanges are incomplete: # policySSHKey.pl --command syncSSHKeys
10.
GUI: Verify Upgrade Manager status display
GUI: Upgrade Manager System Maintenance Open form wait a few seconds for the Status of the servers in the managed network to be displayed.
Verify that status is shown for all servers. Verify that the CMP clusters are upgraded to release 8.0
If Upgrade status is not shown, it may be necessary to execute the operation to Push Script to the servers. To do this: Select each server at the site (one at a time) using the checkbox, and select the Operation: Push Script. Confirm that status information on the form (including Upgrade Status) is updated after a few seconds. This step is not service affecting. It must be done before the Upgrade action is applied.
6.1.3 Key Exchanges between servers of MPE/MRA cluster at a site

This procedure will execute Key Exchanges between servers of MPE/MRA clusters, at the site to be upgraded. Policy 8.0 requires that a key exchange is performed between MPE/MRA servers in a cluster. It will need to be performed for every MPE/MRA cluster.
Procedure 14: Key exchanges between servers of MPE/MRA clusters

Step
1.
Procedure GUI: Open CMP GUI
Result
Login to CMP GUI as Administrator (or as Upgrade Engineer, if an account is defined for this).
UP006174
Version 1.4
70 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 14: Key exchanges between servers of MPE/MRA clusters
Step
2. 3. 4.
Procedure SSH: any CMP server
Result
# cat /etc/hosts | grep <mpe|mra> # ssh <hostname_of_MPE/MRA server> At MPE/MRA: # su platcfg Camiant Configuration Exchange ssh Key with Mate
SSH: from CMP server, ssh to a MPE/MRA server SSH: MPE/MRA perform Key Exchange using platcfg
5.
SSH: MPE/MRA Key Exchange dialog
The Mate IP address will be pre-populated. Select OK
There are two successful results:
A Success Dialog (if key is exchanged) A return to the platcfg menu with no dialog (if key was previously exchanged, and does not need to be exchanged)
UP006174
Version 1.4
71 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 14: Key exchanges between servers of MPE/MRA clusters
Step
6.
Procedure Repeat for each cluster
Result
The key exchange is performed on one of the two servers of the cluster. Repeat this procedure for each MPE/MRA cluster at the site. THIS PROCEDURE HAS BEEN COMPLETED
6.1.4 Verify Deployed software images at site MPE/MRA server

Detailed steps are shown in the procedure below to verify that the image files are correctly deployed and ready for upgrade activity at the MPE/MRA servers. [The software iso files were previously deployed to these servers during upgrade preparation.]
Procedure 15: Verify deployed software image

Step Procedure SSH: Active CMP Log into the server as the root user login: root Password: <root_password> Result
1.
2.
SSH: Active CMP - Verify Image is deployed at MPE/MRA

Rel 8.0 Application Part Numbers: cmp 872-2472-101 mpe 872-2473-101 mpe-li 872-2474-101 mra 872-2475-101
# cat /etc/hostname | grep <mpe/mra> # ssh <MPE/MRA hostname> # getPolicyRev 7.5.x_x.x.x # getPolicyRev p mpe or mra # ls -l /var/TKLC/upgrade total 706236 -rw-r--r-- 1 root root 863408128 Jul 3 03:04 mpe-8.0.0_18.2.0--872-2473-101--x86_64.iso Verify that the iso matches the correct part number for this server function (mra, mpe, mpeli), and Verify there is only one iso in this directory.
UP006174
Version 1.4
72 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 15: Verify deployed software image
Step Procedure SSH: MPE/MRA Validate iso image Result
3.
This step will validate the iso image at the mpe/mra server: # su platcfg Maintenance -> Upgrade -> Validate Note Success of Validation exit from platcfg # exit
4.
Repeat steps 2 and 3 for each MPE and MRA server in the site to be upgraded.
List of MPE _________________ List of MRA _________________
UP006174
Version 1.4
73 of 136
6.2 Upgrade MPEs - Site ________________

This procedure will upgrade one or more MPE clusters at a site. This can be performed before or after MRA upgrade at the site. This section can be replicated for each site to be upgraded, to allow the Upgrade engineer to add cluster and site specific information. Notes: CMPs must be upgraded before executing this procedure. Application software is previously deployed to the upgrade directory on the servers at the site (see pre-upgrade procedure) This procedure will use the Upgrade Manager functionality on the CMP GUI to perform the upgrade of the MPEs.
Procedure 16: Upgrade MPEs Site

Step Procedure Health Checks Result
1.
GUI: - check Active Alarms GUI: - (optional) reset MPE counters to make a baseline GUI: - check KPI Dashboard (take a snap shot) GUI: - verify current call rates (pre-upgrade) to compare after upgrade login: root Password: <enter password> This session will be used for executing a switchover command later in this procedure. Keep this window open.
2.
SSH: Open ssh session to Primary Active CMP server
3.
GUI: Display Upgrade status of selected site __________
Upgrade Manager System Maintenance Option: Filter this form to display only MPEs. Verify that the current Release numbers are the expected values.
4.
GUI: Force Standby on standby MPE(s)
This Activity can be applied at more than one MPEs at a site, in parallel, to reduce time requirements. Upgrade Manager System Maintenance Select the checkbox for a Standby MPE server at the site and select Operation: Force Standby. Confirm that status on the form is updated after several seconds. This step will prevent the server from becoming Active. Active MPEs are un-affected. Alarm 31228 is expected for each cluster to be upgraded.
UP006174
Version 1.4
74 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 16: Upgrade MPEs Site
Step
5.
Procedure Option 1: GUI: Upgrade Manager - Start Upgrade for Force Standby MPE(s)
Result
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to upgrade GUI: Upgrade Manager System Maintenance This Activity can be applied to multiple MPEs at a site, in parallel, to reduce time requirements. Upgrade Manager System Maintenance Select Force Standby MPE server(s) at the site and select Operation Start Upgrade
This step will take 20 minute or more, and the server will boot during this time.
GUI: Monitor Upgrade process
Monitor Upgrade Progress.
1) Follow status on the Upgrade Manager System Maintenance form: Upgrade Status
The following Alarms are expected:

GN_DOWN/WRN monitorRecentAlarms(): no mate heartbeat ^^ [6303:hamonitor.cxx:624] 31229 GN_DOWN/WRN HA cannot Connect to Remote Server 70005 The peer server 10.240.238.81 is degraded. 32305 Platform detected an error condition 311xx <Minor Replication Alarms > 31228
UP006174
Version 1.4
75 of 136
Step
6.
Procedure Option 2: SSH/iLo/Console: Upgrade Force Standby MPE(s) NOTE: This step requires that the 8.0 Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
Result
UP006174
Version 1.4
76 of 136
Step
7.
Procedure SSH: upgraded MPE server(s), Verify upgrade success
Result
After server boot is completed, ssh to upgraded server: # getPolicyRev 8.0.0_x.x.x # tail /var/TKLC/log/upgrade/upgrade.log
1343413625:: UPGRADE IS COMPLETE 1343413625:: 1343413625:: Waiting for reboot 1343413625::DEBUG: ADDING VAR: UPGRADE_STATUS = SUCCESS 1343413625::DEBUG: ADDING VAR: UPGRADE_COMPLETED = 07/27/2012 18:27:05 UTC 1343413625:: Updating platform revision file... 1343413625:: 1343413625:: 1343413625:: A reboot of the server is required. 1343413625:: The server will be rebooted in 10 seconds
# ha.mystate
resourceId DbReplication VIP QP DbReplication_old role Stby Stby Stby Stby node C3691.123 C3691.123 C3691.123 C3691.123 lastUpdate 0727:143326.003 0727:143326.037 0727:143329.774 0727:143326.104
NOTE: the state for some services may be OOS for a couple of minutes after upgrade. DO NOT PROCEED till status shows Stby for all services.
UP006174
Version 1.4
77 of 136
Step
8.
Procedure SSH: Verify Replication of session data
Result
Once upgraded, the server will get the session data from the Active MPE server via audit. Run this command to monitor completion of this data transfer. It may take several minutes, if there is lot of session data. # inetstat Audit state
Audit Completed State
DO NOT PROCEED till status shows that Audit is complete. 9.

SSH: Primary Active CMP
SSH to Primary Active CMP, and Confirm that this is the Active CMP Server # ha.mystate
resourceId DbReplication VIP QP DbReplication_old role Active Active Active Active node A2225.046 A2225.046 A2225.046 A2225.046 lastUpdate 0825:215231.942 0825:215231.965 0825:215231.949 0825:215231.989
10.
SSH: Primary Active CMP - Switchover cluster to Upgraded MPE(s) Service Impact
Note: this step will cause the 8.0 MPE to become Active, and the 7.5.x server to become Force Standby. # policyUpgrade.pl --failover <MPE_Hostname> <MPE_Hostname> is current Active MPE hostname for the target cluster
UP006174
Version 1.4
78 of 136
Step
11.
Procedure GUI: Verify MPE Cluster Configuration
Result
PolicyServer Configuration <cluster> System Tab Verify that Status is on-Line or Degraded
If Configuration mis-match is seen, select Reapply Configuration. 12.

GUI: Verify MPE Cluster Traffic
PolicyServer Configuration <cluster> Reports Tab Verify that Upgraded server is Active and other server is Forced Standby. Verify that the Reports show server is processing traffic.
13.
GUI: Verify KPI Dashboards
SystemWideReports KPI Dashboard Compare to pre-upgrade KPI Dashboard. If possible, confirm with customer that traffic is normal for Network element connected devices.
14.
IF Verify Step fails Backout
If needed, fallback to 7.5 server: From Primary Active CMP: # policyUpgrade.pl --failover <MPE_Hostname> See section for Backout of Partial Upgraded Cluster.
UP006174
Version 1.4
79 of 136
Step
15.
Procedure Option 1: GUI: Upgrade Manager - Upgrade second MPE in Cluster This will take 20 Minutes.
Result
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to upgrade GUI: Upgrade Manager System Maintenance Select check box for current Force Standby MPE server(s) at the site and select Operation Start Upgrade
1) Follow status on the GUI: Upgrade Manager

System Maintenance
3) Optional: ssh to server, run # tail f /var/TKLC/log/upgrade/upgrade.log

Confirm that status on the GUI form is Upgrade Complete. 16.
Option 2: SSH/iLo/Console: Upgrade Force Standby MPE(s) NOTE: This step requires that the 8.0 Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
UP006174
Version 1.4
80 of 136
Step
17.
Result
After server boots, ssh to the upgraded server: # getPolicyRev 8.0.0_x.x.x # tail /var/TKLC/log/upgrade/upgrade.log
# ha.mystate
NOTE: the state for some services may be OOS for a couple of minutes after upgrade. DO NOT PROCEED till status shows Stby for all services. 18.
SSH: Verify Replication
Once upgraded, the server will get the session data from the Active MPE server via audit. Run this command to monitor completion of this data transfer. It may take several minutes, if there is lot of session data. # inetstat DO NOT PROCEED till status shows Audit is complete.
19.
GUI: Remove Forced Standby
Upgrade Manager System Maintenance Select Standby MRE server at the site and select the Operation Cancel Force Standby. Confirm that status on the form is updated to Standby. This step will allow the server to become Active. Active MPE is un-affected. Alarms are expected.
UP006174
Version 1.4
81 of 136
Step Procedure GUI: Verify Alarms and Reports Result
20.
SystemWideReports Active Alarms Confirm if any alarms are unexpected. Note: Some Alarms have a 30 minute auto clearing time. SystemWideReports KPI DashBoard Compare to pre-upgrade collected reports. Policy Server -> Configuration Policy Server: Reports Tab Compare to pre-upgrade collected reports. Policy Server -> Configuration Policy Server: System Tab Confirm status
21.
Optional: Active Server Restart
Optional: From the PolicyServer -> Reports form, select Active server Restart action. This additional switchover can resolve certain conditions where data or connections are not being reported correctly.
22. 23. 24.
IF needed: Backout
If Needed: See Backout Procedure

MPE cluster is upgraded
MPE cluster is upgraded.

REPEAT Above steps for next MPE cluster
If Clusters are being upgraded one-at-a-time, then procede with next cluster: MPE Cluster _______________
Add rows as needed for all MPEs at a site.
25.
Recommended Soak Period
It is Recommended to let the new release soak for a period of time, to view stability and traffic/policy behavior is as expected. THIS PROCEDURE HAS BEEN COMPLETED
UP006174
Version 1.4
82 of 136
6.3 Upgrade Site MRA Cluster Site ____________

This procedure will upgrade an MRA cluster at a site. It can be applied before or after the Upgrade of the MPEs at a site. [This section may be replicated or moved to adjust for the customer choice of upgrade order of MPEs and MRAs.] Notes: CMPs must be upgraded before executing this procedure. Application software is previously deployed to the upgrade directory on the servers at the site (see pre-upgrade procedure) This procedure will use the Upgrade Manager functionality on the CMP GUI to perform the upgrade of the MRA cluster.
Procedure 17: Upgrade Site MRA

Step Procedure GUI: Health Checks Result
1.
GUI: - check Active Alarms GUI: - (optional) reset MRA/MPE counters to make a baseline GUI: - check KPI Dashboard (take a snap shot) GUI: - verify current call rates to compare after upgrade login: root Password: <enter password> This session will be used for executing a switchover command later in this procedure. Keep this window open.
2.
SSH: Open ssh session to Active CMP server at Primary site 1) Access the login prompt. 2) Log into the server as the root user
3.
GUI: Display Upgrade status of selected site
Upgrade Manager System Maintenance Wait for the form to populate. Select Filter for Appl Type = MRA, to display only MRAs (option). Verify information for the MRAs:
Current Release installed Upgrade status Active/Standby status
Note: if the Upgrade status is reporting an Error, it may be needed to use the Operation -> Push Script to get the current status from the server.
UP006174
Version 1.4
83 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 17: Upgrade Site MRA
Step
4.
Procedure GUI: Force Standby on standby MRA
Result
Upgrade Manager System Maintenance Select the checkbox for the Standby MRA server at the site to be upgraded, and select the Operation Force Standby. Confirm that status on the form is updated. This step will prevent the server from becoming Active. The Active MRA is un-affected. Active MRA Server will report Alarm 31228.
5.
Option 1: GUI: Upgrade Manager - Upgrade for Force Standby MRA
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to upgrade GUI: Upgrade Manager System Maintenance Select the check box for the Force Standby MRA server at the site and select Operation: Start Upgrade Note: the 8.0 MRA software image (iso) should have previously been copied to the /var/TKLC/upgrade directory on the server, and this should be the only image in this directory.
Monitor Upgrade Progress.
1) Follow status on the Upgrade Manager System Maintenance form: Upgrade Status
The Upgrade status will proceed through several status messages. 2) Optional: ssh to server, run
# tail f /var/TKLC/log/upgrade/upgrade.log NOTE: the following error messages are seen when the server is re-booting:
UP006174
Version 1.4
84 of 136
Step
6.
Procedure Option 2: SSH/iLo/Console: Upgrade Force Standby MRA NOTE: This step requires that the 8.0 Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
Result
UP006174
Version 1.4
85 of 136
Step
7.
Procedure SSH: upgraded MRA server(s), Verify upgrade success
Result
After upgrade shows Completed, ssh session to upgraded server: # getPolicyRev 8.0.0_x.x.x # tail /var/TKLC/log/upgrade/upgrade.log
# ha.mystate
NOTE: the state for some services may be OOS for a couple of minutes after upgrade. DO NOT PROCEED till status shows Stby for all services.
UP006174
Version 1.4
86 of 136
Step
8.
Procedure SSH: Verify Replication of session data
Result
Once upgraded, the server will get the session data from the Active MPE server via audit. Run this command to monitor completion of this data transfer. It may take several minutes, if there is lot of session data. # inetstat Audit state
Audit Completed State
DO NOT PROCEED till status shows that Audit is complete. 9.

SSH: Primary Active CMP
SSH to Primary Active CMP, and Confirm that this is the Active CMP Server # ha.mystate
resourceId DbReplication VIP QP DbReplication_old role Active Active Active Active node A2225.046 A2225.046 A2225.046 A2225.046 lastUpdate 0825:215231.942 0825:215231.965 0825:215231.949 0825:215231.989
10.
SSH: Primary Active CMP Switchover cluster to Upgraded MRA Service Impact
Note: this step will cause the 8.0 MRA to become Active, and the 7.5.x server to become Force Standby. # policyUpgrade.pl --failover <MRA_Hostname> <MRA_Hostname> is current Active MPE hostname for the target cluster
UP006174
Version 1.4
87 of 136
Step
11.
Procedure GUI: Verify MPE Cluster Traffic
Result
MRA Configuration <cluster> Reports Tab Verify that Upgraded server is Active and other server is On-line
Verify that the Reports show server is processing traffic.
UP006174
Version 1.4
88 of 136
Step
12.
Procedure GUI: Verify KPI Dashboards
Result
SystemWideReports KPI Dashboard Compare to pre-upgrade KPI Dashboard. If possible, confirm with customer that traffic is normal for Network element connected devices.
13.
IF Verify Steps fail Backout
If needed, fallback to 7.5 server: From Primary Active CMP: # policyUpgrade.pl --failover <MRA_Hostname> See section for Backout of Partial Upgraded Cluster.
14.
Option 1: GUI: Upgrade Manager - Upgrade second MRA in Cluster
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to upgrade GUI: Upgrade Manager System Maintenance Select check box for current Force Standby MRA server at the site and select Operation: Start Upgrade
UP006174
Version 1.4
89 of 136
Step Procedure Wait for Upgrade to process Result
1) Follow status on the Upgrade Manager System

Maintenance form: Upgrade Status
The Upgrade status will proceed through several status messages. 2) Optional: ssh to server, run
# tail f /var/TKLC/log/upgrade/upgrade.log NOTE: the following error messages are seen when the server is re-booting:
UP006174
Version 1.4
90 of 136
Step
15.
Procedure Option 2: SSH/iLo/Console: Upgrade Force Standby MRA NOTE: This step requires that the 8.0 Software image (.iso) was previously copied to the server and placed in the /var/TKLC/upgrade directory. This was done in the Upgrade preparations steps.
Result
UP006174
Version 1.4
91 of 136
Step
16.
Result
After upgrade shows Completed, ssh session to upgraded server: # getPolicyRev 8.0.0_x.x.x # tail /var/TKLC/log/upgrade/upgrade.log
# ha.mystate
NOTE: the state for some services may be OOS for a couple of minutes after upgrade. DO NOT PROCEED till status shows Stby for all services. 17.
SSH: Verify Replication
Once upgraded, the server will get the session data from the Active MPE server via audit. Run this command to monitor completion of this data transfer. It may take several minutes, if there is lot of session data. # inetstat DO NOT PROCEED till status shows Audit is complete.
18.
GUI: Remove Forced Standby (second MRA in the cluster)
Upgrade Manager System Maintenance Select check box for just-upgraded Force Standby MRA server at the site. Select the Operation Cancel Force Standby. Confirm that status on the form is updated to Standby.
19.
Optional: Active Server Restart
Optional: From the MRA -> Reports form, select Active server Restart action. This additional switchover can resolve certain conditions where data or connections are not being reported correctly.
UP006174
Version 1.4
92 of 136
Step
20.
Procedure GUI: Verify MRA activity at Upgraded site MRA cluster
Result
Perform health checks as in step 1 of this procedure. View Alarms, KPI Dashboards, and MRA reports to verify that the system is healthy. Recommend a screen capture of post-upgrade status for these forms.
UP006174
Version 1.4
93 of 136
7. ACTIVATE 8.0 REPLICATION FEATURE

For Release 8.0, there is an improved Replication method that needs to be activated after the upgrade. It is recommended that this is done as part of the planned Upgrade activities. After an upgrade from 7.5 to 8.0, the Policy System will be using Legacy Replication. This is the Replication method between servers that was supported in release 7.5. This Replication method will be disabled, and the new Replication method enabled. This Activation should be performed for all servers in the Policy system in a single maintenance window. A roll back procedure is also provided. IMPORTANT: this is only performed after all servers in the network have been upgraded. Notes: All servers in the Policy system are previously upgraded to the 8.x Release This procedure will use the Upgrade Manager functionality on the CMP GUI to perform the replication feature Activation.
Procedure 18: 8.0 Replication Activation

Step
1. 2.
Procedure GUI: Open CMP GUI SSH: Open ssh session to Active CMP server 1) Access the login prompt. 2) Log into the server as the root user
Result
Login to CMP GUI as Administrator (or as Upgrade Engineer, if an account is defined for this). login: root Password: <enter password> This session will be used for executing command line activities. Upgrade Manager System Maintenance Wait 30 seconds for the form to fully update. Review the overall status of all sites and servers.
3.
GUI: review Upgrade and Replication status of all sites and servers
4.
GUI: Verify Replication status
All servers should show: Legacy Replication: On
UP006174
Version 1.4
94 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 18: 8.0 Replication Activation
Step
5.
Procedure GUI: Change Replication mode: MRAs (One cluster at a time)
Result
Upgrade Manager System Maintenance Select MRA cluster _______________________
1) Select the checkbox for the Standby MRA server, and execute this Operation sequence:
Turn Off Replication Turn Off Legacy Replication Turn on Replication
After each step, wait till the status is updated on the form.
2) Select the checkbox for the Active MRA server, and execute this Operation sequence:
After each step, wait till the status is updated on the form. 6.
GUI: Repeat Steps above for each MRA cluster in the Network (One cluster at a time)
Select another MRA cluster _______________________ Perform steps above (Repeat this row of the table for each MRA cluster in the network.)
UP006174
Version 1.4
95 of 136
Step
7.
Procedure GUI: Change Replication mode: MPEs (One cluster at a time)
Result
Upgrade Manager System Maintenance Select MPE cluster _______________________ 1) Select the checkbox for the Standby MPE server, and execute this Operation sequence: Turn Off Replication Turn Off Legacy Replication Turn on Replication
After each step, wait till the status is updated on the form. 2) Select the checkbox for the Active MPE server, and execute this Operation sequence: Turn Off Replication Turn Off Legacy Replication Turn on Replication
GUI: Repeat Steps above for each MPE cluster in the Network (One cluster at a time)
Select MPE cluster _______________________ Perform steps above (Repeat this row of the table for each MPE cluster in the network.)
UP006174
Version 1.4
96 of 136
Step
9.
Procedure GUI: Change Replication mode: Secondary site CMPs
Result
Upgrade Manager System Maintenance Select Secondary-Site CMP cluster _______________________
1) Select Standby Secondary-CMP server, and execute this Operation sequence:

2) Select Active Secondary-CMP server, and execute this Operation sequence:

GUI: Change Replication mode: Active site CMPs
Select CMP cluster _______________________
3) Select Standby CMP server, and execute this Operation sequence:

4) Select Active CMP server, and execute this Operation sequence:

GUI: Active Alarms
UP006174
Version 1.4
97 of 136
Step Procedure GUI: Verify all servers are using new Replication, and synced Result
12.
GUI: Upgrade Manager : System Maintenance
Procedure 19: Remove Replication Exclusions

Step
1.
Procedure SSH: Primary Active CMP Verify Exclusions are still in place
Result
# iqt -p NodeInfo
2.
SSH: Primary Active CMP Remove Replication table exclusions for cluster
Remove Replication exclusions LongParam,AppEventDef,HaCfg for all nodes Version 1.4 98 of 136
UP006174
Policy 7.5 to 8.0 - Upgrade Procedure # ivi NodeInfo SSH: Active CMP ivi NodeInfo
#!/bin/sh iload -ha -xU -fnodeId -fnodeName -fhostName -finhibitFlag fnodeCap \ -fexcludeTables NodeInfo \ <<'!!!!' A0853.107|brbg-cmp-a|brbg-cmp-a,10.250.84.25||MasterCapable| LongParam,AppEventDef,HaCfg A0853.244|brbg-cmp-b|brbg-cmp-b,10.250.84.26||MasterCapable| LongParam,AppEventDef,HaCfg A1408.065|slak-cmp-b|slak-cmpb,10.250.85.26||MasterCapable| A1408.213|slak-cmp-a|slak-cmp-a,10.250.85.25||MasterCapable| LongParam,AppEventDef,HaCfg C0371.030|brbg-mpe-01b|brbg-mpe-01b,10.250.84.8||MasterCapable| LongParam,AppEventDef,HaCfg C0371.252|brbg-mpe-01a|brbg-mpe-01a,10.250.84.7||MasterCapable| LongParam,AppEventDef,HaCfg C1533.011|slak-mpe-01a|slak-mpe-01a,10.250.85.7||MasterCapable| LongParam,AppEventDef,HaCfg C1533.125|slak-mpe-01b|slak-mpe-01b,10.250.85.8||MasterCapable| LongParam,AppEventDef,HaCfg C1751.030|slak-mra-a|slak-mra-a,10.250.85.4||MasterCapable| LongParam,AppEventDef,HaCfg C1751.145|slak-mra-b|slak-mra-b,10.250.85.5||MasterCapable| LongParam,AppEventDef,HaCfg C2080.054|brbg-mra-a|brbg-mra-a,10.250.84.4||MasterCapable| LongParam,AppEventDef,HaCfg C2080.221|brbg-mra-b|brbg-mra-b,10.250.84.5||MasterCapable| LongParam,AppEventDef,HaCfg C2399.016|brbg-mpe-07a|brbg-mpe-07a,10.250.84.28||MasterCapable| LongParam,AppEventDef,HaCfg C2399.048|brbg-mpe-07b|brbg-mpe-07b,10.250.84.29||MasterCapable| LongParam,AppEventDef,HaCfg C3701.051|slak-mpe-07a|slak-mpe-07a,10.250.85.28||MasterCapable| LongParam,AppEventDef,HaCfg C3701.117|slak-mpe-07b|slak-mpe-07b,10.250.85.29||MasterCapable| LongParam,AppEventDef,HaCfg !!!! #!/bin/sh iload -ha -xU -fnodeId -fnodeName -fhostName -finhibitFlag fnodeCap \ -fexcludeTables NodeInfo \ <<'!!!!' A0853.107|brbg-cmp-a|brbg-cmp-a,10.250.84.25||MasterCapable| A0853.244|brbg-cmp-b|brbg-cmp-b,10.250.84.26||MasterCapable| A1408.065|slak-cmp-b|slak-cmp-b,10.250.85.26||MasterCapable| A1408.213|slak-cmp-a|slak-cmp-a,10.250.85.25||MasterCapable| C0371.030|brbg-mpe-01b|brbg-mpe-01b,10.250.84.8||MasterCapable| C0371.252|brbg-mpe-01a|brbg-mpe-01a,10.250.84.7||MasterCapable| C1533.011|slak-mpe-01a|slak-mpe-01a,10.250.85.7||MasterCapable| C1533.125|slak-mpe-01b|slak-mpe-01b,10.250.85.8||MasterCapable| C1751.030|slak-mra-a|slak-mra-a,10.250.85.4||MasterCapable| C1751.145|slak-mra-b|slak-mra-b,10.250.85.5||MasterCapable| C2080.054|brbg-mra-a|brbg-mra-a,10.250.84.4||MasterCapable| C2080.221|brbg-mra-b|brbg-mra-b,10.250.84.5||MasterCapable| C2399.016|brbg-mpe-07a|brbg-mpe-07a,10.250.84.28||MasterCapable| C2399.048|brbg-mpe-07b|brbg-mpe-07b,10.250.84.29||MasterCapable| C3701.051|slak-mpe-07a|slak-mpe-07a,10.250.85.28||MasterCapable| C3701.117|slak-mpe-07b|slak-mpe-07b,10.250.85.29||MasterCapable| !!!!
SSH: Active CMP ivi NodeInfo Edit to remove Exclusions for all clusters
SSH: ivi NodeInfo Save or Quit the NodeInfo table
IF it was needed to Edit the Table: Save and quit - Exit ivi using the command ZZ or :wq (no quotes) - Answer y to the question: APPLY THE CHANGES [yn]? IF no edit was needed: Quit: - Exit ivi using the command :q (no quotes)
UP006174
Version 1.4
99 of 136

SSH: Primary Active CMP Verify Exclusions are removed from previous step. Verify Health
Software Upgrade Procedure # iqt -p NodeInfo
4.
Verify Alarms
UP006174
Version 1.4
100 of 136
8. POST UPGRADE ACTIVITIES

To complete an upgrade, complete the procedures in the section 8.2.
8.1 Configuration settings

Procedure 20: Configuration Settings
Step Procedure SPR Data Access GUI Form Result
1.
In Policy 8.0, the SPR Data Access from the Policy GUI requires a new setting on the Data Source form. This is the IP address of the SPR server OAM interface. Edit the Data Source form on the Policy Server(s) to add this setting. Confirm that the SPR Data Access form is working. In Policy 8.0, the Stale session cleanup settings are more aggressive that in 7.5. Monitor TPS rates during the periodic Stale session cleanup activity, to confirm that the activity does not put too much load on the system (particularly for the PP5160 architecture). If adjustments are needed, contact Tekelec Support. Re-apply any needed Advanced configuration settings
2.
Adjust Settings for Stale Session clean up
3.
Advanced settings from preupgrade
8.2 Verify System Upgrade

This procedure is used to verify that the Policy 8.0 software upgrade was successful.
Procedure 21: Verify System Upgrade

Step Procedure Monitor system Daily Result
1.
Monitor system health once per day, for several days. Note new Trending Reports available on the 8.0 system, to observe performance over time.
2. 3. 4. 5. Add additional Verify steps, based on network specifics and Operator need.
UP006174
Version 1.4
101 of 136
8.3 Additional Instructions

Refer to both the Release Notes for the target release and the Tekelec Customer Care Method of Procedure to determine if additional instructions are to be followed to successfully complete the Policy 8.0 software Upgrade for servers running specific Tekelec Applications.
UP006174
Version 1.4
102 of 136
9. BACKOUT (ROLLBACK)
To complete a backout, complete the procedures in this section. If the Upgrade has succeeded, but an issue is found after upgrade that is causing network impact, then the system can be backed out (rolled back) to the previous release. [Note: If an Upgrade fails, it will automatically attempt to backout.] The backout order is the reverse of the upgrade order:
1) 2) 3) 4)
Backout the Replication Activation Backout the MRA and MPE clusters Backout the secondary CMP cluster Backout the primary CMP cluster.
During a backout, it is important to control what version of the software is currently active. This control needs to be maintained even if there are unexpected failures. This MOP uses the forced standby flag to ensure that a server cant become active until the flag is cleared. Setting and clearing the forced standby flag is critical to having an orderly backout. Failing to follow the conventions can lead to loss of service and even possible data corruption. In the case of an MPE/MRA, the upgrade/backout is NOT complete until the operator does a configuration push from the CMP. The MRA/MPE can still operate to a degree but it is not fully functional.
UP006174
Version 1.4
103 of 136
9.1 Backout of Replication Activation

This procedure performs a backout of the Replication Activation. It must be applied to all servers, if any server cluster will be backout to the 7.5 release.
Pre-conditions: All servers in the Policy system are previously upgraded to the 8.0 Release Some or all servers had the Upgrade Completion applied (which activ ates the new replication).
Procedure 22: Backout of Replication Activation

Step
1. 2.
Procedure GUI: Open CMP GUI SSH: Open ssh session to Primary Active CMP server 1) Access the login prompt. 2) Log into the server as the root user
Result
Login to CMP GUI login: root Password: <enter password>
3.
SSH: Primary Active CMP Confirm that new Replication is active to all servers
# irepstat
UP006174
Version 1.4
104 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 22: Backout of Replication Activation
Step
4.
Procedure SSH: Primary Active CMP -- Re-execute the PrepareUpgrade, if needed
Result
If the Procedure to Remove Replication Exclusions was previously executed, this needs to be backed out as described in this step:
Execute command to Disable Replication for certain data tables, before the backout.
# iqt -p NodeInfo
nodeId nodeName hostName inhibitFlag nodeCap excludeTables A3411.121 slak-cmp-b slak-cmp-b,10.250.85.26 MasterCapable A3411.190 slak-cmp-a slak-cmp-a,10.250.85.25 H MasterCapable C1428.038 slak-mpe-07a slak-mpe-07a,10.250.85.28 MasterCapable C1428.073 slak-mpe-07b slak-mpe-07b,10.250.85.29 MasterCapable C3265.167 slak-mra-b slak-mra-b,10.250.85.5 MasterCapable C3265.212 slak-mra-a slak-mra-a,10.250.85.4 MasterCapable C3573.020 slak-mpe-01a slak-mpe-01a,10.250.85.7 MasterCapable C3573.027 slak-mpe-01b slak-mpe-01b,10.250.85.8 MasterCapable
# policyUpgrade.pl --prepareUpgrade
Verify that file is updated with excludeTables LongParam,AppEventDef,HaCfg
# iqt -p NodeInfo
nodeId nodeName hostName inhibitFlag nodeCap excludeTables A3411.121 slak-cmp-b slak-cmp-b,10.250.85.26 MasterCapable LongParam,AppEventDef,HaCfg A3411.190 slak-cmp-a slak-cmp-a,10.250.85.25 H MasterCapable LongParam,AppEventDef,HaCfg C1428.038 slak-mpe-07a slak-mpe-07a,10.250.85.28 MasterCapable LongParam,AppEventDef,HaCfg C1428.073 slak-mpe-07b slak-mpe-07b,10.250.85.29 MasterCapable LongParam,AppEventDef,HaCfg C3265.167 slak-mra-b slak-mra-b,10.250.85.5 MasterCapable LongParam,AppEventDef,HaCfg C3265.212 slak-mra-a slak-mra-a,10.250.85.4 MasterCapable LongParam,AppEventDef,HaCfg C3573.020 slak-mpe-01a slak-mpe-01a,10.250.85.7 MasterCapable LongParam,AppEventDef,HaCfg C3573.027 slak-mpe-01b slak-mpe-01b,10.250.85.8 MasterCapable LongParam,AppEventDef,HaCfg
Note: this change is automatically replicated to all 7.5.x servers from the Active CMP, and notifies the servers not to process any further updates to these tables. This step is needed since the upgraded CMPs (8.0) may send table updates to the 7.5.x servers that they will not be able to process correctly. Note: This Minor Alarm may be expected from servers
31101 - GN_WARNING/WRN configuration change forcing re-init [SyncMaster.cxx:587], but will clear itself very quickly.
5.
GUI: Confirm Upgrade status of all sites and servers
Upgrade Manager System Maintenance Confirm in the Running Release column that all servers in the network are upgraded to 8.0.
UP006174
Version 1.4
105 of 136
Step
6.
Procedure GUI: Change Replication mode: CMPs (One cluster at a time)
Result
If one of more CMP clusters need to be backout out. Do Primary Site First, then Secondardy Site. Upgrade Manager System Maintenance Select CMP cluster
1) Select the checkbox for the Standby CMP server, and execute:
Turn Off Replication Turn On Legacy Replication Turn On Replication
2) Select the checkbox for the Active CMP server, and execute:
GUI: Change Replication mode: MPEs (One cluster at a time)
Upgrade Manager System Maintenance Select MPE cluster
1) Select the checkbox for the Standby MPE server, and execute:
After each step, wait till the status is updated on the form
2) Select the checkbox for the Active MPE server, and execute:
After each step, wait till the status is updated on the form REPEAT for other MPE clusters to backout. UP006174 Version 1.4 106 of 136
Step Procedure GUI: Verify KPI Dashboard Result
8.
System Wide Reports KPI Dashboard Verify network status is normal. IMPORTANT: if a MPE connections go down for more than a few seconds, it may be necessary to do a Restart on the Active server. (This is because the Signaling Default Route may be lost, in some cases.)
9.
GUI: Change Replication mode: MRAs
Upgrade Manager System Maintenance Select Secondary-Site MRA cluster
1) Select the checkbox for the Standby CMP server, and execute:
After each step, wait till the status is updated on the form
2) Select the checkbox for the Active CMP server, and execute:
After each step, wait till the status is updated on the form REPEAT for other MRAs System Wide Reports KPI Dashboard Verify network status is normal. 11.
10.
GUI: Verify KPI Dashboard
System Wide Reports Active Alarms All related alarms should be cleared.
UP006174
Version 1.4
107 of 136
Step Procedure SSH: Primary Active CMP, confirm that Legacy replication to MPE/MRAs is Active/Standby Result
12.
# inetstat
9.2 Backout of Partial-upgraded Cluster

This procedure is used to backout a cluster that has been partially upgraded. Expected Pre-conditions: Primary Active CMP is on 8.0 Cluster is any of MPE, MRA or Secondary CMP One server of target cluster is on 8.0, and Active One server of target cluster is on 7.5.x and Force Standby
At the end of this procedure, both servers of the target cluster will be on 7.5.x, and Active/Standby.
Procedure 23: Backout Partial-upgraded Cluster

Step Procedure GUI: Upgrade Manager Verify cluster status Result
1.
Upgrade Manager System Management Confirm status of the cluster to be backed out. IF cluster 8.0 server to backout is currently Active -Execute this step to make 8.0 server Force Standby, and make the 7.5.x server Active. [IF 8.0 server is already Force Standby, skip this step.] IMPORTANT: the current MRA or MPE session data is dropped in this step. The 7.5.x MRA/MPE will start from a clean data set. Login to Primary Active CMP as root # policyUpgrade.pl --failover <MPE/MRA_Hostname>
2.
SSH: Primary Active CMP server Switch active server back to 7.5.x Service Affecting for MPE/MRA
UP006174
Version 1.4
108 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 23: Backout Partial-upgraded Cluster
Step
3. 4.
Procedure GUI: Upgrade Manager Verify cluster status GUI: View KPI Dashboard Verify 7.5.x server is handling traffic
Result
Verify failover is completed, and 7.5.x server is Active. Verify steps: - KPI Dashboard - View MRA/MPE Report IF there is a problem Consult with Tekelec Support.
5.
Option 1: GUI: Upgrade Manager - Backout the 8.0 server software
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to backout GUI: Upgrade Manager System Maintenance Select Checkbox for the Server to be backed out. Current state must be Force Standby Select Operation: Backout Server backout takes several minutes, and the final step will be a re-boot of the server. Verify: GUI: Upgrade Manager System Maintenance Select Operation: Push Script - Confirm Upgrade Manager now shows correct release, and - Upgrade status = Completed: backout was completed at Option 2 execute backout from ssh root login to target Log into the target 8.0 server as root: # getPolicyRev 8.0.0_xxx # cd /var/TKLC/backout # ./ugwrap --backout NOTE: there are two dashs (--) before backout Initializing Upgrade Wrapper... Executing any special platform directives Setting up application for install/upgrade Running backout_server script... Starting backout_server... Verifying that backout is possible. Current platform version: 5.0.1-72.45.0 Backing out to platform version: 4.2.4-70.90.0 compare_platform_versions (5.0.1-72.45.0, 4.2.470.90.0) compare with major upgrade boundary (3.0.0-60.0.0, 4.2.4-70.90.0) compare with no backout boundary (4.0.0-70.0.0,
6.
Option 2: SSH: Backout the target 8.0 server
UP006174
Version 1.4
109 of 136
4.2.4-70.90.0) Backout Date: 08/10/2012 02:10:24 UTC Continue backout? [y/N]:y Server backout takes several minutes. After returning to prompt, verify success: # tail /var/TKLC/log/upgrade/upgrade.log Daemon is not running... 1344561040::DEBUG: lib/upgrade.sh - app_enable() - APP_ENABLE=[0] 1344561040::DEBUG: lib/upgrade.sh - app_enable() - MODE_FLAG=[-backout] 1344561040:: Enabling applications on the server... 1344561040:: 1344561041:: 1344561042:: 1344561043:: Applications Enabled. 1344561043:: Running /usr/TKLC/plat/bin/service_conf reconfig 1344561045:: Backout is complete. A reboot of the server is now required. # shutdown -r now Verify: After reboot, login and check status of server: # getPolicyRev 7.5_xxx # syscheck # ha.stat 7.
GUI: Re-Apply Configuration to MPE/MRA
IF target is MPE or MRA, Re-Apply the configuration from the CMP GUI. For MPE - GUI: Policy Server -> Configuration: System -> ReApply Configuration For MRA - GUI: MRA -> Configuration: System -> Re-Apply Configuration
UP006174
Version 1.4
110 of 136
Step
8.
Procedure GUI: Upgrade Manager Cancel Force Standby
Result
GUI: Upgrade Manager System Maintenance Select Checkbox for the Server that is Force Standby Operation: Cancel Force Standby Verify status of the Server changes to Standby.
9.
GUI: Verify cluster is handling traffic as normal
Verify-
KPI Dashboard
MPE/MRA Reports Active Alarms
UP006174
Version 1.4
111 of 136
9.3 Backout of Fully Upgraded Cluster

This procedure is used to backout a cluster that has been fully upgraded. i.e. Both servers in the cluster are installed with 8.0 and they are Active/Standby. Pre-conditions: Primary Active CMP is on 8.0 Cluster is any of MPE, MRA or Secondary CMP One server of target cluster is on 8.0, and Active One server of target cluster is on 8.0 and either Standby or Force Standby
At the end of this procedure, both servers of the target cluster will be on 7.5.x, and Active/Standby.
Procedure 24: Backout Fully Upgraded Cluster

Step Procedure GUI: Upgrade Manager Set Standby server to Force Standby (Backout first server in cluster) Result
5.
IF cluster is Active/Standby, set Standby Server to Force Standby. GUI: Upgrade Manager System Maintenance View Select Checkbox for the standby Server. Select Operation: Force Standby
6.
Option 1: GUI: Upgrade Manager - Backout the 8.0 server software (Backout first server in cluster)
Choose Option 1 or Option 2 below, to Backout the server: Option 1 use the Upgrade Manager GUI tool to backout GUI: Upgrade Manager System Maintenance View Select Checkbox for the Force Standby Server to be backed out. Operation: Backout Server backout takes several minutes, and the final step will be a re-boot of the server. Verify: - Confirm Upgrade Manager shows server has correct release
UP006174
Version 1.4
112 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 24: Backout Fully Upgraded Cluster
Step Procedure Option 2: SSH: Backout the target 8.0 server (Backout first server in cluster) Result
7.
Option 2 execute backout from ssh root login to target Log into the target 8.0 server as root: # getPolicyRev 8.0.0_xxx # cd /var/TKLC/backout # ./ugwrap --backout NOTE: there are two dashs (--) before backout.. <Answer yes> Server backout takes several minutes. After the backout script completes, it is necessary to reboot the server. # shutdown -r now Verify: After reboot, login and check status of server: # getPolicyRev 7.5.x_x.x.x # syscheck # ha.stat
8.
IF target is MPE or MRA, Re-Apply the configuration from the CMP GUI. For MPE - GUI: Policy Server -> Configuration: System -> Re-Apply Configuration For MRA - GUI: MRA -> Configuration: System -> Re-Apply Configuration The Backed out 7.5.x server is Force Standby. IMPORTANT: DO not Remove Force Standby. A 8.0 server and a 7.5.x cannot be Active/Standby. must remain in the Force Standby state. Note: The Following steps are same as the procedure above for a Partial Upgraded server. One
9.
Cluster is now Partially Upgraded (one server 8.0, and one server 7.5.x)
UP006174
Version 1.4
113 of 136
Step
10.
Procedure SSH: Primary Active CMP server Switch active server back to 7.5.x Service Affecting for MPE/MRA
Result
IF cluster 8.0 server to backout is currently Active -Execute this step to make 8.0 server Force Standby, and make the 7.5.x server Active. [IF 8.0 server is already Force Standby, skip this step.] IMPORTANT: the current MRA or MPE session data is dropped in this step. The 7.5.x MRA/MPE will start from a clean data set. Login to Primary Active CMP as root # policyUpgrade.pl --failover <CMP/MPE/MRA_Hostname>
11.
GUI: Upgrade Manager Verify cluster status GUI: View KPI Dashboard - Verify 7.5.x server is handling traffic
Verify failover is completed, and 7.5.x server is Active. (If CMP is backed out, close Browser and re-open.) Verify steps: - KPI Dashboard - View MRA/MPE Report IF there is a problem Consult with Tekelec Support.
12.
13.
GUI: Verify 7.5.x (Active) server is handling Traffic
The backed out 7.5.x server should be handling traffic. Verify steps: - View Reports on the GUI - View status from Upgrade Manager System View IF there is a problem Consult with Tekelec Support.
14.
Option 1: GUI: Upgrade Manager - Backout the 8.0 server software
Choose Option 1 or Option 2: Option 1 use the Upgrade Manager GUI tool to backout View Upgrade Manager System Maintenance View Select Checkbox for the Server to be backed out. Current state must be Force Standby Select Operation: Backout Server backout takes several minutes, and the final step will be a re-boot of the server. Verify: - Confirm Upgrade Manager shows server of correct release
UP006174
Version 1.4
114 of 136
Step Procedure Option 2: SSH: Backout the target 8.0 server Result
15.
Option 2 execute backout from ssh root login to target Log into the target 8.0 server as root: # getPolicyRev 8.0.0_xxx # cd /var/TKLC/backout # ./ugwrap --backout NOTE: there are two dashs (--) before backout.. <Answer yes> Server backout takes several minutes. . mysql stopped Installing JDK with option --nomd5 No JDK backout package found, ignoring... # # shutdown -r now Verify: After reboot, login and check status of server: # getPolicyRev 7.5_xxx # syscheck # ha.stat
16.
IF target is MPE or MRA, Re-Apply the configuration from the CMP GUI. For MPE - GUI: Policy Server -> Configuration: System -> Re-Apply Configuration For MRA - GUI: MRA -> Configuration: System -> Re-Apply Configuration
UP006174
Version 1.4
115 of 136
Step
17.
Procedure SSH: Primary Active CMP - Remove Force Standby
Result
Login to Primary Active CMP as root # getPolicyRev # ivi NodeInfo Edit depends on the result of the getPolicyRev: - If 7.5.x CMP, then remove the H - If 8.0 CMP, then change the Stby to Active :wq Answer yes Verify: # ha.stat Should show active/standby after a minute or so. THIS PROCEDURE HAS BEEN COMPLETED
9.4 Recovery of Server (from Backup or Initial Config)

This procedure is used to recover a server that is in an unknown state, as a result of Upgrade/Backout activities. In this procedure, the server will be installed again as 7.5.x (or 8.0?), and the needed Initial Configuration applied, or data recovered from a previous backup. Before taking this step, consult with Tekelec Support. Required Materials: TPD version for Policy 7.5 Application iso for the server (7.5 cmp, mpe or mra iso) Server backup, or IP/hostname information for Initial Configuration of the server.
Expected Pre-conditions: Primary Active CMP is 8.0 At the end of this procedure, one server will be recovered to 7.5.x, and may be Active or Standby.
Procedure 25: Recovery of Server from Backup

Step Procedure Caution Result
1.
Caution: Do not remove the affected server from the Topology forms on the CMP GUI. Modification of the Topology forms is not supported during upgrade activities.
UP006174
Version 1.4
116 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 25: Recovery of Server from Backup
Step
2.
Procedure Console/iLo/PMAC: Clean Install server (re-install TPD OS)
Result
At this step, the purpose is clean any disk areas previously used, and re-install the TPD OS. Access a login on the server, and perform these commands: # service qp_procmgr stop # prod.stop # getPlatRev If the plat rev is 4.0, then: # ./usr/TKLC/plat/sbin removeVG --scrub If the plat rev is 5.0, then: # ./usr/TKLC/plat/sbin storageClean lvm --vgName=vgroot --level=scrub If PMAC is available, use PMAC to Install OS on the server. If PMAC is not available, then: Access iLo/RMM port of server, and start remote Console. Mount the TPD OS iso on the server (either CD drive, or iLo Virtual Mount). # shutdown r now boot: <enter boot command for the server>
3.
Console/iLo/PMAC: Install the Application
If PMAC is available, use PMAC to install (Upgrade) the Application. If PMAC is not available, Mount the Application iso on the server (either CD drive, or iLo Virtual Mount). # su - platcfg Maintenance Upgrade
4. 5.
Console/iLo: Copy server backup to the server. Console/iLo: Execute Restore from Backup
Use iLo access to transfer the Backup file to the server. # su platcfg Execute Restore Wait for boot
6.
GUI: Confirm server is synced to the CMP.
Platform Administration Topology View the cluster from the Topology form, to confirm that the re-installed server is detected. THIS PROCEDURE HAS BEEN COMPLETED
UP006174
Version 1.4
117 of 136
APPENDIX A. MANAGING HA STATUS OF SERVERS A.1 Understanding the ha.states and ha.mystate commands
IMPORTANT: ha.stat command is no longer supported in Rel 8.0. It is replaced with 2 commands:
ha.mystate ha.states
The ha.states or ha.mystate command is executed as root on any of the CMP, MRA or MPE servers. It reports the High Availability status of the clustered servers, or just the single server, respectively. The ha.states command refreshes the status every second, and will run continually till the user exits with a cntl-C. The ha.mystate command runs once and exits. Both have the same data format. This is the example of the normal display of these commands on a server which is fully clustered: ha.mystate command output:
ha.states command output:
UP006174
Version 1.4
118 of 136
There are several key fields in the ha.states. Resource function on the Node that is being reported: QP (Application), Replication, and IP VIP ownership Role Status of HA relationship for the Resource: Active, Standby, OOS NodeId Identifier used in the software for this specific Node instance
In the Normal condition: One server will show Active for QP, Replication and VIP Other server will show Standby for QP, Replication and VIP The same ha.states status will be reported from both servers in the cluster
UP006174
Version 1.4
119 of 136
APPENDIX B. METHODS OF DELIVERING SOFTWARE UPGRADE ISO

There are several methods to deliver the Software iso to the server. The above Upgrade procedure assumes scp is used. In this appendix is a list of several other methods that may be useful. IMPORTANT: There should be a TPD DVD and Application DVD left on-site, to aid in re-installing a server after a field repair.
B.1 Copy iso from USB Key

It is possible to put the upgrade iso on a USB key, and use this to load the iso to the server. To do this: USB must be formatted with FAT 32, and at least 1G Copy iso to USB key, from a laptop or any computer Insert USB key to server Mount to /mnt/upgrade Copy the iso file to /var/TKLC/upgrade. Unmount USB and remove
B.2 Copy iso from DVD {PP5160, DL360}

If a three Application DVDs are delivered to a site (CMP, MRA, MPE), but there multiple servers to be upgraded, it may be useful to extract the iso from the DVD, and copy to the servers that need it, prior to the Maintenance interface. As long as the iso is placed in the /var/TKLC/upgrade directory, the Upgrade will find the iso, and use it for the installation.
Procedure 26: Upgrade from physical CD media {PP5160, DL360}

Step Procedure Insert Policy 8.0 Upgrade CD Result
1.
Insert media in CD-ROM tray
2.
1) Access the login prompt. 2) Log into the server as the root user on the iLO or RMM.
CentOS release 4.6 (Final) Kernel 2.6.18-1.2849prerel3.3.0_63.1.0 on an i686 localhost login: root Password: <root_password>
3.
Verify ISO images do not already exist by examining contents of /var/TKLC/upgrade directory If ISO image files exist you will need to remove them.
# ls al /var/TKLC/upgrade total 16 dr-xr-xr-x dr-xr-xr-x # 2 root root 4096 Oct 22 16:31 . 21 root root 4096 Oct 18 13:40 ..
UP006174
Version 1.4
120 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 26: Upgrade from physical CD media {PP5160, DL360}
Step Procedure Determine the physical device name. The primary physical device will be the first device listed. In the example it is device hda. Result
4.
# getCDROM SONY DVD RW AW-G540A|hda Intel(R) RMM2 VDrive 2|scd0 Intel(R) RMM2 VDrive 3|scd1 Intel(R) RMM2 VDrive 4|scd2 Intel(R) RMM2 VDrive 1|scd3
# mount /dev/<dev> /mnt/upgrade
Example: # mount /dev/hda /mnt/upgrade
5.
Mount the physical media
6.
Validate physical media Verify that the command output indicates the CDROM is Valid.
# /mnt/upgrade/upgrade/.validate/validate_cd
Below is an example of the command output. Actual values returned may vary depending on version of software and firmware installed.
Validating cdrom... UMVT Validate Utility v1.10.0, (c)Tekelec, January 2009 Validating /var/TKLC/upgrade/872-2069-021.1.0_70.36.0_SUP35.iso Date&Time: 2010-03-18 14:21:16 Volume ID: 872-2069-02_Rev_A;70.36.0 Part Number: 872-2069-02_Rev_A Version: 70.36.0 Disc Label: TPD Disc description: TPD The media validation is complete, the result is: PASS CDROM is Valid
Note: Do not continue if CD validation reports any errors or is invalid until new physical media can be obtained.
7.
Change to the upgrade directory
# cd /var/TKLC/upgrade
UP006174
Version 1.4
121 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 26: Upgrade from physical CD media {PP5160, DL360}
Step Procedure Verify enough space exists for ISO Verify that there is at least 600M in the Avail column. If not, clean up files until there is space available. Make sure you know what files you can remove safely before cleaning up. It is recommended that you only clean up files in the /var/TKLC/upgrade directory as this is a platform owned directory that should only contain ISO images. This directory should not be expected to contain images for any length of time as they can get purged. Removing files other than those in directory /var/TKLC/upgrade is potentially dangerous. Result
8.
# df -h /var/TKLC Filesystem /dev/md8 Size 4.0G Used Avail Use% Mounted on 89M 3.7G 3% /var/TKLC
9. 10. 11.
Copy ISO
# cp /mnt/upgrade/*.iso /var/TKLC/upgrade
Remove media in CD-ROM tray
Remove CD Procedure to ISO image validation
Proceed to Procedure 5 in Section Error! Reference source not found.
REF _Ref254685737 \h \* MERGEFORMAT Error! Reference source not found.
UP006174
Version 1.4
122 of 136
APPENDIX C. UPGRADE PMAC ON A C-CLASS SYSTEM

In Policy Rel 8.0, PMAC is not used to perform Upgrades. It is used for Installation activities, growth of new servers and Field repair activities. It is also used for deploying Firmware upgrades. This section includes procedures to upgrade the PMAC, from the following reference: PM&C 3.x/4.x Incremental Upgrade; 909-1619-001 Procedure 1: PM&C Health Check S T E P # 1. This procedure provides instructions on how to perform a healthcheck on the management server hosting the PM&C application.
Check off () each step as it is completed. Boxes have been provided for this purpose under each step number. IF THIS PROCEDURE FAILS, CONTACT TEKELEC TECHNICAL SERVICES AND ASK FOR ASSISTANCE.
Access 1 the management server command prompt At 1 the command prompt, run the sentry status command to verify the status of the PM&C application.
Access the management server command prompt.
2.
[root@foo-1060101-a ~]# sentry status sending status command... SMAC Sentry Status -----------------sentryd started: Sun Dec 6 07:47:31 2009 Current activity mode: ACTIVE Process PID Status StartTS ------------------ ------ ----------- ------------------------smacTalk 5932 running Sun Dec 6 07:47:31 2009 smacMon 5935 running Sun Dec 6 07:47:31 2009 hpiPortAudit 5951 running Sun Dec 6 07:47:31 2009 snmpEventHandler 5962 running Sun Dec 6 07:47:31 2009 snmpSubagent 5967 running Sun Dec 6 07:47:31 2009 eclipseHelp 5971 running Sun Dec 6 07:47:31 2009 Sun Dec 6 07:47:57 2009 Command Complete. [root@foo-1060101-a ~]# [root@foo-1060101-a ~]#alarmMgr alarmStatus [root@foo-1060101-a ~]#
NumR ---1 1 1 1 1 1
3.
At 1 the command prompt, run the alarmMgr utility.
UP006174
Version 1.4
123 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 1: PM&C Health Check 4.
Verify there is no 1 problem with the management server or PM&C application. Contact Tekelec TAC for information on how to proceed.
If sentry shows any PM&C processes not running or alarmMgr shows any failures, then the healthcheck was not successful. If the healthcheck was not successful, contact Tekelec TAC for information on how to proceed. Otherwise, PM&C appears to be running normally.
Procedure 2: Ensure that /etc/dhcpd.conf Exists on the primary Management Server S T E P # This procedure creates the dhcpd.conf file if it does not exist. In PM&C releases prior to 3.1.1-31.8.0, a bug in the resetProfileConfig command caused /etc/dhcpd.conf to be deleted, resulting in failed upgrades. This procedure acts as a workaround.
Check off () each step as it is completed. Boxes have been provided for this purpose under each step number.
1.
Should this procedure fail, contact the Tekelec Customer Care Center and ask for UPGRADE ASSISTANCE. [root@PMACDev3 ~]# touch /etc/dhcpd.conf Execute 1 the touch command on /etc/dhcpd.conf.
If the PM&C application ISO was delivered to the system remotely (via SCP or SFTP) then make sure the image is located in the /var/TKLC/upgrade directory prior to executing this procedure. This should have been done as part of the healthcheck procedure. Procedure 3: PM&C Upgrade Procedure on the primary Management Server S T E P # 1. This procedure provides instructions to perform a software upgrade of the PM&C server.
If PM&C 1 application ISO is delivered on CD/DVD media, Insert the CD/DVD containing the appropriate release into the optical drive of the management server.
Insert the CD/DVDinto the optical drive of the management server.
NOTE: This step can be skipped if image was delivered to the /var/TKLC/upgrade directory.
UP006174
Version 1.4
124 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 3: PM&C Upgrade Procedure on the primary Management Server 2.
3.
Access 1 the management server command prompt Run 1 the platcfg utility.
Access the management server command prompt as detailed in Error! eference source not found. (Accessing the management server command prompt).
[root@foo-1060101-a ~]# su platcfg
4.
Execute 1 the following steps using the Arrow and the [ENTER] keys to navigate through the menu options: a) Select Maintenance to navigate to the Maintenance Menu. b) Select Upgrade to navigate to the Upgrade Menu. c) Finally, select Initiate Upgrade to start the upgrade process.
UP006174
Version 1.4
125 of 136
The 1 screen shown to the right may be displayed several times as the Platcfg utility searches for available upgrade media.
6.
Select 1 the target release level (use the Arrow keys if necessary) and press [ENTER].
If the image is located on CD/DVD, then the menu would look similar to this:
If the image was copied to the /var/TKLC/upgrade directory, then the menu would look similar to this:
7.
Screens 1 similar to the one shown to the right will be displayed as the upgrade progresses.
UP006174
Version 1.4
126 of 136
Screens 1 similar to the one shown to the right will be displayed as the upgrade progresses.
9.
If the 1 upgrade completes successfully, the screen shown to the right will be displayed as the upgrade progresses.
NOTE: If the PM&C upgrade fails to complete, contact Tekelec Customer Service for assistance;
Tekelec Customer Care Center US: 1-888-367-8552 Intl: +1-919-460-2150
UP006174
Version 1.4
127 of 136
Upon 1 successful completion of the blade reboot, the user should be returned to the login prompt.
The output at the right would be seen if connected via a serial or VGA connection to the management server.
11.
If present, 1 remove the CD/DVD containing the appropriate release from the optical drive of the management server.
Remove the CD/DVD from the optical drive of the management server.
Procedure has been completed.
Procedure 4: Post Upgrade Verification on the primary Management Server S T E P # 1. This procedure provides instructions to perform after upgrade of the PM&C application to verify the success of the upgrade.
2.
Access 1 the management server command prompt Verify 1 that the date/time stamp of the upgrade log aligns with the time of the upgrade.
Access the management server command prompt.
[root@foo-1060101-a ~]# ls -l /var/TKLC/log/upgrade/upgrade.log -rw-rw-r-- 1 platcfg root 22623 Feb 9 18:28 /var/TKLC/log/upgrade/upgrade.log [root@foo-1060101-a ~]#
UP006174
Version 1.4
128 of 136
Policy 7.5 to 8.0 - Upgrade Procedure Procedure 4: Post Upgrade Verification on the primary Management Server 3.
Verify 1 that the release has been updated.
Execute the following command:

[root@foo-1060101-a ~]# cat /usr/TKLC/smac/etc/pmac-release
If PM&C release 3.x, output like the following is expected:

3.0.0_30.5.0 [root@foo-1060101-a ~]#
If PM&C release 4.x, output like the following is expected:4.0.0_40.11.0

[root@foo-1060101-a ~]#
If the new target release number is not displayed, then upgrade was not successful. Contact Tekelec Customer Service and do not proceed until instructed by a Tekelec Customer Care representative. 4. Verify 1 upgrade completion through the upgrade and ugwrap logs. Examine the upgrade logs in the directory /var/TKLC/log/upgrade and verify that no errors were reported. Execute the following command on the management server:
[root@foo-1060101-a ~]# grep COMPLETE /var/TKLC/log/upgrade/upgrade.log
Output like the following is expected (the timestamp will be different): NOTE: If the PM&C upgrade has failed, contact Tekelec Customer Service for assistance;
1204213764:: UPGRADE IS COMPLETE 1204213764::DEBUG: ADDING VAR: UPGRADE_COMPLETED = 02/28/2008 10:49:24 [root@foo-1060101-a ~]#
Execute the following command: NOTE: This command can take over a minute to complete
[root@foo-1060101-a ~]# verifyUpgrade
Tekelec Customer Care Center US: 1-888-367-8552 Intl: +1-919-460-2150
No output is expected:
[root@foo-1060101-a ~]#
Execute the following command:

[root@foo-1060101-a ~]# grep ERROR /var/TKLC/log/upgrade/ugwrap.log
No output is expected:
[root@foo-1060101-a ~]#
If UPGRADE IS COMPLETE is not output from the first command, or if any output showing errors results from the other commands, contact Tekelec Customer Service and do not proceed until instructed by a Tekelec Customer Care representative. 5.
6.
Execute 1 the system healthcheck.
Execute the PM&C Healthcheck procedures in PM&C Health Check procedure If any error or failure conditions are discovered on the management server or PM&C application then do not proceed. Contact Tekelec TAC to work to resolve the failure conditions. If the upgrade was done remotely, then remove the copy of the Application ISO image file that was copied to the /var/TKLC/upgrade directory.
If present, 1 remove the target-release Application ISO Image file
UP006174
Version 1.4
129 of 136
APPENDIX D. USING ILO (OR RMM) TO REMOTELY ACCESS A SERVER

The iLo (or RMM for PP5160) interface of the server is a method to get access to the server, even if it wont boot. The remote console access option of the iLo (or RMM) can be used to get console access to the server. This has the benefit that the user will see the console output while the server is re-booting. The remote console access can also be used in case the server IP interfaces are down, and the server state is unknown. From this interface, it is also possible to mount an iso located on your computer to the server, using the iLo (or RMM) Virtual Mount utility. You can also remotely force a boot of the server.
UP006174
Version 1.4
130 of 136
UP006174
Version 1.4
131 of 136
APPENDIX E. ACCESSING TEKELECS CUSTOMER SUPPORT SITE

Access to Tekelec's Customer Support site is restricted to current Tekelec customers only. This section describes how to log into Tekelecs Customer Support site and locate a document. Viewing the document requires Adobe Acrobat Reader, which can be downloaded at www.adobe.com. 1. Log into Tekelecs new Customer Support site at support.tekelec.com. Note: If you have not registered yet for this new site, click the Register Here link. Have your customer number available. The response time to registration requests is 24 to 48 hours. 2. 3. 4. 5. Click the Product Support tab. Use the Search field to locate quickly a document by its part number, release number, document name, or document type. The Search field accepts both full and partial entries. Click a subject folder to browse through a list of related files. To download a file to your location, right-click the file name and select Save Target As.
UP006174
Version 1.4
132 of 136
APPENDIX F. USING SCREEN SHELL TOOL

The Linux screen tool is provided on the Policy servers, to allow a ssh session that will not be lost during a remote access disconnect. It also provides a method to log the user activities and responses into a file. To execute some action that should not be interrupted, ssh to the server, and start a screen session. Once the session is started, all the same privileges and commands are available but the session will be maintained even if the user connection to the server is broken. Start Screen Session ------------------------SSH to server, login as root # screen # <user commands for upgrade> Terminate a previously started session ----------------------------------------------To terminate the screen session: # exit Detach from session -------------------------To leave the session, but keep the session running (detach): # screen -d # exit Re-Attach to a previously started session --------------------------------------------------SSH to server, login as root # screen -ls There is a screen on: 31808.pts-0.<hostname> (Detached) 1 Socket in /var/run/screen/S-root. # screen -x 31808 <user is now back in the same session started before the detach or disconnect>
Screen Logging -------------------Using Ctrl-A H, creates a running log of the session. Screen will keep appending data to the file through multiple sessions. Using the log function is very useful for capturing what you have done, especially if you are making a lot of changes.
Screen Help ---------------Ctrl-A then ?
Other capabilities ----------------------The screen tool has multiple capabilities. Further information is available on the Web.
UP006174
Version 1.4
133 of 136
APPENDIX G. TN003516 TO CLEAN UP FILES

The /var partition of an MPE system may inadvertently reach 100% full due to several MPE cron jobs reporting output in /var/spool/clientmqueue/. To determine if this issue is present, the following commands can be used # df -h /var Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-plat_var 992M 873M 69M 93% /var
# du -h /var/spool/clientmqueue
703M /var/spool/clientmqueue
Clear the existing usage If the /var partition is over 95%, the following commands can be used to clean the partition 1. SSH into the system 2. Run the following command # rm -rf /var/spool/clientmqueue/* 3. Verify the disk space has decreased # df -h /var Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-plat_var 992M 189M 752M 21% /var
Permanent Fix Instructions 4. SSH into the system 5. Append > /dev/null 2>&1 to each line in file /etc/cron.d/sysplot 6. Execute command to refresh crontab task, so we will see consistent output for crontab l. # crontab /etc/cron.d/sysplot 7. Verify the settings using the crontab l (dash lowercase L) command # crontab -l
*/10 * * * * root /opt/camiant/bin/sysplot.pl -count=119 > /dev/null 2>&1 59 23 * * * root /opt/camiant/bin/sysplot.pl -png > /dev/null 2>&1 01 01 * * * root /opt/camiant/bin/sysplot.pl -remove > /dev/null 2>&1
UP006174
Version 1.4
134 of 136
APPENDIX H. HANDLING ICMP BLOCKING SCENARIO

If icmp is blocked between the servers the command "keyexchange user@hostname" can fail. This is because the command initiates pings towards the destination server. Ssh between the hosts will not be blocked but the keys will not be exchanged. If you run in to this issue use the command "key exchange debug noping hostname.
UP006174
Version 1.4
135 of 136
APPENDIX I. REVIEW COMMENTS

Add PMAC Backout section o Recovery of a PMAC failed upgrade is a PMAC re-install. See installation document. Add step in upgrade MOP to verify that all Customer installation customizations are still in place after upgrade (customer specific): ex: o Stale session cleanup settings o Other: Watchdog timer var cleanup
300 ms delay
UP006174
Version 1.4
136 of 136

UP006174 7 5to8 0

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

UP006174 7 5to8 0

Загружено:

Авторское право:

Доступные форматы

Policy 7.5 to 8.

Software Upgrade Procedure

Broadband Network Solutions Software Upgrade Procedure

Policy 7.5 to 8.0 - Upgrade Procedure

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

10/16/12 10/16/12 10/30/12 12/03/12

1.3 1.4 1.5 1.6

NPx NPx NPx CS

Yes Yes Yes Yes Yes

3/07/13 1.7 CS *Through Formal Peer Review

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

2.11 Upgrade Manager Process ......................................................................................... 13

Policy 7.5 to 8.0 - Upgrade Procedure 6.1.1 6.1.2 6.1.3 6.1.4

Software Upgrade Procedure

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

1. INTRODUCTION 1.1 Purpose and Scope

1.3 Software Release Numbering

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

System Health Check PP5160 Upgrade Upgrade Ready Watchdog

Software Upgrade Procedure

Mount the Update ISO

# mount -o loop /var/TKLC/upgrade/<Flash /mnt/upgrade

Example: # mount -o loop /var/TKLC/upgrade/872-2068-01-1.0.0_70.28.0.iso /mnt/upgrade

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

2.1 Upgrade Path

2.3 SPR Upgrade to Rel 8.0

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

2.4 Upgrade Sequence

2.5 Customer Impacts

2.6 Rollback (Backout)

2.7 TPD Version

2.8 Server Hardware Platforms

2.9 Loading Application software

Specific limitations are to be determined. Version 1.4 10 of 136

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

2.10 Required Materials and Access

2.10.1 Upgrade Media

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

2.10.2 Logins, Passwords and Server IP Addresses

CMP servers (each CMP server)

Software Upgrade Pack Target Release 4

Policy 7.5 to 8.0 - Upgrade Procedure

Software Upgrade Procedure

2.11 Upgrade Manager Process

Force the selected server(s) to standby

Cancel Force Standby

Cancel the selected server(s) force standby

Turn on the selected server(s) Replication mode.

Turn off Replication

Turn off the selected server(s) Replication mode.

Turn on Legacy Replication

Turn on the selected server(s) Legacy Replication mode