12 Tannery Road #08-03

HB Centre 1, Singapore 347722

Tel: +65 6744 2602
Fax: +65 6747 6007

METHODS OF PROCEDURES
Objective :
To replace Primary A10 ADC Damage Fan 3 Module

Overview :
A10 ADC is deployed on HA Configurations. The Primary ADC FAN 1 Module was reported
damage and was confirmed by A10 TAC Engineers. The replacement module was provided by
A10 as part of Support and Maintenance.

Date and Time of Execution:

1am to 5am; GlobalNet to provide confirmed and detailed schedule

Date and Time of Completion:

1am to 5am; GlobalNet to provide confirmed and detailed schedule

Activity Maintenance Window (in Minutes):

240 (4hrs)

Severity:
Moderate, may result to downtime or network intermittent access

Equipment Requires Corrective Action:

OSS/BSS A10 Primary ADC (SN: TH09014017360020)

System Logs :
GN-OSSBSS-ADC1-Active-vMaster[1/1]#show environment
Updated information every 30 Seconds
Physical System temperature: 36C / 96F : OK-low/med
Physical CPUTIN temperature: 58C / 136F
Fan1A : OK-med/high
Fan2A : OK-med/high
Fan3A : FAILED
System Voltage 12V : OK
System Voltage 5V : OK
System Voltage AVCC 3.3V : OK
System Voltage PCH 1.05V : OK
System Voltage DDR 1.5V : OK
Right Power Unit(view from front) State: On
Left Power Unit(view from front) State: On

GN-OSSBSS-ADC1-Active-vMaster[1/1]#sho version
Thunder Series Unified Application Service Gateway TH940
Copyright 2007-2018 by A10 Networks, Inc. All A10 Networks products are
 protected by one or more of the following US patents:
9756071, 9742879, 9722918, 9712493, 9705800, 9661026, 9621575, 9609052
9602442, 9596286, 9596134, 9584318, 9544364, 9537886, 9531846, 9497201
9477563, 9398011, 9386088, 9356910, 9350744, 9344456, 9344421, 9338225
9294503, 9294467, 9270774, 9270705, 9258332, 9253152, 9231915, 9219751
9215275, 9154584, 9154577, 9124550, 9122853, 9118620, 9118618, 9106561

9094364, 9060003, 9032502, 8977749, 8943577, 8918857, 8914871, 8904512

8897154, 8868765, 8849938, 8826372, 8813180, 8782751, 8782221, 8595819
8595791, 8595383, 8584199, 8464333, 8423676, 8387128, 8332925, 8312507
8291487, 8266235, 8151322, 8079077, 7979585, 7804956, 7716378, 7665138
7675854, 7647635, 7627672, 7596695, 7577833, 7552126, 7392241, 7236491
7139267, 6748084, 6658114, 6535516, 6363075, 6324286, RE44701, 8392563
8103770, 7831712, 7606912, 7346695, 7287084, 6970933, 6473802, 6374300

64-bit Advanced Core OS (ACOS) version 4.1.4-P1, build 69 (Apr-26-2018,19:17)

Booted from Hard Disk primary image
Number of control CPUs is set to 1
Serial Number: TH09014017360020
aFleX version: 2.0.0
GUI primary image (default) version 4_1_4-P1-1_0_1-24
GUI secondary image version N/A
aXAPI version: 3.0
Cylance version: N/A
Hard Disk primary image (default) version 4.1.4-P1, build 69
Hard Disk secondary image version 4.1.3, build 50
Last configuration saved at Nov-30-2018, 05:54
Hardware: 8 CPUs(Stepping 1), Single 29G drive, Free storage is 12G
Total System Memory 8141 Mbytes, Free Memory 1887 Mbytes
Hardware Manufacturing Code: 173611
Current time is Dec-2-2018, 21:35
The system has been up 109 days, 20 hours, 55 minutes

Tools and Requirements:

1. Tools such as screw drivers (+ and -)
2. Tester for Power, Fiber and UTP (Optional)
3. Standby UTP Cables, Fiber Cables and Power Cables (Optional)
4. Replacement Fan Module
5. Privilege Access to Primary and Secondary ADC
6. Knowledge to use SSH and basic ADC commands and operations

Corrective Procedure:
1. Back-up Primary and Secondary Running Configurations
 Primary ADC: 172.30.0.1
 Secondary ADC: 172.30.0.2
 VCS: 172.30.0.5

a. SSH to Primary and Secondary ADC and secured privileged mode

i. Open Putty client and key-in the details
ii. Key-in user and password
iii. Secure privileged mode, Execute: ADC$ enable
b. Execute: show tech
c. Copy all outputs to a notepad and save
2. Check Secondary ADC Status and Health to ensure good working condition before
failover.
a. show environment
b. show int brief
c. show vcs summary
d. show vrrp-a
e. show ip ospf
f. show ip ospf redistributed (no redistributed will be seen)
g. show slb virtual server
h. show session

3. Failover VCS Master from Primary ADC to Secondary ADC

a. Secure privileged mod on Primary ADC
b. conf t
c. vcs device 1
d. priority 180 (note that – present value is 200, new value 180 is lower than
Secondary ADC under VSC configuration)
e. Secure privileged mode on Secondary ADC
f. vcs master-taken-over 210 (note that present value is 190, new value is higher than
Primary ADC under VCS configuration)

4. Validate successful VCS failover to Secondary ADC

a. Secure privileged mode on Secondary ADC
b. Show vcs summary (Local is vMaster & Remote is vBlade)
c. ping 172.30.0.5 (ping to VSC IP)
d. ssh to VCS ip (Cluster Access is point to Secondary ADC)

5. Failover VRRP Primary Role from Primary ADC to Secondary ADC

a. Secure privileged mode on Primary ADC
b. conf t
c. vrrp-a vrid1
d. device-context 1
e. blade-parameters
f. priority 100 (note that present value is 200. New value is lower than Secondary
ADC value under VRRP-A configuration)
g. secure privileged mode on Primary ADC
h. conf t
i. vrrp-a force-self-standby enable

6. Validate successful VRRP failover to Secondary ADC

a. Secure privileged mode on Secondary ADC
b. show vrrp-a (Secondary ADC (local) status will be seen as Active and the Primary
ADC (Peer) as Standby)

7. Validate Secondary ADC functions now acting as Master (VCS) and Active (VRRP-A)
 a. Secure privileged mode on Secondary ADC
b. show environment (all indicators must show OK and ON status)
c. show int brief (mgmt., 1, 2, 6, 7, ve20, ve40, ve88 must be all UP)
d. show vcs summary (Secondary ADC(local) = vMaster | Primary ADC(Remote)
=Blade)
e. show vrrp-a (Secondary ADC(local) = Active & Primary ADC(Peer) = Standby
f. show ip ospf neighbour (neighbours status must be Full)
g. show ip ospf redistributed (redistributed subnets will be seen)
h. show slb virtual server (Virtual Servers must be up)
i. show session (Total Sessions counter must be incrementing/decrementing.
Sessions from BNG and EPC is showing too)
j. Run some testing using FTTX Network to ensure services are not affected and fully
operational

8. Save the configurations for both Primary and Secondary ADC

a. Secure privileged mode on Primary, Secondary and VCS
b. Write memory all-partitions

9. Power-off Primary ADC

a. Secure privileged mode on Primary ADC
b. Shutdown

10. Physically remove Primary ADC from the rack and its cables (Power, Fiber and UTP)

11. Perform Fan Replacement procedure as per A10 Thunder Series 1040_1040S_940
Installation Guide.pdf document

12. Mount Primary ADC to the rack and connect back its cables (Power, Faber and UTP)

13. Power-up Primary ADC

14 Check Primary ADC Health and Status

a. secure privileged mode on Primary ADC
b. show environment (Ensure that Newly inserted Fan Module (FAN3) and other
FANS are operational
c. show vcs summary
d. show vrrp-a
e. show int brief
f. show ip ospf neighbour
g. show ip ospf redistributed
h. show slb virtual server
i. show session

15 Failover Master VCS Role from Secondary ADC to Primary ADC

a. Secure privileged mode on Secondary ADC
 b. conf t
c. vcs device 1
d. priority 200 (Revert original value (original value assumed to be 200). Ensure
that the value is higher than Secondary ADC under VSC configuration)
e. Alternately you cian run (use this if you use the alternate option stated on 3.e)
f. Secure privileged mode on Primary ADC
g. vcs master-take-over 220

16 Validate successful VCS failover to Primary ADC

a. Secure privileged mode on Primary ADC
b. show vcs summary (Primary ADC (local) = vMaster | Secondary ADC
(Remote) = Blade)
c. ping 172.30.0.5
d. ssh to VCS IP to ensure that Cluster Access is pointed to Primary ADC

17 Failover VRRP Primary Role from Secondary ADC to Primary ADC

a. Secure privileged mode on Primary ADC
b. conf t
c. vrrp-a vrid 1
d. device-context 1
e. blade-parameters
f. priority 200 (Revert original value (original value assumed to be 200). Ensure
that the value is higher than Secondary ADC under VRRP configuration)
g. Alternatively, you can run (user this only if you use the alternate option stated on
5.g)
h. Secure privileged mode on Primary ADC
i. conf t
j. vrrp-a force-self-standby disable

18 Validate successful VRRP failover to Primary ADC

a. Secure privileged mode on Primary ADC
b. show vrrp-a (Primary ADC(local) = Active & Secondary ADC(Peer) = Standby)
19 Validate Primary ADC functions now acting as Master (VCS) and Active (VRRP-A)
a. Secure privileged mode on Primary ADC
b. Show environment (all indicators must show OK and ON status)
c. Show int brief (mgmt., 1, 2, 6, 7, ve20, ve40, ve88 must be all up)
d. Show vcs summary (Primary ADC(local) = vMaster | Secondary ADC(Remote)
= vBlade)
e. Show vrrp–a (Primary ADC (local) = Active & Secondary ADC(Peer) =
Standby)
f. Show ip ospf neighbour (neighbour status must be Full)
g. Show ip ospf redistributed (redistributed subnets will be seen)
h. Show session (Total sessions from BNG and EPC is showing too)
i. Run some testing using FTTX Network to ensure services are not affected and
fully operational.
 20 Save the configurations for both Primary and Secondary ADC
a. Secure privileged mode on Primary, Secondary and VCS
b. write memory all

21 Done

Abort Procedure :
Any abnormalities seen while running the Corrective Procedure will automatically trigger this
procedure
1. Any changes made needs to be reverted the soonest
2. Netsis, GlobalNet and other personnel’s involve will do immediate investigation and plan
for actions
3. GlobalNet at full discretion will decide if the new plan will be accepted given time and
confidence that newly formulated plans will work

Success Criteria :
1. Replacement of Primary ADC FAN3 Module
2. Fully restoration of Services where Primary ADC is acting as Primary for VCS and VRRP-
A
3. FTTX and LTE customers are able to connect and disconnect connectivity without issues