You are on page 1of 10

CUDB_21 SC_2_1# ssh sc_2_2

Last login: Wed Aug 16 10:22:27 2017 from 192.168.7.152


CUDB_21 SC_2_2# ssh -p 2024 expert@10.202.4.150
expert@10.202.4.150's password:
Welcome to the DMX CLI
expert connected from 10.202.4.149 using ssh on blade_0_0

expert@blade_0_0 13:56:48> conf


Entering configuration mode private
[ok][2017-08-16 13:57:02]
expert@blade_0_0 13:57:02% set ManagedElement 1 Equipment 1 Shelf 0 Slot 9 Blade 1
administrativeState locked
[ok][2017-08-16 13:57:13]

[edit]
expert@blade_0_0 13:57:13% commit
[edit]
expert@blade_0_0 13:57:02% set ManagedElement 1 Equipment 1 Shelf 0 Slot 9 Blade 1
administrativeState unlocked
[ok][2017-08-16 13:57:13]

[edit]
expert@blade_0_0 13:57:13% commit
Commit complete.
[ok][2017-08-16 13:57:17]

expert@blade_0_0 14:06:17> exit


Connection to 10.202.4.150 closed.
CUDB_21 SC_2_2# ssh PL_2_5
PL_2_5:~ # df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 2.0G 443M 1.6G 22% /
root 2.0G 443M 1.6G 22% /
tmpfs 12G 0 12G 0% /dev/shm
shm 12G 0 12G 0% /dev/shm
192.168.0.100:/.cluster 62G 35G 23G 61% /cluster
PL_2_5:~ # exit
logout
Connection to PL_2_5 closed.
CUDB_21 SC_2_2# cp /cluster/nodes/5/etc/rpm.conf_FULL /cluster/nodes/5/etc/rpm.conf
CUDB_21 SC_2_2# cd /opt/ericsson/cudb/OAM/support/bin
CUDB_21 SC_2_2# ./cudbPartTool check -n 5

CUDB partitioning tool for EBS

-= Cluster filesystem analysis =-

Payload PL_2_5 report:


ERROR: Invalid primary partitions for /local and /local2 storage.

Done.
CUDB_21 SC_2_2# ./cudbPartTool rebuild -n 5

CUDB partitioning tool for EBS

-= Rebuilding payloads =-
Reset partition table in PL_2_5
WARNING: command "/bin/umount -f /local" in PL_2_5 exit with non-zero status.
WARNING: command "/bin/umount -f /local2" in PL_2_5 exit with non-zero status.
Building partitions in PL_2_5
Formatting partitions in PL_2_5

Done.
CUDB_21 SC_2_2# ./cudbPartTool check -n 5

CUDB partitioning tool for EBS

-= Cluster filesystem analysis =-

Payload PL_2_5 report:


WARNING: local storages not mounted.

Done.
CUDB_21 SC_2_2# cp /cluster/nodes/5/etc/rpm.conf_FULL /cluster/nodes/5/etc/rpm.conf
CUDB_21 SC_2_2# cluster reboot -n 5
Rebooting node 5 (PL_2_5)
Succeeded to execute /sbin/reboot on 5 (PL_2_5)
CUDB_21 SC_2_2# ssh PL_2_5
Last login: Wed Aug 16 16:15:25 2017 from sc_2_2
CUDB_21 PL_2_5# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 2.0G 1.3G 749M 64% /
root 2.0G 1.3G 749M 64% /
tmpfs 12G 696K 12G 1% /dev/shm
shm 12G 696K 12G 1% /dev/shm
192.168.0.100:/.cluster 62G 35G 23G 61% /cluster
/dev/sdb1 138G 1.9G 129G 2% /local
/dev/sdb2 138G 188M 131G 1% /local2
CUDB_21 PL_2_5# smartctl -a /dev/sdb
smartctl 5.39 2008-10-24 22:33 [x86_64-suse-linux-gnu] (openSUSE RPM)
Copyright (C) 2002-8 by Bruce Allen, http://smartmontools.sourceforge.net

Device: HP EG0300FBDBR Version: HPD7


Serial number: PQJPPSTB
Device type: disk
Transport protocol: SAS
Local Time is: Wed Aug 16 16:22:48 2017 CAT
Device supports SMART and is Enabled
Temperature Warning Enabled
Log Sense for temperature failed [scsi response fails sanity test]
SMART Health Status: OK

Log Sense for temperature failed [scsi response fails sanity test]
scsiGetStartStopData Failed [scsi response fails sanity test]
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 17517726645878784

Error counter log:


Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed
uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 20400 0 0 0 27898.585 0
write: 0 138294 0 138294 0 117260.621 0

Non-medium error count: 57


No self-tests have been logged
Long (extended) Self Test duration: 2462 seconds [41.0 minutes]

CUDB_21 SC_2_2# cmw-status node


safAmfNode=PL-5,safAmfCluster=myAmfCluster
AdminState=LOCKED-INSTANTIATION(3)
OperState=ENABLED(1)
CUDB_21 SC_2_2# cmw-node-unlock PL_2_5
CUDB_21 SC_2_2# cmw-status node
Status OK
CUDB_21 SC_2_2# cudbUnitDataBackupAndRestore -p 0 -n 21

CREATE PART
--------------------------------

creating backup on node 11

cudbManageStore stores to process: pl.

Starting Backup ...


Launching order Backup for pl in dsgroup 0.
Obtaining Mgm Information.
Trying backup on mgmt access 1, wait a moment ...
ndb_mgm 10.22.0.1 2290 -e "START BACKUP 999 WAIT COMPLETED"
..ok
BACKUP-999 renamed in PL_2_3 to /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-
08-16_16-31
BACKUP-999 renamed in PL_2_4 to /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-
08-16_16-31
BACKUP-999 renamed in PL_2_5 to /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-
08-16_16-31
BACKUP-999 renamed in PL_2_6 to /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-
08-16_16-31

Backup finished successfully for store pl.


Stores where order backup was successfully completed: pl.
cudbManageStore command successful.

DIR CREATE PART


--------------------------------

creating backup directory /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-


16_16-31 on node 21 on blade 10.22.0.3
creating backup directory /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-
16_16-31 on node 21 on blade 10.22.0.4
creating backup directory /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-
16_16-31 on node 21 on blade 10.22.0.5
creating backup directory /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-
16_16-31 on node 21 on blade 10.22.0.6

COPY PART
--------------------------------
copying backup files from node 11 blade
10.22.0.3:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31 to node 21
blade 10.22.0.3:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31
copying backup files from node 11 blade
10.22.0.4:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31 to node 21
blade 10.22.0.4:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31
copying backup files from node 11 blade
10.22.0.5:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31 to node 21
blade 10.22.0.5:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31
copying backup files from node 11 blade
10.22.0.6:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31 to node 21
blade 10.22.0.6:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31

RESTORE PART
--------------------------------

Restoring backup on node 21

cudbManageStore stores to process: pl.

Launching restore order in CUDB Node 21 to store pl in dsgroup 0.


Starting restore in CUDB Node 21 for store pl, backup path
/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_16-31, sql scripts path
/home/cudb/storageEngine/config/schema/pl/internal/restoreTempSql.
Keeping LDAPCounters stopped until restore have finished ....
Waiting for restore order(s) to be completed in CUDB Node 21 for stores : pl.
restore order finished successfully in CUDB Node 21 for store pl.
Resuming LDAPCounters.
Repopulating system monitor tables with data from configuration model (can take one
minute).
restore order(s) completed in CUDB Node 21 for stores : pl.
Stores where order restore was successfully completed: pl.
Closing connections for all blades of DSUnitGroup 0.
cudbManageStore command successful.

Performing cleanup
cudbUnitDataBackupAndRestore successfully ended

CUDB_21 SC_2_2# cudbManageStore --all --order status

cudbManageStore stores to process: pl ds1 (in dsgroup1) ds2 (in dsgroup2) ds3
(in dsgroup3) ds4 (in dsgroup4).

Store pl in dsgroup 0 is alive and reporting status ACTIVE.


Store ds1 in dsgroup 1 is alive and reporting status ACTIVE.
Store ds2 in dsgroup 2 is alive and reporting status ACTIVE.
Store ds3 in dsgroup 3 is alive and reporting status ACTIVE.
Store ds4 in dsgroup 4 is alive and reporting status ACTIVE.
cudbManageStore command successful.

CUDB_21 SC_2_2# cd /cluster/software/app_counters


CUDB_21 SC_2_2# chmod +x app_counters.pl
CUDB_21 SC_2_2# ./app_counters.pl -u

UDC HLR and HSS Applications Counters Installation, version Rev. E 4.0.6
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking Active OAM blade...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reading System Hosts info...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reading System config...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking Active Alarms...
[warn]
* Alarms exist in the node...Are you sure you want to continue ? (y/n)y
* Proceeding...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking the CUDB System Status..
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Please choose a system application counter uninstallation option:
* A. Uninstallation of HLR application counters
* B. Uninstallation of HSS application counters
* C. Uninstallation of All application counters
* X. Exit
* Please enter your choice(a/b/c/x):a
* Proceeding with HLR application counter uninstallation.

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Deleting previous counter installation crontab files..
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Deleting previous counter installation files...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reading Counters config files..
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* INFORMATION: Creation of the Pm Counter Job file for the active application
counters
* can be handled by either an Ericsson OSS-RC or this program!
* WARNING : Will an Ericsson OSS-RC used to handle the Pm Counter Job file for
application counters ? (y/n)y
* The Pm Counter Jobs file for the active application counters will be created by
OSS-RC!
[warn]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reloading Jobs..
* Stopping PmAgent in node 10.22.0.1 ... OK
* ESA PmAgent has been successfully stopped.
* Starting PmAgent in node 10.22.0.1 ... OK
* Stopping PmAgent in node 10.22.0.2 ... OK
* ESA PmAgent has been successfully stopped.
* Starting PmAgent in node 10.22.0.2 ... OK

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking Mastership of PLDB...
[ ok ]
* This node DOES NOT have the Mastership of the PLDB !!!
* Proceeding with Slave uninstallation...
* Deleting procedures in DS1_0..
[ ok ]
* Deleting procedures in DS1_1..
[ ok ]
* Deleting procedures in DS2_0..
[ ok ]
* Deleting procedures in DS2_1..
[ ok ]
* Deleting procedures in DS3_0..
[ ok ]
* Deleting procedures in DS3_1..
[ ok ]
* Deleting procedures in DS4_0..
[ ok ]
* Deleting procedures in DS4_1..
[ ok ]
* Deleting procedures in PL0..
[ ok ]
* Deleting procedures in PL1..
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Clearing Application Counter Alarms, 0 alarms found
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
CUDB_21 SC_2_2# ./app_counters.pl -i

UDC HLR and HSS Applications Counters Installation, version Rev. E 4.0.6

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking Active OAM blade...
[ ok ]
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reading System Hosts info...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reading System config...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking Active Alarms...
[warn]
* Alarms exist in the node...Are you sure you want to continue ? (y/n)y
* Proceeding...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking the CUDB System Status..
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Please choose a system application counter installation option:
* A. Installation of HLR application counters
* B. Installation of HSS application counters
* C. Installation of All application counters
* X. Exit
* Please enter your choice(a/b/c/x):a
* Proceeding with HLR application counter installation.

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Copying Files...HLR...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Checking Mastership of PLDB...
[ ok ]
* This node DOES NOT have the Mastership of the PLDB !!!
* Proceeding with Slave installation...
* Creating procedures in DS1_0...
[ ok ]
* Creating procedures in DS1_1...
[ ok ]
* Creating procedures in DS2_0...
[ ok ]
* Creating procedures in DS2_1...
[ ok ]
* Creating procedures in DS3_0...
[ ok ]
* Creating procedures in DS3_1...
[ ok ]
* Creating procedures in DS4_0...
[ ok ]
* Creating procedures in DS4_1...
[ ok ]
* Creating procedures in PL0...
[ ok ]
* Creating procedures in PL1...
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reading Counters config files..
[ ok ]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Please choose an installation option
* A. Installation of HLR counters (Includes all HLR default and regional
counters)
* B. Installation of HLR default counters (Excludes all HLR regional counters)
* C. Installation of HLR default counters and custom selection of HLR regional
counters
* D. Custom installation of HLR default and regional counters
* X. Exit
* Please enter your choice(a/b/c/d/x):a

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* INFORMATION: Creation of the Pm Counter Job file for the active application
counters
* can be handled by either an Ericsson OSS-RC or this program!
* WARNING : Will an Ericsson OSS-RC used to handle the Pm Counter Job file for
application counters ? (y/n)y
* The Pm Counter Jobs file for the active application counters will be created by
OSS-RC!
[warn]

-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------
* Reloading Jobs..
* Stopping PmAgent in node 10.22.0.1 ... OK
* ESA PmAgent has been successfully stopped.
* Starting PmAgent in node 10.22.0.1 ... OK
* Stopping PmAgent in node 10.22.0.2 ... OK
* ESA PmAgent has been successfully stopped.
* Starting PmAgent in node 10.22.0.2 ... OK
CUDB_21 SC_2_2# cudbSystemStatus -R

Execution date: Wed Aug 16 17:42:09 CAT 2017

Checking Replication Channels in the System:


Node | 11 | 21
====================
PLDB ___|__M__|__S1_
DSG 1 __|__M__|__S2_
DSG 2 __|__M__|__S2_
DSG 3 __|__M__|__S2_
DSG 4 __|__M__|__S2_

CUDB_21 SC_2_2# cudbUnitDataBackupAndRestore -d 4 -n 21


CREATE PART
--------------------------------

creating backup on node 11

cudbManageStore stores to process: ds4 (in dsgroup4).

Starting Backup ...


Launching order Backup for ds4 in dsgroup 4.
Obtaining Mgm Information.
Trying backup on mgmt access 1, wait a moment ...
ndb_mgm 10.22.0.1 2375 -e "START BACKUP 999 WAIT COMPLETED"
..ok
BACKUP-999 renamed in PL_2_13 to /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-
08-16_17-30
BACKUP-999 renamed in PL_2_14 to /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-
08-16_17-30

Backup finished successfully for store ds4.


Stores where order backup was successfully completed: ds4.
cudbManageStore command successful.

DIR CREATE PART


--------------------------------

creating backup directory /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-


16_17-30 on node 21 on blade 10.22.0.13
creating backup directory /local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-
16_17-30 on node 21 on blade 10.22.0.14

COPY PART
--------------------------------

copying backup files from node 11 blade


10.22.0.13:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_17-30 to node 21
blade 10.22.0.13:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_17-30
copying backup files from node 11 blade
10.22.0.14:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_17-30 to node 21
blade 10.22.0.14:/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_17-30
Checking Replication Channels in the System:
Node | 11 | 21
=====Checking Replication Channels in the System:
Node | 11 | 21
=====
RESTORE PART
--------------------------------

Restoring backup on node 21

cudbManageStore stores to process: ds4 (in dsgroup4).

Launching restore order in CUDB Node 21 to store ds4 in dsgroup 4.


Starting restore in CUDB Node 21 for store ds4, backup path
/local/cudb/mysql/ndbd/backup/BACKUP/BACKUP-2017-08-16_17-30, sql scripts path
/home/cudb/storageEngine/config/schema/ds/internal/restoreTempSql.
Waiting for restore order(s) to be completed in CUDB Node 21 for stores : ds4.
CUDB_21 SC_2_1# cudbDsgMastershipChange -d --dsg 2
Processing DSG mastership change...
Replication checking...
Replication in DSG2(Node=21--Chan=2).... OK -- Delay = 0
Warning: This step could stuck forever if the allowed synchronization delay between
the current master replica and the future master replica is never reached. If you
experience that the command takes too long to finish you can stop the command with
CTRL+C. You would have more chances to finish the command execution by slowling
down the traffic in the master or using the option --time with a higher value.
| Distance to master is 100. Proceeding...

Putting the master in maintenance...


Waiting for mastership switchover...
Putting the original master back to ready mode...

Master replica on node: 11


Distance to master replica: 0 seconds
cudbDsgMastershipChange: Success.
DSG 2 master replica moved from node 11 to 21 successfully

CUDB_21 SC_2_1# cudbSystemStatus -R

Execution date: Thu Aug 17 15:11:00 CAT 2017

Checking Replication Channels in the System:


Node | 11 | 21
====================
PLDB ___|__M__|__S1_
DSG 1 __|__M__|__S2_
DSG 2 __|__S1_|__M__
DSG 3 __|__M__|__S2_
DSG 4 __|__M__|__S2_

CUDB_21 SC_2_1# cudbDsgMastershipChange -d --dsg 4


Processing DSG mastership change...
Replication checking...
Replication in DSG4(Node=21--Chan=2).... OK -- Delay = 0
Warning: This step could stuck forever if the allowed synchronization delay between
the current master replica and the future master replica is never reached. If you
experience that the command takes too long to finish you can stop the command with
CTRL+C. You would have more chances to finish the command execution by slowling
down the traffic in the master or using the option --time with a higher value.
\ Distance to master is 200. Proceeding...

Putting the master in maintenance...


Waiting for mastership switchover...
Putting the original master back to ready mode...

Master replica on node: 11


Distance to master replica: 0 seconds
cudbDsgMastershipChange: Success.
DSG 4 master replica moved from node 11 to 21 successfully