You are on page 1of 43

Security Level: Internal

M2000 Management and Maintenance Operations

www.huawei.com

HUAWEI Confidential

This document mainly describes the common M2000 maintenance operations, including the M2000 operation commands and the Sybase operation commands.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 2

This helps you to familiarize yourself with:


M2000 common operations and common logs M2000 common maintenance commands

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 3

References
M2000 Administrator Guide M2000 Software Installation Guide OMCAtoStar User Manual

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 4

Chapter 1 Solaris Maintenance Commands Chapter 2 Sybase Maintenance Commands Chapter 3 M2000 Maintenance Commands Chapter 4 M2000 Logs Chapter 5 M2000 Log Collection Tool and Health Check Tool

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 5

Solaris Maintenance Commands


Command ifconfig a ping netstat an netstat rn df -k du k prstat Display the IP address Test if a IP or host is reachable Display the ports status of server Display the routes of server Display the amount of disk space Display the space occupied by files and directory Display the CPU and memory occupied by process Description

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 6

ifconfig

Function Configure or display the network interface parameters. We mainly use it to display the IP address.

Usage ifconfig -a
root@osssvr # ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.10.10.100 netmask ffffff00 broadcast ether 0:14:4f:90:45:8e

Port status, 10.161.226.255 RUNNING is normal

PORT NAME:

Port IP addr Port MAC addr

1 Port name vary with server model 2 IP addresses assigned to one port will be displayed as follows: bge0, bge0:1, bge0:2,
HUAWEI TECHNoLOGIES Co., Ltd.

Network mask, in hexadecimal, eg: ffffff00 here means 255.255.255.0


Page 7

HUAWEI Confidential

ping

Function Test if the IP address is reachable Usage ping [s] <ip_addr>


root@m2k-server5 # ping -s 10.10.5.207 PING 10.10.5.207: 56 data bytes 64 bytes from 10.10.5.207: icmp_seq=0. time=3.78 ms 64 bytes from 10.10.5.207: icmp_seq=1. time=1.97 ms 64 bytes from 10.10.5.207: icmp_seq=2. time=1.89 ms ^C ----10.10.5.207PING Statistics---6 packets transmitted, 6 packets received, 0% packet loss round-trip (ms) min/avg/max/stddev = 1.89/3.53/4.93/1.3

The ping wont stop untill you press ctrl+c


HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 8

Netstat -an

Function

Display the port status of server

Usage netstat an

Normal netstat an output a lot information, we can use grep to filter specified port number or IP. Eg, normal NE use port 6001 to report alarms to M2000, display the status of port 6001 with command:
root@osssvr # netstat -an|grep 6001 10.10.10.100.6001 *.* 5840 0 0 49152 0 49640 0 LISTEN 0 ESTABLISHED 10.10.10.100.37113 10.10.5.207.6001

Local IP

Local port

Remote IP (NE IP address)

Remote port (NE port)


HUAWEI Confidential
Page 9

Port Status ESTABLISHED is normal

HUAWEI TECHNoLOGIES Co., Ltd.

Netstat -rn

Function

Display the routes of server

Usage: netstat rn
root@osssvr # netstat -rn Routing Table: IPv4 Destination 10.161.224.145 10.161.226.0 224.0.0.0 default 127.0.0.1 Gateway 10.161.226.1 10.161.226.100 10.161.226.100 10.161.226.1 127.0.0.1 Flags UGH U U UG UH Ref 1 1 1 1 2 Use 0 165 eri0 0 eri0 1502 3929 lo0 Interface -------------------- -------------------- ----- ----- ------ ---------

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 10

df -k

Function

Display the disk space

Usage: df -k
root@osssvr # df -k Filesystem /dev/md/dsk/d100 /dev/md/dsk/d112 kbytes used avail capacity 85% 82% Mounted on / /export/home 17146994 14394586 2580939 13574307 10957280 2481284

Total space (KB) Occupied space(KB)

Available space (KB) Usage Ratio

Directory
/ is root directory on local disk /export/home is diskarray in general

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 11

du -k

Function

Display the space occupied by directory and files

Usage: du k <path>

Display spaces occupied by files and each subdirectory of <path>, the unit is KB. This command is used to find out the largest directories when disk usage ratio is too high
root@m2k-server5 # du -k /training 24 204802 /training/files /training

The above output shows /training occupies total 204820KB space, its subdirectory /training/files occupies 24KB space. We can conclude that the files in /training occupy most of the 204802KB space which /training occupy.
HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 12

prstat

Function

Display the CPU and memory occupied by processes

Usage: prstat [interval [count]]


interval: the sampling interval in seconds count: number of times that the reports are repeated

By default, prstat reports statistics every 5 seconds until ctrl+c press.


root@m2k-server5 # prstat PID USERNAME 608 dbuser 1953 root 1974 root 961 root SIZE 219M 64M RSS STATE 127M sleep 21M sleep PRI NICE 59 59 59 59 0 0 0 0 TIME CPU PROCESS/NLWP 1586M 1563M sleep 4:46:37 4.3% dataserver/39 0:19:33 0.5% imapsvcd/120 0:10:41 0.5% imapsvcd/38 0:03:37 0.3% ResourceMonitor/4

PID of120 processes, porcess Total:

43M 6536K sleep

Use ps ef|grep pid to find out the porcess name

2323 lwps, Total averages: 1.43, 1.62, 1.66 load virtual

memory size of the process

The percentage of recent CPU time used by the process

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 13

Chapter 1 Solaris Maintenance Commands Chapter 2 Sybase Maintenance Commands Chapter 3 M2000 Maintenance Commands Chapter 4 M2000 Logs Chapter 5 M2000 Log Collection Tool and Health Check Tool

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 14

Sybase Maintenance Commands


Commands showserver isql select @@version sp_helpdb Description Display the status of Sybase database server Start SQL script execution tool Display the Sybase version Display the status of Sybase database

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 15

showserver

Function

Display the status of Sybase database server

Usage: showserver
-bash-3.00$ showserver UID dbuser dbuser PID 608 919 PPID 606 917 C STIME TTY TIME CMD 298:25 /opt/sybase/ASE0:00 /opt/sybase/ASE26 19:29:04 ? 0 19:31:04 ?

15_0/bin/dataserver -sSYB -d/export/home/sybdev/master.dat -e/o 15_0/bin/backupserver -SSYB_back -e/opt/sybase/ASE-15_0/install

1 Must login as dbuser to excute Sybase command, DO NOT use root user ever. The prompt of root is like root@hostname #, such as root@osssvr # The prompt of dbuser is like bash-3.00$ 2 Check if both of dataserver and backupserver are running
HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 16

isql

Function

Start SQL script execution tool

Usage: isql

S<DB server name> -U<username> -P<password>

The default DB server name of M2000 is SYB Super user of Sybase is sa default password is emsems For example
-bash-3.00$ isql -SSYB -Usa -Pemsems 1> 2> 3>

If success, prompt 1> is displayed. Input SQL script after prompt. go Execute command quit or exit quit isql
HUAWEI Confidential
Page 17

HUAWEI TECHNoLOGIES Co., Ltd.

Display Sybase Version

Execute select @@version in isql

-bash-3.00$ isql -SSYB -Usa -Pemsems 1> select @@version 2> go

-------------------------------------------------------------------------Adaptive Server Enterprise/15.0.2/EBF 15162 ESD#1 ONE-OFF/P/Sun_svr4/OS 5.8/ase 1502/2493/64-bit/FBO/Mon Oct 15 16:15:21 2007

Sybase Version

Sybase Patch

(1 row affected)
HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 18

sp_helpdb

Function

Display the status of Sybase database

Usage: sp_helpdb [dbname]


Execute in isql Without parameter dbname, display the status of all database: check if the status is offline With dbname, display the specified database space, for example:

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 19

Start Sybase

Single System

Login as dbuser, execute the following commands:

-bash-3.00$ cd /opt/sybase/ASE-15_0/install -bash-3.00$ ./startserver -f ./RUN_SYB -bash-3.00$ nohup ./startserver -f ./RUN_SYB_back >/dev/null

HA System

Login as root, execute the following command on active node or standby node:

root@osssvr-1 # scswitch -e -j sybase_rs


HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 20

Stop Sybase

Single System

Login as dbuser, execute the following commands:

-bash-3.00$ isql SSYB -Usa P[password of sa] 1> shutdown SYB_BACKUP 2> go 1> shutdown 2> go

HA System

Login as root, execute the following command on active node or standby node:

root@osssvr-1 # scswitch -n -j sybase_rs

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 21

Chapter 1 Solaris Maintenance Commands Chapter 2 Sybase Maintenance Commands Chapter 3 M2000 Maintenance Commands Chapter 4 M2000 Logs Chapter 5 M2000 Log Collection Tool and Health Check Tool

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 22

M2000 Maintenance Commands


Commands
start_svc

Description
Starting the entire M2000 system, including the system process and the service process (This command is not applicable to the HA system.)

stop_svc

Stopping the service process of the M2000 network management system (This command is not applicable to the HA system.)

kill_svc svc_adm displayVersion scswitch

Forcibly stopping the process of the M2000 network management system Managing M2000 services, such as starting, stopping, and querying services Display version of M2000 and installed mediation Managing the system resource of HA system, such as starting/stop M2000, start/stop Sybase, query M2000/Sybase

start_daem

Starting the M2000 system daemon process (This command is not applicable to the HA system.)

stop_daem

Starting the M2000 system daemon process (This command is not applicable to the HA system.)

daem_ps

Display the status of M2000 system daemon processes

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 23

svc_profile.sh
Function of the Scripts You need to run the svc_profile.sh script before running the M2000 commands. This script is used to set users working environment such as searching paths through commands and dynamically searching paths through the database.

Operation Methods Manual operation and automatic operation Manual Operation Taking the default M2000 installation directory /opt/OMC as an example, the method of manually running the script is as follows: -bash-3.00$ . /opt/OMC/svc_profile.sh (There is a space between . and /)

Automatic Operation After the system is correctly installed, this script is added to the .profile file of user omcuser and user root. If you log in to the server as user omcuser or user root, the system automatically runs the script.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 24

Start M2000

Single System

Login as root, execute the following commands:

root@osssvr-1 # start_svc

HA System

Login as root, execute the following command on active node or standby node:

root@osssvr-1 # scswitch -e -j ossapp_rs

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 25

Stop M2000

Single System

Login as root, execute the following commands: Stop service process of M2000 system:
root@osssvr-1 # stop_svc

Stop daemon process of M2000 system:


root@osssvr-1 # stop_daem

Stop the TAO process

root@osssvr-1 # stop_tao_services

HA System

Login as root to active server or standby server, execute the following command on active node or standby node:

root@osssvr-1 # scswitch -n -j ossapp_rs


HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 26

Display Status of M2000 Services


Usage: svc_adm cmd status Notice:

Applicable to both of single and HA system


Process name Service name
One process may ] include several services

root@osssvr # svc_adm -cmd status Host: 10.10.10.1

Service Agent: 3rdTool_agent [1 service(s)] pid: 11597 3rdToolService [running ] Service Agent: pm_agent [1 service(s)] pid: 11685 PMService [running . . .

[All Services: 45 ] [Running: 45 ] [No License: 0 ] [Not Running: 0 ]

Service status

Statistic s
HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 27

Managing the M2000 Services

Stop specified service: svc_adm cmd stopsvc [ServiceName]


root@osssvr # svc_adm -cmd stopsvc PMService

This command will stop the service specified by parameter ServiceName, and the services that depend on it.

Start specified service: svc_adm cmd startsvc [ServiceName]


root@osssvr # svc_adm -cmd stopsvc PMService

This command will start the service specified by parameter ServiceName, and the services that it depend on.

Start all services: svc_adm cmd start

If you use svc_adm to stopped a service, its suggested to use this command to start the stopped services.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 28

Display Version of M2000 and Mediation

Usage: displayVersion

root@osssvr-1 # displayVersion ------------------------OMC Version-------------------------Product Name: Version: Release Date: iManagerM2000 iManagerM2000V200R006ENGC01B060 11/24/07

-----------------Installed Mediation Version----------------<DBS3800> Match Version : iManagerM2000_DBS3800_MATCH_ENG_V200R005C01B035 NE Version is : DBS3800V100R008ENGC01B071

<RNC> Match Version : iManagerM2000_RNC_MATCH_ENG_V200R005C01B206 NE Version is : BSC6810V200R009ENGC01B072 root@osssvr-1 #


HUAWEI TECHNoLOGIES Co., Ltd. HUAWEI Confidential
Page 29

Sun Cluster3.0 Commands for Dual Server


Command
scstat scswitch -z -g oss_rg -h osssvr-2

Description
Querying the status of the cluster Switch to osssvr-2 (starting the oss_rg resource group on the osssvr-2 host).

scswitch -e -j sybase_rs

Activating the sybase_rs resource and activating the cluster to monitor the Sybase.

scswitch -n -j sybase_rs

Deactivating the sybase_rs resource and stopping the cluster from monitoring the Sybase.

scswitch -e -j ossapp_rs

Activating the ossapp_rs resource and activating the cluster to monitor the M2000.

scswitch -n -j sybase_rs

Deactivating the ossapp_rs resource and activating the cluster to monitor the M2000.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 30

Chapter 1 Solaris Maintenance Commands Chapter 2 Sybase Maintenance Commands Chapter 3 M2000 Maintenance Commands Chapter 4 M2000 Logs Chapter 5 M2000 Log Collection Tool and Health Check Tool

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 31

Collecting Information about the M2000 System


Solaris Log Sybase Log M2000 Server Log M2000 Client Log Core File

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 32

Solaris Log

Application Scenarios

Use the message log to familiarize yourself with the starting and stopping process of the system. The hardware or other devices incur faults, such as the disk light is abnormal or the CPU and the memory cannot be found.

Brief Introduction of the Log


The log exists under the var/adm/messages directory. The log records the information about the Sybase starting and stopping process, the operating status of the hardware and software, and the main error information. Under the /var/adm/ directory, there are messages, messages.0, messages.1, messages.2 , and messages.3. Where messages saves the latest logs. Other message files are cyclically recovered.

# ls -al /var/adm/messages* -rw-r--r-- 1 root -rw-r--r-- 1 root -rw-r--r-- 1 root -rw-r--r-- 1 root -rw-r--r-- 1 root

root root root root root

311912 Dec 27 10:02 messages 7840 Nov 19 15:34 messages.0 101162 Nov 12 16:26 messages.1 8882 Nov 5 16:07 messages.2 170931 Oct 29 10:03 messages.3

Obtaining and Checking the Files You can download the message log to the local computer through FTP and open it in .txt format. You can check the files through the unix files such as more and cat.
HUAWEI Confidential
Page 33

HUAWEI TECHNoLOGIES Co., Ltd.

Sybase Log

Application Scenarios

Use this log to familiarize yourself with the process of starting and stopping the Sybase. The Sybase incurs faults, such as the Sybase cannot be started or errors occur during the operation.

Brief Introduction to the Log


Sybase12 version: The log exists under the /opt/sybase/ASE-12_0/install/ directory. Sybase15 version: The log exists under the /opt/sybase/ASE-15_0/install/ directory. The SYB.log and SYB_back.log files record the process of starting and stopping the Sybase and the main error information.

Obtaining and Checking Files


You can download the log to the local computer through FTP and open it in .txt format. You can check the files through the unix files such as more and cat.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 34

M2000 Server Log trace

Application Scenarios

Use this log to familiarize yourself with the operating status of the M2000 services and the information exchange between the services. This log is required when the M2000 is faulty.

M2000 V2R3/V2R5 Current Log /export/home/omc/var/logs Log file name


iMAP.root.trace

M2000 V2R6

Each log file corresponds to one process , such as: iMAP.cmserver_agent.trace iMAP.pm_agent.trace iMAP.med_agent.trace

History trace

/export/home/omc/var/logs/tracebak/ If log file is larger than 50 MB, the system automatically compresses it and transfers it to the history trace directory The file name contains detailed date and time, which refers to the time of the last log in the trace file, such as: iMAP.monitor_agent.trace.20080204122937.tar.gz

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 35

M2000 Client Log

Application Scenarios

Use this log to familiarize yourself with the operating status of the M2000 client and the operation records on the client. The log is required when faults occur to the M2000 (especially when some problems occur only on the specified client).

Brief Introduction to the Log

The log exists under the .\client\tracefile\ directory. For example, D:\iManagerM2000V200R005C01B024\client\tracefile The file name contains the detailed time, which refers to the recorded time of the first log. For example, DebugTrace_2007-08-27_16#21#20#991.txt

Obtaining and Checking Files

Assume that the M2000 is installed in the D:\iManagerM2000Client directory, then the trace information about the foreground is saved in the D:\iManagerM2000Client\client\tracefile directory.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 36

Core Files
Application Scenarios

Some services are abnormally killed and the core files are generated. The alarm console is abnormally terminated or alarms of restarting occur.

Brief Introduction to the Log


The log exists under the /export/home/omc/var/logs/ directory. The logs comprise the core files of all the M2000 services and the core files generated by the system The file name contains the detailed time, which refers to the recorded time of the first log.

For example, core.imapsvcd.0.11539.1196415653

Obtaining and Checking Files

Collect related information about the core files.

# pmap core.imapsvcd.0.22531.1086 > core.imapsvcd.0.22531.1086.mapinfo # pstack core.imapsvcd.0.22531.1086 | c++filt > core.imapsvcd.0.22531.1086.stack # pflags core.imapsvcd.0.22531.1086 > core.imapsvcd.0.22531.1086.pflags # gtar cvfz core.stack.tar.gz core.xxx.xxx.xxx.stack

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 37

Chapter 1 Solaris Maintenance Commands Chapter 2 Sybase Maintenance Commands Chapter 3 M2000 Maintenance Commands Chapter 4 M2000 Logs Chapter 5 M2000 Log Collection Tool and Health Check Tool

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 38

M2000 Log Collection Tool and Health Check Tool


Tool Default Installation Sun Explorer Basic Solaris Basic Sybase Basic M2000 Information Information Information Information M2000 Trace

Sun Explore Client information collecting tool

Yes Yes

Yes No

Yes Yes

No Yes

No Yes

No Yes

OMCAutoStar No

Optional

Yes

Yes

Yes

No

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 39

Sun Explorer

Application Scenarios

The Sun Explorer is provided to troubleshoot the faults of the Sun server. The faults comprise hardware faults and system faults.

Brief Introduction to the Log

Collect the configuration information and operating status of the server hardware and the Solaris. The scripts exist under the /opt/SUNWexplo/bin/explorer directory. By default, the collected results are saved under the /opt/SUNWexplo/output directory. For example, explorer.837cdded.m2k-server5-2008.01.03.08.31.tar.gz 837cdded refers to the host ID of the server. m2k-ser5 refers to the host name of the server. 2008.01.03.08.31 refers to the time that the results are generated.

Obtaining and checking files


Run the collectinfo script to collect the information about the Sun Explorer. Manually run the explorer script to separately collect the information about the Sun Explorer.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 40

M2000 Client Log Information Collecting Tool

Opening the Interface Program > iManager M2000 client > M2000 log information collecting tool Configuring Related Information Enter the IP address of the M2000 server. Enter the port number. Enter the FTP user name and the password of the destination server. Obtaining and Checking Files By default, the files are saved in the M2000 client installation directory D:\R200R006B060CHS\client\diagnosis\collected files.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 41

OMCAutoStar
Application Scenarios

It mplements the function of immediate and periodic health check. In addtion, it automatically analzyes the collected package about the health check information. On the basis that the Sun Explorer automatically runs, it can collect other log information about the operating system, Sybase, and applications related to the M2000. This offers the firsthand information for locating problems when the system incurs faults.

Detect and clear some useless M2000 data.


Download from support.huawei.com:

How to get OMCAutoStar tool?

Documentation Core Network Product Line Wireless-OSS iManager M2000-II Technical Guide Quality Check Guide

How to use OMCAutoStar tool?

Follow the user manual

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 42

Precautions for the M2000 Engineers

Record the logs.

Use the log function of the CRT to automatically record all the operations to files. The delivered M2000 commercial servers are without the KVMs. Do not edit a file by using the graphic terminal software.

Perform remote operations rather than local operations.

Use character terminal rather than graphic terminal.

Make the serial port cable ready. Perform serial port operations rather than network port operations.

When performing operations that cannot be rolled back and consume a long time such as upgrading the M2000 version, perform operations through the serial ports. This avoids problems because of the sudden power-off of the local PC or the LAN Switch.

The serial port operations refer to the operations performed through the maintenance serial port of the host or the maintenance network ports. The network port here refers to the network port of the operating system.

HUAWEI TECHNoLOGIES Co., Ltd.

HUAWEI Confidential

Page 43