UNIX/LINUX TECH NOTES: Troubleshooting a high system CPU usag...

1 of 5

More

http://honglus.blogspot.com/2010/01/troubleshooting-high-system-cpu-u...

Next Blog

Create Blog

W E D N E S D AY, J A N U A R Y 1 3 , 2 0 1 0

Sign In

V I S I TO R

Troubleshooting a high system CPU usage issue on

Linux/Solaris
LABELS

A Linux server has high %system CPU usage, following are steps to find the r

Linux (86)

Vmstat show %system CPU usage is high.

Scripting (33)

# vmstat 2

Virtualization (19)
NMS (16)

procs -----------memory---------- ---swap-- -----io---- --system-r

swpd

so

bi

buff

cache

si

0 191420

8688

35780

0 124468

9208

98020

0 15626

2074 1195

188

0 110716

9316 110996

3268

4144 1366

84

97048

9416 122272

2818 11855 1314

109

Tips/Tricks (10)

80476

9544 137888

3908

2786 1272

172

Windows (9)

72860

9612 145848

1930

0 1193

141

RHCE/RHCA (8)

74300

9620 145860

6 1208

67

75680

9620 145860

6929 1364

101

bo

in

cs u

Server Provisioning (15)

free

0 1006

31

Performance (13)
Solaris (13)
Storage (13)
Troubleshooting (12)

Linux Internals (5)

Apps (4)
Security (4)

Lets run mpstat to show more detailed CPU usage,it showed CPU was b

Cluster (3)
Hardware (3)

# mpstat 2
Linux 2.6.18-92.el5 (centos-ks)

01/14/2010

02:03:50 AM

CPU

%user

%nice

%sys %iowait

02:04:04 AM

all

1.33

0.00

41.78

02:04:06 AM

all

0.00

0.00

02:04:08 AM

all

0.00

0.00

02:04:10 AM

all

0.00

02:04:12 AM

all

02:04:14 AM

BLOG ARCHIVE

2014 (4)

%irq

%soft

0.00

0.44

3.56

8.04

38.69

29.65

23.62

2012 (22)

8.70

30.43

27.54

28.50

2011 (37)

0.00

5.47

46.77

27.36

20.40

0.50

0.00

6.47

63.18

19.40

10.45

all

1.01

0.00

6.53

62.31

21.11

9.05

02:04:16 AM

all

0.00

0.00

8.04

26.63

43.72

21.61

02:04:18 AM

all

0.00

0.00

1.50

0.00

0.00

0.50

Use sar to find out which interrupt number was culprit. #9 was the high

2013 (8)

2010 (27)
December (3)
October (1)
September (4)
August (3)

7/26/2016 9:39 PM

UNIX/LINUX TECH NOTES: Troubleshooting a high system CPU usag...

2 of 5

# sar -I XALL

http://honglus.blogspot.com/2010/01/troubleshooting-high-system-cpu-u...

July (3)

2 10

02:07:10 AM

INTR

intr/s

June (2)

02:07:12 AM

992.57

March (4)

02:07:12 AM

0.00

February (3)

02:07:12 AM

0.00

02:07:12 AM

0.00

January (4)

02:07:12 AM

0.00

02:07:12 AM

0.00

02:07:12 AM

0.00

02:07:12 AM

0.00

02:07:12 AM

0.00

02:07:12 AM

350.50

Benchmarking disk
IO accurately by
avoiding file c...
Double network
throughput by
tuning network
parame...
Change MTU for
Solaris on e1000g

[ Solaris equivalent command]

interface

Solaris# intrstat 2

device |

cpu0 %tim

cpu1 %tim

-------------+-----------------------------bge#0 |

cpqary3#0 |

0.0
0

128
0.0

0.6
14

0.0

Troubleshooting a
high system CPU
usage issue on L...
2009 (33)

# cat /proc/interrupts
S U B S C R I B E TO

CPU0
0:

702980

XT-PIC

timer

1:

439

XT-PIC

i8042

2:

XT-PIC

cascade

6:

XT-PIC

floppy

8:

XT-PIC

rtc

9:

14464

XT-PIC

12

XT-PIC

eth0

12:

400

XT-PIC

i8042

14:

6091

XT-PIC

ide0

15:

22

XT-PIC

ide1

LOC:

700623

ERR:

MIS:

Comments

FOLLOWERS

acpi, eth2

11:

NMI:

Posts

[ OpenSolaris equivalent command ]

Solaris#echo ::interrupts | mdb k

SEARCH THIS BLOG

Native Solaris has to search the interrupt from output of prtconf

Solution:
When the card transmits or receives a frame, the system must be notifi

POPULAR POSTS

7/26/2016 9:39 PM

UNIX/LINUX TECH NOTES: Troubleshooting a high system CPU usag...

3 of 5

http://honglus.blogspot.com/2010/01/troubleshooting-high-system-cpu-u...

Interrupt Coalescence essentially means that the card interrupts the system a

oubleshooting a high
system CPU usage issue
on Linux/Solaris

you can enable adaptive moderation ( Adaptive RX: off TX: off) to let system

Linux server has high

%system CPU usage,

A interrupt is generated by the card to the host when either frame counter or

following are steps to find

the root cause of the issue
and how to resolve it.

RX for example:

Vmstat show %syste...

Timer counter in microseconds: rx-usecs/rx-usecs-irq

Frames counter:rx-frames/rx-frames-irq

rce puppet agent to

regenerate certificate
request

# A sample output with default values.

# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: off

TX: off

stats-block-usecs: 999936

puppet agents certificate

is accidentally revoked or
deleted, you can force
agent to regenerate
certificate request. In
general, it is...

sample-interval: 0
pkt-rate-low: 0

eate GPT partition for LVM

pkt-rate-high: 0

using parted tool

rx-usecs: 18

aditional MBR(MSDOS)
disk label has limitation of

rx-frames: 6

2^32 (2TiB) in capacity

rx-usecs-irq: 18

and 15 in partition

rx-frames-irq: 6

numbers(including logical
partitions), while...

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 80
tx-frames-irq: 20

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0

HCS(Red Hat Cluster Suite)

quorum disk
he last post " RHCS I/O
fencing " is about dealing
with split-brain situation,
in which cluster members
lost heartbeat
communicat...

tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0

ow to generate core dump

for an application in Linux

tx-usecs-high: 0

order to troubleshoot
application issue, a

tx-frame-high: 0

process content in

<>

memory can be dumped

[ Solaris equivalent command]

Varying on driver, Find out the driver's capability.

to a file and then the file

can be analysed with
debug...

7/26/2016 9:39 PM

UNIX/LINUX TECH NOTES: Troubleshooting a high system CPU usag...

4 of 5

http://honglus.blogspot.com/2010/01/troubleshooting-high-system-cpu-u...

Solaris#ndd -get /dev/e1000g0 \? | egrep interrupt |intr

uthenticate Linux users by

The Value shoud be set in driver conf file:

Windows AD:

Solaris#/platform/`uname -m`/kernel/drv/*.conf

LDAP+Kerberos or
Winbind method

Alternative Workaround:

uthenticating Linux users

I couldn't config Interrupt Coalescence because virtual machine NIC didn't sup

by Windows AD has

You don't need to care Interrupt Coalescence if your CPU resource is

become popular in many

abundant, But for high load NFS/CIFS/ISCSI/ NAS servers, it is very useful.

organizations for the

convenience of centralized

POS TE D BY H ON GLUS AT 4: 40 PM

account management. It

LABELS: LINUX, PE RF ORM AN CE , SO LA RIS , TR OUB L ESHOOTING

...
se script to fetch URL
protected by NTLM

3 COMMENTS:

authentication.
indows IIS server use

Vinay January 1, 2014 at 12:37 AM

NTLM authentication, the

thank you , that helped

following show three

methods -wget, curl, and

Reply

Perl- to download URL

protected by NTLM. wget
Raul Moisi February 16, 2014 at 1:20 AM
What about the case when %sys is high and the others
not(except %user which is also high in my case)?

egular expression in
condition statement, the
ksh and bash examples

Reply
Replies
Alexander December 4, 2014 at 10:52 PM
+1
mpstat 2
02:43:19 PM CPU %user %nice
%soft %steal %idle intr/s
02:43:21 PM all 4.24 0.00 50.12
45.14 1050.99
02:43:23 PM all 4.24 0.00 49.63
46.13 1089.45
02:43:25 PM all 4.51 0.00 49.62
45.36 1066.00
Reply

...

sh has better regular

expression support than
bash bash has = to
support Basic regular

%sys %iowait %irq

0.50 0.00 0.00 0.00
0.00 0.00 0.00 0.00
0.50 0.00 0.00 0.00

expressions and its result

is exact match bash has
=...
x curl client certificate
error: curl: (58) unable to
set private key file
used curl to post xml
payload to a web service
by client certificate
authentication. It worked
fine Linux(curl 7.15.5
(x86_64-redhat-li...
abbix VS Zenoss
enoss was my favourite
network monitoring tool,
But it is Zabbix now. The

7/26/2016 9:39 PM

UNIX/LINUX TECH NOTES: Troubleshooting a high system CPU usag...

5 of 5

http://honglus.blogspot.com/2010/01/troubleshooting-high-system-cpu-u...

conclusion was made after

comparing Zenoss 3.0.3
to Zabbix 1.8....

Comment as:

Publish

Newer Post

Home

Older Post

Subscribe to: Post Comments (Atom)

7/26/2016 9:39 PM