Академический Документы
Профессиональный Документы
Культура Документы
John Shakshober Director of Performance Engineering, Red Hat Jeremy Eder Principal Performance Engineer, Red Hat June 13, !13
Part I
RHEL tuned profiles top !enc"mar# results $cala!ilty %&$ $c"eduler tuna!les ' %groups Hugepages Transparent Hugepages 2()'*+) ,on-niform (emory Access .,-(A/ and ,-(A0 ,et1or# Performance and Latency2performance 0is# and &ilesystem I3 2 T"roug"put2performance $ystem Performance'Tools perf tuna systemtap
Part II
45A
Disable unnecessary ser"ices, runle"el 3 #ollo$ "endor guidelines for %&'S (uning
)ogical cores * Po$er +anagement * (urbo * Disabling filesystem ,ournal Ensure mount using relatime SSD-+emory Storage Running s$apless Reducing $riteback thresholds if your app does disk &-'
latency2 performance
3!
deadline
deadline 'ff
deadline
deadline
#ilesystem %arriers 'n 8P: ;o"ernor Disk Read.ahead 0isa!le THP %P- %2$tates ondemand
8es Loc#ed 9 *
https=--access/redhat/com-site-solutions-35>!>3
Locality of Pac#ets
Stream from 8ustomer 1 Socket1-8ore1
Socket1-8ore
Socket1-8ore3
Socket1-8ore1 $oc#et*
:se ir?balance for the common case @e$ ir?balance automates @:+4 affinity for &R9s #lo$.Steering (echnologies +o"e Ap1p1BA &R9s to Socket 1= # tuna -q p1p1* -S1 -m -x # tuna -Q | grep p1p1
Sets 8P: affinity for AcommandA to 8P: node 1 4llocates memory out of +emory node 1 8hose node 1 because of P8&.bus $iring
:pstream kernel community $orking on automatic @:+4 balancing/ (est numad in RHE)5
Reduce the D of packets-&R9s the kernel processes (hroughput "s )atency trade.off
3ffload $ummary +(:.chunking offloaded to @&8 Protocol 0irection (8P (C
:DP
(C
;S' +(:.chunking done in.kernel generic segment offload ;R' generic recei"e offload )R' large recei"e offload @&8-dri"er batches certain RC packets
(C
RC
RC
Eernel buffers=
Local ddress!"ort 1'(.1'.1.)*!)+,*( 1'(.1'.1.)*!-++-* "eer ddress!"ort 1'(.1'.1.),!1(+*1'(.1'.1.),!-),.1
# ss |grep -v ssh State Recv-Q Send-Q #S$ % & & #S$ % & )'()1(+
/2!
/!!
Latency .(icroseconds/
*@< *2< *<< AD A< 6< @< 2< < *B *6 2< * *C 2* @ RHEL6?@ .$R2I3> tuned/ RHEL6?@ .)ridge tuned/ *B *6 62 @C BD 22 E@
(in RHEL6?@ .tuned/ RHEL6?@ .untuned/ (ean EE?EF %are metal $td0e:
EG+ H SR.&'G
EG+ H %ridge
Latency .(icroseconds/
*A
<?@
(a7
Gariable idle states for each core 8state lock disables turbo, but increases determinism
8.state &mpact on Jitter
2!
)atency 6+icroseconds7
!! 12! 1!! 2! !
85
83
81
8!
+a<
#-./ .-0* .-11 .-2/ *-#* .-11 .-.0 *-#1 .-11 )-,2 )-,2 .-11
latency-performance pk cor C ! # # # # # ) . * "c# $%& '(C "c) "c* #-## #-## #-## #-## "c+ #-## #-## #-## #-## "c, #-## #-## #-## #-##
*-*# .-0# )##-## *-*# .-0# )##-## *-*# .-0# )##-## *-*# .-0# )##-##
:se cron to set latency.performance tuned profile $hen necessary/ Set tuned profile in application init script
4dded counters in RHE) 5/ Enhanced again to reduce o"erhead in 5/1 # egrep /trans|thp/ /proc/vmstat
nr0anon0transparent0hugepages (&1+ thp01ault0alloc ')&( thp01ault01all2ac3 & thp0collapse0alloc ,&1 thp0collapse0alloc01a4led & thp0spl4t (1
Transparent Hugepages
p4d S67#80 rtpr4 a114n4t9 voluntar9 nonvoluntar9 1),++ :$7#R & &x111 1 &
# tuna -c( -t netserver -m # tuna -t netserver -" thread ctxt0s54tches cmd netserver
p4d S67#80 rtpr4 a114n4t9 voluntar9 nonvoluntar9 1),++ :$7#R & ( 1 &
+o"e Ap1p1BA &R9s to Socket 1= # tuna -q p1p1* -S& -m -x # tuna -Q | grep p1p1 '+ p1p1-& '. p1p1-1 +& p1p1-( +1 p1p1-) +( p1p1-, 6ore & 1 ( ) , s1c s1c s1c s1c s1c
...
Ser"ers may ha"e more than 1 P8& bus/ &nstall adapters IcloseJ to the 8P: that $ill run the performance critical application/ Lhen %&'S reports locality, ir?balance handles @:+4-&R9 affinity automatically/
/.:##-# 3et4ork controller: 5ellano6 'echnologies 5'.,2## 7amily 8Connect9-*: # cat ;sys;devices;pci####<:/#;####<:/#<:#*-#;####<:/.<:##-#;local=cpulist )>*>2>,>0>))>)*>)2
# dmesg ? grep @3!5A node@ pci=bus ####:##: on 3!5A node # (p6m )) pci=bus ####:/#: on 3!5A node ) (p6m .) pci=bus ####:*f: on 3!5A node # (p6m )) pci=bus ####:,f: on 3!5A node ) (p6m .)
# lstopo-no-graph4cs |egrep /N?@ |eth,/ N?@ Node L#& <"#& 1,,A%= N?@ Node L#1 <"#1 1,,A%= Net L#1& Beth,B
RHE) 5/1 includes perf from upstream kernel 3/5 https=--perf/$iki/kernel/org perf top 6dynamic7 perf record - report 6sa"e and replay7 perf stat OcommandP 6analyNe a particular $orkload7
perf/data recordings
turbostat included in cpupo$erutils package h$loc no$ reports P8& bus topology P(P (ech Pre"ie$ +ellano< &nfiniband SR.&'G (ech Pre"ie$
S In "oth ##$ and #pti%i&ed cases Red Hat 'nterprise (in)* deli+ers "etter thro)ghp)t and lo,er latenc- to critical net,or./hea+- applications S http0//,,,.principledtechnologies.co%/RedHat/RH'(12net,or.20113.pd!
34
05/29/13
Deadline
($o ?ueues per de"ice, one for read and one for $rites &-'s dispatched based on time spent in ?ueue Per process ?ueue Each process ?ueue gets fi<ed time slice 6based on process priority7 #&#' Simple &-' +erging )o$est 8P: 8ost
8#9
@oop
RHE) 5/3 2!!! 12!! 1!!! 32!! 3!!! 2!! !!! 12!! 1!!! 2!! ! e<t3 e<t1
RHE) 5/1 2!!! 12!! 1!!! 32!! 3!!! 2!! !!! 12!! 1!!! 2!! ! gfs <f s e<t3
RHE) 5/3
RHE) 5/1
e<t1
gfs
<f s
+a< file system siNe 1!!(% +a< file siNe %est performing 1!!(%
e7t@ recommended
e7t@
e7tB
< B6<< C2<< *<A<< *@@<<
latency. performance
3!
deadline
deadline 'ff
deadline
deadline
#ilesystem %arriers 'n 8P: ;o"ernor Disk Read.ahead 0isa!le THP 0isa!le %2$tates ondemand
Tes Tes
https0//access.redhat.co%/site/sol)tions/319093
&ree sla!cac"e
W!!
5!!
2!!
T"roug"put in ( )'$ec
1!!
3!!
!!
then %4RR&ERSYoff 6for mounts other than root-boot "ols7 D set cpuspeed go"ernorsYperformance
http0//,,,.principledtechnologies.co%/RedHat/RH'(12I#20113.pd!
43
05/29/13
http0//,,,.principledtechnologies.co%/RedHat/RH'(12I#20113.pd!
44
05/29/13
Each file system or block de"ice has its o$n flush daemon 4llo$s different flushing thresholds and resources for each daemon-de"ice-file system/ Pre"ents some de"ices from not getting flushed because a shared daemon blocks used all resources Replaces pdflushd $here a pool of threads flushed all de"ices/
Read67-Lrite67
memory copy
Pagecac"e page
#lush daemon
buffer
:ser space
#ile system
Eernel
RHE) 2/2
RHE) 5
RHE) 5 ZtunedZ
34
dirty0ratio6 !3 of R4+ dirty7 U processes start synchronous $rites flushd $rites dirty buffers in background
gluster :olume set <volume> group :irt C#S mkfs .n siNeYX1> , mount inode51, noatime RHS ser"er= tuned2adm profile r"s2:irtualiJation
&ncrease in readahead, lo$er dirty ratioAs %etter response time shrink guest block de"ice ?ueue
tuned r"s2:irtualiJation
rnd.$rite
rnd.read
se?.$rite
se?.read
I(:@EDJ tool U ad,usts system parameters to match en"ironments . throughput-latency/ (ransparent Huge Pages U auto select large pages for anonymous memory, static hugepages for shared mem @on.uniform +emory 4ccess 6@:+47
@:+4stat enhancements @:+4ctl for manual control @:+4D daemon for auto placement
Helpful Lin#s
Performance (uning ;uide )o$ )atency Performance (uning ;uide 'ptimiNing RHE) Performance by (uning &R9 4ffinity EG+ Performance ;uide S(48 @et$ork &-' S&; %log= http=--$$$/breakage/org- or [,eremyeder
B <<< <<< 2 D<< <<< 2 <<< <<< * D<< <<< * <<< <<< D<< <<< <
RHA $ RHE LB RHE L@ RHE L@ RHE L@ RHE L@ RHE L@ RHE L6 RHE LB RHE LD RHE LD
tpm% Q'tpm%
o%parison o! T P / res)lts )sing the Red Hat operat/ ing s-ste%. 5or %ore in!or%ation a"o)t the T P and the "ench%ar. res)lts re!erenced here see ,,,.tpc.org.
* D<B D@@
11!!!!!
* B2< <A2
3!
2C6 BAB
!!!!!
!
HP << 26 r7
Q*?A*
pD er : er e$ < DC
QB?EB
s es pr 7 <E DD
Q2?DD
m te ys $ e7 &l @< 72
Q<?DB
m te ys $ e7 &l < 6D 7B
Q<?D*
B EL H R
@ EL RH
( I)
D EL RH
( I)
r 1e Po
6 EL RH
( I)
>( '; 6 EL RH
( I)
4p"H Q'4p"H
Q*<?<<
Q<?<<
RHELB
RHEL@
RHELD
RHEL6
RHEL6
RHEL6
o%parison o! T P / res)lts )sing the Red Hat operat/ ing s-ste%. 5or %ore in!or%ation a"o)t the T P and the "ench%ar. res)lts re!erenced here see ,,,.tpc.org.
@<B 2B<
@2< <E2
\3/!!
\ /2!
2D* D6*
\1/2!
9phH \-9phH
\1/!!
EA ADC
\!/3X \!/15 \!/12 \!/1 \!/11 \!/!! \!/2!
RHEL@
RHEL6
RHEL6
RHEL6
RHEL6
RHEL6
o%parison o! T P / res)lts )sing the Red Hat operat/ ing s-ste%. 5or %ore in!or%ation a"o)t the T P and the "ench%ar. res)lts re!erenced here see ,,,.tpc.org.