Corporate Technology

Architecture of the Kernel !ased "irtual #achine $K"#%

Jan Kiszka, Siemens AG, CT T DE IT 1 Corporate Competence Center Embedded Linux an!kiszka"siemens!com

Copyright Siemens AG 2010. All rights reserved.

Agenda &ntroduction 'asic K"# model #emory A(& )ptimi*ations (aravirtual devices )utloo+

S#ide $

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

"irtuali*ation of Commodity Computers

&nstruction Set

)n Chip 0esources

'usses &.) /evices

C(,
##, &nterrupt Controllers #emory Cloc+s Timers

S#ide (

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

"irtuali*ing the 123 &nstruction Set Architecture 123 originally virtuali*ation 4unfriendly5 .o *ard/are pro0isions Instructions be*a0e di11erent#, dependin+ on pri0i#e+e context 2er1ormance su11ered on trap&and&emu#ate CISC nature comp#icates instruction rep#acements 6arly approaches to 123 virtuali*ation 3inar, trans#ation 4e!+! 56/are7
Execute substitution code 1or pri0i#e+ed +uest code 6a, re8uire substantia# rep#acements to preser0e i##usion

C29 para0irtua#ization 4e!+ :en7

Guest is a/are o1 instruction restrictions ;,per0isor pro0ides rep#acement ser0ices 4*,perca##s7 <aised abstraction #e0e#s 1or better per1ormance

S#ide -

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

7ard8are assisted 123 C(, "irtuali*ation T8o variants Inte#>s 5irtua#ization Tec*no#o+,, 5T&x A6D&5 4aka Secure 5irtua# 6ac*ine7 &dentical core concept
C(,

( $ 1 %

7ost State

Guest State

"C(,

9 2 1 0

S#ide =

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

Advent and 6volution of K"# &ntroduced to ma+e "T 1.A#/ " availa!le to user space Exposes 0irtua#ization 1eatures secure#, Inter1ace@ Ade0Ak0m #erged :uic+ly A0ai#ab#e since $!?!$% 4$%%?7 Brom 1irst LK6L postin+ to mer+e@ ( mont*s Cne reason@ ori+ina##, 1%%D ort*o+ona# to core kerne# 6volved significantly since then 2orted to 1urt*er arc*itectures 4s('%, 2o/er2C, IA?-7 A#/a,s /it* #atest xE? 0irtua#ization 1eatures 3ecame reco+nized F dri0in+ part o1 Linux

S#ide ?

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

The K"# #odel (rocesses can create virtual machines "#s can contain 6emor, 5irtua# C29s In&kerne# de0ice mode#s Guest physical memory part of creating process; address space "C(,s run in process e1ecution conte1ts 2rocess usua##, maps 5C29s on t*reads
Guest 6emor, 7yper visor (rocess

5C29 5C29
T*read T*read T*read

K"#

<inu1 Kernel C29 C29

C29

S#ide G

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

Architectural Advantages of the K"# #odel (ro1imity of guest and user space hypervisor Cn#, one address space s/itc*@ +uest H *ost Less resc*edu#in+ #assive <inu1 +ernel reuse Sc*edu#er 6emor, mana+ement /it* s/appin+ 4t*ou+* ,ou don>t /*at t*is7 IAC stacks 2o/er mana+ement ;ost C29 *ot&p#u++in+ I #assive <inu1 user land reuse .et/ork con1i+uration ;and#in+ 56 ima+es Lo++in+, tracin+, debu++in+ !!!
S#ide E $%1%&%'&$( Jan Kiszka, CT T DE IT 1 ) Siemens AG, Corporate Tec*no#o+,

"C(, 61ecution =lo8 $K"# "ie8%

User Space CPU Kernel

,pdate conte1t> raise &0?s

0un

7andle J &.) J &nvalid states J ... 7andle J &n Kernel &.) J @v##,A J ...

7andle Signal

,pdate guest state

Save 7ost> <oad Guest State

Save Guest> <oad 7ost State

7andle 7ost &0?

"# entry

61ecute native guest code

Jan Kiszka, CT T DE IT 1

"# e1it $8ith reason%

S#ide '

$%1%&%'&$(

) Siemens AG, Corporate Tec*no#o+,

K"# #emory #odel Slot !ased guest memory 6aps +uest p*,sica# to *ost 0irtua# memor, <econ1i+urab#e Supports dirt, trackin+ &n Kernel "irtual ##, Coalesced ##&) Cptimizes +uest access to <A6&#ike 0irtua# 66IC re+ions )ut of scope 6emor, ba##oonin+ 4+uest H user space *,per0isor7 Kerne# Same&pa+e 6er+in+ 4not K56&speci1ic7
S#ide 1% $%1%&%'&$(

<A6 <A6 <A6 Coa#esced 66IC 9nassi+ned <A6 Guest Address Space ;,per0isor Address Space

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

K"# A(& )vervie8 Step B1C open .dev.+vm Three groups of &)CT<s S,stem&#e0e# re8uests 56&#e0e# re8uests 5C29&#e0e# re8uests (er group file descriptors Ade0Ak0m 1d 1or s,stem #e0e# Creatin+ a 56 or 5C29 returns ne/ 1d mmap on file descriptors 5C29@ 1ast kerne#&user communication se+ment Bre8uent#, readAmodi1ied part o1 5C29 state Inc#udes coa#esced 66IC back#o+ VM: map guest physical memory (deprecated)

S#ide 11

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

'asic K"# &)CT<s K56KC<EATEK56

K56KSETK9SE<K6E6C<LK<EGIC. K56KC<EATEKI<MC;I2 A !!!2IT K56KC<EATEK5C29

4xE?7

K56KSETK<EGS A !!!S<EGS A !!!B29 A !!! K56KSETKC29ID A !!!6S<S A !!!5C29KE5E.TS A !!! K56KSETKLA2IC K"#D0,E

4xE?7 4xE?7

S#ide 1$

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

)ptimi*ations of K"# 7ard8are evolves :uic+ly .ear&nati0e per1ormance in +uest mode Decreasin+ costs o1 mode s/itc*es Additiona# 1eatures a0oid so1t/are so#utions, t*us exits .ested pa+e tab#es TL3 ta++in+ A2IC 0irtua#ization !!! Fhat 8ill continue to consume cyclesG Code pat* bet/een 56&exit and 56&entr, 6ode s/itc*es, i!e! t*e need to exit at a##

S#ide 1(

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

<ight8eight vs. 7eavy 8eight "# 61its 61its cost timeH 3asic state s/itc* in *ard/are Additiona# state s/itc*es in so1t/are Ana#,ze exit reason <eturn to A2IC user space In&kerne# Ana#,ze exit reason In&kerne# IC&A2IC O 2IC Cbtain K56 state 45C29, de0ices7 Coa#escin+ 66IC ;and#e exit cause In&kerne# instruction interpreter 4detect 66IC access7 Nrite back states IJ.000 cycles In&kerne# net/ork stub 40*ost&net7 In0oke K56K<9. So1t/are&mana+ed state s/itc* ;ard/are state s/itc*
I10.000 cycles

S#ide 1-

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

)ptimi*ing <ight8eight 61its <et;s get la*yH 2er1orm on#, partia# state s/itc*es Comp#ete at #atest possib#e point Late restorin+ 1or +uest and *ost state Candidates $123% B29 Debu+ re+isters 6ode#&speci1ic re+isters 46S<s7 0e:uirements 9sa+e detection /*en in +uest mode Demand detection /*i#e in *ost mode
2reemption noti1iers 9ser&return noti1ier
$%1%&%'&$(

z z

Depends on *ard/are support

S#ide 1=

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

<a*y #S0 S8itching Fhy is this possi!leG Some 6S<s unused b, Linux Some 6S<s on#, re#e0ant /*en in user space Some are identica# 1or *ost F +uest Approach Keep +uest 0a#ues o1 certain 6S<s unti#!!! sc*ed&out 1ires K56K<9. ICCTL returns Keep ot*ers unti# user&return 1ires 4Inte# on#,7 )ptimi*ations are vendor specific 61emplary savingC $%%% c,c#es 1or +uest P id#e t*read P +uest

S#ide 1?

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

(aravirtual /evices Advantages <educe 56 exits or make t*em #i+*t/ei+*t Impro0e IAC t*rou+*put F #atenc, 4#ess emu#ation7 Compensates 0irtua#ization e11ects Enab#e direct *ost&+uest interaction Availa!le interfaces - implementions 0irtio 42CI or a#ternati0e transports7 .et/ork 3#ock Seria# IAC 4conso#e, *ost&+uest c*anne#, I7 6emor, ba##oon Bi#e s,stem 4'27 C#ock 4xE? on#,7 5ia s*ared pa+e O 6S<s Enab#es sa1eT6 TSC +uest usa+e
S#ide 1G $%1%&%'&$( Jan Kiszka, CT T DE IT 1

user space !usiness 4primari#,7

K"# !usiness
) Siemens AG, Corporate Tec*no#o+,

An Almost &n Kernel /evice K vhost net GoalC high throughput . lo8 latency guest net8or+ing A0oid *ea0, exits <educe packet cop,in+ .o in&kerne# ME69, p#easeQ

"C(,

memory r.8

The vhost net model ir:fd ;ost user space opens and vhost net con1i+ures kerne# *e#per 8or+er 0irtio as +uest&*ost inter1ace +thread K56 inter1ace@ e0ent1d T: tri++er P ioe0ent1d <: si+na# P ir81d Linux inter1ace 0ie tap or mac0tap <inu1 6na!les multi giga!it throughput
S#ide 1E $%1%&%'&$(

K"# ioeventfd

virtio ring !uffers

r.8 r memory slot ta!le

hypervisor process

net8or+ stac+

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,

Fhat;s ne1tG Generic <inu1 improvements Transparent *u+e pa+es 4mm topic7 .96A optimizations 4sc*edu#er topic7 &mprove spin loc+ holder preemption effects Lero copy - multi :ueue vhost net =urther optimi*e e1its Instruction interpretation 4*ard/are ma, *e#p7 Baster in&kerne# de0ice dispatc*in+ Eested virtuali*ation as standard feature A6D&5 bits a#read, mer+ed and /orkin+ 5T&x more comp#ex but #ike#, so#0ab#e 7ard8are assisted virtuali*ation on non 123 2o/er2C ISA $!%? A<60G&A REa+#eS extensions I
S#ide 1' $%1%&%'&$( Jan Kiszka, CT T DE IT 1 ) Siemens AG, Corporate Tec*no#o+,

Than+s you for listeningH

?uestionsG

S#ide $%

$%1%&%'&$(

Jan Kiszka, CT T DE IT 1

) Siemens AG, Corporate Tec*no#o+,