Академический Документы
Профессиональный Документы
Культура Документы
#vmworldapps
Disclaimer
Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
I have been with VMware for the last 7 years, working on Java
and vSphere
Open source contributions Prior work with Cisco, Oracle, and Banking/Trading Systems. Authored the Enterprise Java Applications Architecture on VMware
Disclaimer
Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
Agenda
Conventional Middleware Platforms Middleware Platform Architecture on VMware vSphere
Enterprise Java applications are multitier (Client-Server) The - in Client-Server is essentially the Middleware
DB Server Tier
DB Servers
Web Servers
VMware vSphere
APPLICATION SERVICES
Capacity On Demand
Dynamic
High Availability
VMware vFabric
Programming Model
Rich Web
Data Access
Integration Patterns
Batch Framework
WaveMaker
Cloud Foundry
Data Director)
Virtual Datacenter
Cloud Infrastructure and Management
App Director
Scale Up Test
ESTABLISH BUILDING BLOCK VM Establish Vertical scalability Scale Up Test Establish how many JVMs on a VM? Establish how large a VM would be in terms of vCPU and memory
Building Block VM
No
Test complete
SLA OK?
DETERMINE HOW MANY VMs Establish Horizontal Scalability Scale Out Test How many VMs do you need to meet your Response Time SLAs without reaching 70%-80% saturation of CPU? Establish your Horizontal scalability Factor before bottleneck appear in your application
VM Memory
Guest OS Memory -Xss per thread Java Stack JVM Memory Perm Gen
Other mem
-XX:MaxPermSize
Direct native
memory Non Direct Memory Virtual
Initial Heap
-Xms
Address space
10
Guest OS=Memory (depends on OS/other processes) VM Memory Guest OS approx Memory1G + JVM Memory
JVM Memory = JVM Max Heap (-Xmx value) + JVM Perm Size (-XX:MaxPermSize) +
Perm Size is an area additional the Xmx (Max Heap) value and NumberOfConcurrentThreads * (-Xss) +to other Mem is not GC-ed because it contains class-level information. other mem is additional mem required for NIO buffers, JIT code
cache, classloaders, Socket Buffers (receive/send), JNI, GC internal info
If you have multiple JVMs (N JVMs) on a VM then: VM Memory = Guest OS memory + N * JVM Memory
11
Sizing Example
set mem Reservation to 5088m VM Memory (5088m) JVM Memory (4588m) Guest OS Memory Java Stack
Perm Gen
-XX:MaxPermSize (256m)
12
Perm Gen
-XX:MaxPermSize (0.5g)
13
components
Memory Available for all VMs = 96*0.99 -1GB => 94GB Per NUMA memory => 94/2 47GB
14
96GB RAM 47GB RAM VMs with 8vCPU 2 sockets 8 pCPU per socket
ESX Scheduler
96 GB RAM on Server
15
Performance Perspective
% CPU
R/T
80% Threshold
16
1 VM
4vCPU 1 JVM 4GB
2 VMs
2vCPU on each VM 2 JVMs 2.5 GB each
4 VMs
1vCPU on each VM 4 JVMs 2 GB each
17
2 vCPU VM with 1 JVM, for tier-1 production workloads Maintain this ratio as you scale out or scale-up, i.e. 1 JVM : 2vCPU Scale out preferred over Scale-up, but both can work You can diverge from this ratio for less critical workloads
18
JVM-1
JVM-2
JVM-1
JVM-2
Vertical
Web
Horizontal
Job
21
Which GC?
22
Either you tune for Throughput or Latency, one at the cost of the
other Reduce Latency improved R/T reduce latency impact slightly reduced throughput
Tuning Decisions
Increase Throughput
23
S S 0 1
minor GC threads
24
application threads
-XX:+UseStringCache
This JVM configuration scales up and down effectively -Xmx=-Xms, and Xmn 33% of Xmx -XX:ParallelGCThreads=< minimum 2 but less than 50% of available
vCPU to the JVM. NOTE: Ideally use it for 4vCPU VMs plus, but if used on 2vCPU VMs drop the -XX:ParallelGCThreads option and let Java select it
26
http://www.vmware.com/resources/techresources/10220
http://www.vmware.com/resources/techresources/10231
27
Follow the design and sizing examples we discussed thus far Set appropriate memory reservation Leave HT enabled, size bases on vCPU=1.25pCPU if needed RHEL6 and SLES 11 SP1 have tickless kernel that does not rely on a high frequency interrupt-based timer, and is therefore much friendlier to virtualized latency-sensitive workloads
Do not over commit memory Locators/heartbeat process should not be vMotion migrated, it
otherwise would lead to network split brain problems
vMotion over 10Gbps when doing scheduled maintenance Use Affinity and Anti-Affinity rules to avoid redundant copies on
the same VMware ESX/ESXi host
28
Disable NIC interrupt coalescing on physical and virtual NIC Extremely helpful in reducing latency for latency-sensitive virtual
machines
29
30
31
SQLFire scaled 4x compared to RDBMS Response times of SQLFire are 5x to 30x faster than RDBMS Response times on SQLFire are more stable and constant with increased load. RDBMS response times increase with increased load
32
Flexibility to change compute resources, VM sizes, add more hosts Ability to apply hardware and OS patches while minimizing
downtime
Ability to tune the entire stack within one platform Ability to monitor the entire stack within one platform Ability to handle seasonal workloads, commit resources when
they are needed and then remove them when not needed
33
Jeff Battisti
Sr. Enterprise Architect August 30, 2012
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
The business behind healthcare with the broadest view of the supply chain
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Agenda
Virtualization journey Why virtualize on WebSphere on VMWare Factors Impacting Migration/Expansion Virtualization Questions to answer
Performance and scalability High availability Licensing
Summary
36
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Virtualization
Journey
Theme Centralized IT Shared Service Capital Intensive - High Response
Timeline
Virtual
2005 2008
Consolidation < 40% Virtual <2,000 VMs <2,355 physical Data Center Optimization 30 DCs to 2 DCs
2009 2012
Internal cloud >81% Virtual >4,054 VMs <1147 physical Power Remediation P2Vs on refresh
2013 2016
Cloud Resources >95% Virtual >10,000 VMs <800 physical Optimizing DCs Internal disaster recovery Metered service offerings (SAAS, PAAS, IAAS) Shrinking HW Footprint > 50% Utilization > 100:1 VM/Physical Cloud Computing Virtualized Databases Open Source Migrations
DC
HW
SW
Business Critical Systems WebSphere ~ 490 Unix to Linux ~ 655 SAP ~ 550
37
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Virtualization
38
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Virtualization
Questions to Answer
Alignment across teams and consistent messaging to stakeholders was critical Can the system perform & scale?
JVM Stacking & Vertical scaling Massive OLTP Performance Memory - over commitment, DRS, and large JVMs IO Impacts
Affinity/Anti-affinity Complexity must remain low Storage, VMWare, servers, chassis, and switches cannot be a single point of failure User error should not take down both sides of critical applications
39
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Non-functional enhancements:
Central Logging framework Remote Portlet Rendering Positiong for future cross datacenter failover 64-bit JVM and JEE 6 support
Systems 111 DB Instances 31 Portal Servers 19 Commerce Servers 15 WAS Servers 7 WCM Servers 3 BODL Servers 11 Business Object Servers 15 Endecca Servers 9 Planet Press Servers
40
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
WebSphere VM Performance
Sweet spot is between 2 4 cores JVM stacking/vertical scaling reduced, but still in place for licensing reasons
Source: http://www.vmware.com/resources/techresources/10095
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
2,000,000
1,433,000
1,500,000
1,000,000
509,962 350,642
500,000
0 HP BL490 Nehalem 2.93 GHz 8 cores HP DL580 G7 32 cores* IBM Power7 750 3.55GHz IBM Power6 550 4.2 GHz 32 cores 8 cores
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
450%
400% 350% 300%
50% 13%
View Dashboard Order Submit Buy Now View Product Details Quick Add Search Results View Cart 0%
Version 1
Version 2
% Diff
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
High Availability
Low Complexity
High availability for thousands instead of thousands of high availability solutions
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Licensing
45
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
Summary
Culture shift from virtualization to save costs to virtualization for superior capabilities We are continually improving performance, resiliency, availability, and manageability The Simplicity of this design enables us to effectively and efficiently manage these systems
Copyright 2011, Cardinal Health, Inc. or one of its subsidiaries. All rights reserved.
47
Backup slides
48
Ballooning makes the Guest OS aware that ESX Host memory is short Due to VMs isolation, the Guest OS is not aware that it is in a VM and is
not aware of the state of other VMs, or the ESX Host memory situation
Balloon driver is loaded into Guest OS, communicates with ESX via
private channel, and ESX will instruct it to inflate (allocate Guest OS Physical Pages) by an amount. An amount that it needs to reclaim.
49
Memory Reservation
We used an amount of 5088MB as Memory Reservation for the VM when all
the various memory segments were added up.
50
51
52
GC Policy Types
GC Policy Type Description
Concurrent GC
Concurrent Mark and Sweep, no compaction Concurrent implies when GC is running it doesn't pause your application threads this is the key difference to throughput/parallel GC Suited for application that care more about response time than throughput CMS does use more heap when compared to throughput/ParallelGC CMS works on OLD gen concurrently, but young generation is collected using ParNewGC, a version of the throughput collector Has multiple phases: Initial mark (short pause) concurrent mark (no pause)
ESX Host Swapping this is a last resort, and things are quite bad at
this stage. If TPS and Ballooning didnt work, then ESX will swap VM memory to swap file.
54
55
S S 0 1
56
Minor GC threads
58
minor GC threads
59
application threads
60
-Xmn16g -XX:+UseConcMarkSweepGC
Fixed size Young Generation The concurrent collector is used to collect the tenured generation and does most of the collection concurrently with the execution of the application. The application is paused for short periods during the collection. A parallel version of the young generation copying collector is used with the concurrent collector. This sets whether to use multiple threads in the young generation (with CMS only!). By default, this is enabled in Java 6u13, probably any Java 6, when the machine has multiple processor cores. This sets the percentage of the heap that must be full before the JVM starts a concurrent collection in the tenured generation. The default is some where around 92 in Java 6, but that can lead to significant problems. Setting this lower allows CMS to run more often (all the time sometimes), but it often clears more quickly to avoid fragmentation.
-XX:+UseParNewGC
XX:CMSInitiatingOccupancyFraction= 51
61
Indicates all concurrent CMS cycles should start based on XX:CMSInitiatingOccupancyFraction=51 Do young generation GC prior to a full GC. Desired percentage of survivor space used after scavenge. Ratio of eden/survivor space size
62
-XX:+UseBiasedLocking
Enables a technique for improving the performance of uncontended synchronization. An object is "biased" toward the thread which first acquires its monitor via a monitorenter bytecode or synchronized method invocation; subsequent monitor-related operations performed by that thread are relatively much faster on multiprocessor machines. Some applications with significant amounts of uncontended synchronization may attain significant speedups with this flag enabled; some applications with certain patterns of locking may see slowdowns, though attempts have been made to minimize the negative impact. Sets the maximum tenuring threshold for use in adaptive GC sizing. The current largest value is 15. The default value is 15 for the parallel collector and is 4 for CMS.
-XX:MaxTenuringThreshold=15
63
-XX:ParallelGCThreads=6
Sets the number of garbage collection threads in the young and old parallel garbage collectors. The default value varies with the platform on which the JVM is running. Enables the use of compressed pointers (object references represented as 32 bit offsets instead of 64-bit pointers) for optimized 64-bit performance with Java heap sizes less than 32gb.
-XX:CompressedOops
-XX:+OptimizeStringConcat
-XX:+UseCompressedStrings
Optimize String concatenation operations where possible. (Introduced in Java 6 Update 20)
Use a byte[] for Strings which can be represented as pure ASCII. (Introduced in Java 6 Update 21 Performance Release) Enables caching of commonly allocated strings
-XX:+UseStringCache
64
Xgcpolicy:Optavgpause
-Xgcpolicy:Gencon
Latency sensitive apps, objects in the transaction don't survive beyond the transaction commit
65
Optimizes for short and even pause times. Can use -XpauseTarget:time The pause target affects the application throughput. A lower pause target inflicts more overhead on the memory management system. Optimizes for very short and deterministic pause times Can use XpauseTarget:time
-Xgc:deterministic
66
What is the practical limit for JVM Memory sizing (not to scale)
Most limiting practical sizing factor is the per NUMA node RAM Guest OS Limit 1 to 16 TB 16 Exa Bytes
67
<4GB
Enterprise webapps internal/ external
69
Throughput Collector -XX:+UseParallelOldGC should be good enough Benefit from automatic 32bit address compression even though you are
in 64bit JVM, most JVMs do this automatically
<4GB
Enterprise webapps internal/ external
70
Medium amount of GC tuning needed MinorGC frequency, -Xmn, and FullGC duration adjustments start to
become a consideration
71
Medium amount of GC tuning needed MinorGC frequency, -Xmn, and FullGC duration adjustments start to
become a consideration
12 GB to 32GB
72
32GB to 128 GB
Extensive amount of GC tuning needed MinorGC frequency, -Xmn, and FullGC duration adjustments
start to become a consideration, survivor space sizing, *.*
Perm Gen
-XX:MaxPermSize (256m)
74
75
This limit the number of horizontally scaled out, or concurrent threads you can fit
within the memory space
This is more suited to large JVMs and data intensive in memory data management
systems, i.e. distributed cache etc.
We are trying to localize as much of the execution within the L1, L2 and L3 and
NUMA, in that order of priority
76
2 vCPU VM with 1 JVM, for tier-1 production workloads Maintain this ratio as you scale out or scale-up, i.e. 1 JVM : 2vCPU Scale out preferred over Scale-up, but both can work You can diverge from this ratio for less critical workloads
77
ESX memory overhead 1% 128*0.99/8 => 15.84GB Java process is typically 25% in addition
to Java heap, hence JVM Heap can be 11.88GB
79
80
81
Performance Perspective
Below 80% CPU utilization in the VM, the native and virtual
configurations have essentially identical performance, with only minimal absolute differences in response-times.
% CPU
R/T
80% Threshold
82
Performance Perspective
% CPU R/T
80% Threshold
83
Performance Perspective
Shows the peak throughput for a single instance of Olio running on tc Server, both natively and in a VM, with 1, 2, and 4 CPUs
84
1 VM
4vCPU 1 JVM 4GB
2 VMs
2vCPU on each VM 2 JVMs 2.5 GB each
4 VMs
1vCPU on each VM 4 JVMs 2 GB each
85
Performance Perspective
Number of VMs
4 2 1
Best case
1 VM
4vCPU 1 JVM 4GB
2 VMs
2vCPU on each VM 2 JVMs 2.5 GB each
4 VMs
1vCPU on each VM 4 JVMs 2 GB each
87
JVM-1 Web
JVM-2 Web
JVM-3 Web
JVM-4 Web
JVM-2
Web
88
Option -3 JVM Stacking (third best option) 1 off 4 vCPU VMs 2 JVMs on each VM 4GB Heap on each JVM Reduce JVM/VM sprawl Reduce OS count Increased number of JVM instances on a single OS While JVM to vCPU ratio is still 1 JVM to 2 vCPU it is still not as prudent as Option-1
JVM-1 JVM-2 Web Web
It would likely not perform if you configured a 2vCPU VM In busy JVMs GC would require 1vCPU while other user
transactions would take additional available vCPUs
89
Performance Perspective
2 VMs 2 vCPU each, 2.5G RAM on each VM showed best Throughput for the amount of memory used and R/T achieved
90
Performance Perspective
Scaling of Peak Throughput
91
APP-CAP1426
#vmworldapps