Академический Документы
Профессиональный Документы
Культура Документы
Charles Levine
Microsoft
clevine@microsoft.com
Outline
Introduction
History of TPC
TPC-A/B Legacy
TPC-C
TPC-H/R
TPC Futures
What is a benchmark?
Domain specific
Desirable attributes
Good benchmarks
Benchmark abuse
Benchmarketing
Benchmark wars
more $ on ads than development
Outline
Introduction
History of TPC
TPC-A Legacy
TPC-C
TPC-H/R
TPC Futures
Administered by:
Shanley Public Relations
650 N. Winchester Blvd, Suite 1
San Jose, CA 95128
DebitCredit
212 tps on NonStop SQL in 1987 (!)
Audited by Tom Sawyer of Codd and Date (A first)
Full Disclosure of all aspects of tests (A first)
Started the ET1/TP1 Benchmark wars of 87-89
TPC Milestones
Unix centric
CPU centric
SPECMarks
Ecumenical
Database/System/TP centric
Price/Performance
Outline
Introduction
History of TPC
TPC-A/B Legacy
TPC-C
TPC-H/R
TPC Futures
TPC-A Legacy
Outline
Introduction
History of TPC
TPC-A/B Legacy
TPC-C
TPC-D
TPC Futures
TPC-C Overview
OLTP transactions:
Stock
100K
W*100K
100K (fixed)
Legend
10
Table Name
District
one-to-many
relationship
<cardinality>
W*10
secondary index
3K
Customer
W*30K
Item
Order
1+
W*30K+
1+
10-15
History
Order-Line
W*30K+
W*300K+
New-Order
0-1
W*5K
TPC-C Workflow
1
Select txn from menu:
1. New-Order
2. Payment
3. Order-Status
4. Delivery
5. Stock-Level
45%
43%
4%
4%
4%
Input screen
3
Output screen
Menu = 0.3
Keying time
Keying = 9.6
Txn RT = 2.1
Think = 11.4
Data Skew
NURand Distribution
0.08
0.07
0.06
0.05
cumulative
distribution
0.04
0.03
0.02
0.01
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
90
80
70
60
50
40
30
20
10
0
0
toThisR
ecord
R
elativeFrequencyof A
ccess
0.09
ACID Tests
Consistency
Isolation
Durability
Transparency
Node A
Node B
select *
from warehouse
where W_ID = 150
select *
from warehouse
where W_ID = 77
1-100
100-200
Transparency (cont.)
Payment txn: 15% of Customer table records are non-local to the home
warehouse.
New-order txn: 1% of Stock table records are non-local to the home
warehouse.
% Network Txns
0
5.5
7.3
TOO HOT!
OK
Hardware
Presentation Services
Term.
LAN
Client
Database Functions
C/S
LAN
Software
Response Time
measured here
RTE, e.g.:
Performix,
LoadRunner,
or proprietary
TPC-C application +
Txn Monitor and/or
database RPC library
e.g., Tuxedo, ODBC
Database
Server
...
TPC-C application
(stored procedures) +
Database engine
e.g., SQL Server
Browser
LAN
Driver System
Legend:
Products
Application Code
Database Functions
Client
DB Server
Web Server
UI APP
COM+
Response Time
measured here
Remote
Terminal
Emulator
(RTE)
Presentation Services
COMPONENT
ODBC APP
ODBC
C/S
LAN
SQL
Server
New-Order
Payment
Delivery
Stock-Level
Order-Status
...
IBM
$350
HP
Digital
$300
$250
$200
Sun
$150
Compaq
$100
$50
$0
0
5000
10000
15000
20000
25000
30000
35000
Compaq
Sequent
$140
IBM
$120
Sun
$100
HP
$80
$60
$40
Unisys
$20
20,000
40,000
60,000
80,000
100,000
120,000
TPC-C Summary
Outline
Introduction
History of TPC
TPC-A/B Legacy
TPC-C
TPC-H/R
TPC Futures
TPC-H/R Overview
TPC-H/R Schema
Customer
Nation
Region
SF*150K
25
Order
Supplier
Part
SF*1500K
SF*10K
SF*200K
LineItem
PartSupp
SF*6000K
SF*800K
Legend:
Arrows point in the direction of one-to-many relationships.
The value below each table name is its cardinality. SF is the Scale Factor.
1, 10, 30, 100, 300, 1000, 3000, 10000 (note that 3 is missing, not a typo)
These correspond to the nominal database size in GB.
(i.e., SF 10 is approx. 10 GB, not including indexes and temp tables.)
Indices and temporary tables can significantly increase the total disk
capacity. (3-5x is typical)
DBGEN is a C program which is part of the TPC-H/R specs
Use of DBGEN is strongly recommended.
TPC-H/R database contents must be exact.
Partitioning
Indexes
TPC-H: only on primary keys, foreign keys, and date columns; only
using simple key breaks
TPC-R: unrestricted for horizontal partitioning
Vertical partitioning is not allowed
TPC-H: only on primary keys, foreign keys, and date columns; cannot
span multiple tables
TPC-R: unrestricted
Auxiliary Structures
TPC-H/R Execution
Database Build
Load
Data
Build
Indexes
Proceed directly to
Power Test
Power Test
Query
Set 0
Timed Sequence
RF2
Proceed directly to
Throughput Test
Query Set N
Updates: RF1 RF2 RF1 RF2 RF1 RF2
1
2
Power Metric
Power @ Size
24
i 22
j 2
i 1
j 1
QI (i,0) RI ( j,0)
where
QI(i,0) Timing Interval for Query i, stream 0
RI(j,0) Timing Interval for refresh function RFj
SF Scale Factor
Throughput Metric
Composite: QphH@Size
Price/Performance: $/QphH@Size
Availability Date
Comparability:
TPC-H/R Results
Outline
Introduction
History of TPC
TPC-A/B
TPC-C
TPC-H/R
TPC Futures
TPC-W Characteristics
TPC-W features:
The simultaneous execution of multiple transaction types that span a
breadth of complexity.
On-line transaction execution modes.
Databases consisting of many tables with a wide variety of sizes, attributes,
and relationship.
Multiple on-line browser sessions.
Secure browser interaction for confidential data.
On-line secure payment authorization to an external server.
Consistent web object update.
Transaction integrity (ACID properties).
Contention on data access and update.
24x7 operations requirement.
Three year total cost of ownership pricing model.
TPC-W Metrics
Reference Material
The End
TPC-A Overview
Teller
B*10
10
100K
Account
History
B*100K
B*2.6M
Legend
Table Name
<cardinality>
one-to-many
relationship
TPC-A Transaction
Easy to understand
Easy to measured
Stresses high transaction rate, lots of physical IO
Account
History
sequential insert
hotspot at end
90-day capacity ensures reasonable ratio of disk to cpu
RTE SUT
Model of system:
SUT
RTE
Host System(s)
T-C
Network*
C
L
I
E
N
T
C-S
Network*
S
E
R
V
E
R
S-S
Network*
TPC-A Metric
tpsA = transactions per second, average rate over 15+ minute
interval, at which 90% of txns get <= 2 second RT
Average Response Time
90th Percentile
Response Time
Number of Transactions
10 11 12 13 14 15 16 17 18 19 20
TPC-A Price
Price
No terminals
No think times
Less disk in priced configuration
TPC Loopholes
Pricing
Client/Server
Package pricing
Price does not include cost of five star wizards needed to get optimal
performance, so performance is not what a customer could get.
Offload presentation services to cheap clients, but report performance
of server
Benchmark specials
Discrete transactions
Custom transaction monitors
Hand coded presentation services