Newport Group, Inc.

IT Trends Research and Reporting

Annual Load Test Market

Summary and Analysis
Analysts: Billie Shea and Kevin Gallagher
Load Testing: The process of analyzing software applications and supporting infrastructures to determine acceptable performance, scalability, capacity and transaction handling capabilities by modeling real-world application
usage conditions and executing them against the application and supporting infrastructure under test.

INTRODUCTION
The year 2000 was an exceptionally strong year of market growth for load testing
tools and related services. The worldwide market (inclusive of remote/hosted load
testing services) for distributed environments exceeded Newport Group forecasts
and reached $359.3 million, up 68 percent from its 1999 level of $214 million, and

Numbers In
Millions
Source: Newport Group, Inc. 2001

outpacing the previous years percentage growth rate of 55 percent. Newport Group
forecasts that the market for automated load testing solutions will continue to grow,
but at a slower pace in 2001 given the challenging economic conditions. We believe
the market will reach $445.6 million in 2001 and reach nearly $880 million by 2004
(see figure 1, Worldwide Load Testing Market Growth, Distributed Environments,
1998 - 2004).

This annual market summary report will discuss the state of

the market for load testing tools and related services in distributed environments, inclusive of remote/hosted load testing services for the year 2000. This report will provide forecasts and market trends as well as highlight market share in
distributed environments and its largest subsegment, Web or
IP-based environments. Further, load testing offerings from
the leading vendors shaping this space will be profiled and
some new vendors will be introduced. Vendors in this report
range widely in size and vary from vendors providing a
wide array of products and services to those that offer
focused point solutions in the product or service arena only.

As it relates to all products and related services comprising

the $823.7 million automated software quality (ASQ) market for distributed environments, the submarket of tools and
related services tackling load testing continues to be the
most significant driver of ASQ market performance, representing a full 43.6 percent of the total market. This is an
increase of 9 percent over last year when the load testing
submarket was representative of 35 percent of the overall
ASQ market for distributed environments. For purposes of
this report, distributed environments include traditional
client/server application environments, packaged ERP
(Enterprise Resource Planning) and CRM (Customer
Relationship Management) application environments and
the full range of Internet, Intranet and extranet application
environments.

COMMENTARY ON MARKET TRENDS

The year 2000 was the strongest growth year for load testing solutions to date. Once being a market with only a handful of players, Newport Group recognizes the introduction
of many new players over the past 24 months. Along with
new players, new approaches to load testing have emerged,
most with the intention of driving price for load testing tools
down. For example, a hardware-based load-generation
appliance approach was recently introduced in addition to
networks of peer-to-peer (P2P) architectures for generating
traffic over the Web to test IP-based applications.

The primary reason behind the strong growth of the load

testing market is the continued increase and widespread
reliance on Web-based computing environments as business
revenue generators and customer service tools. Recent
Newport Group research reveals that on average todays
Global 2000 business attributes 15 percent of bottom line
revenue to their Web-enabled business channel and that
number is expected to grow to 22 percent by the end of
2001. Couple the business importance of Web applications
with the complex and diverse nature of these computing
environments and it is clear why performance, scalability
and overall reliability must be a top priority in support of
business operations and growth. Therefore, as the business
value of load testing for Web-enabled environments has
been proven and recognized, load testing efforts are now
more readily supported by business executives as an integral
part of the application development, deployment and management process. The result is that load testing solutions
aimed at Web-based environments realized overall growth
of 114 percent, growing from their 1999 level of $123.5 million to $263.8 million in 2000.

As a result of converging market conditions, the average

expenditure for an automated load testing tool in 2000 was
$33,000, 33 percent less than the average price of $49,500
in 1999. Newport Group believes that a number of market
conditions attributed to the decrease. First, there was an
increase of vendor competition, with newer vendors leveraging price in competitive situations to win deals. In addition, some vendors chose to blend the use of in-house load
testing with outsourced, hosted/remote load testing services.
Therefore, end-users tended to purchase tools with lower
Virtual User (VU) licenses for testing throughout development and then chose to supplement their in-house testing
with hosted/remote services when high volumes of simulated users were required. In 1999, end-users purchased tools
with an average license for 600 VUs. In 2000 that number
dropped to an average of less than 500. The number of users
simulated via hosted/remote load testing services ranged
from 1000 to 5000, on average.

And, although adoption of load testing solutions outside of

the United States accounts for only 29.5 percent of the market revenue in 2000, growth rates for load testing solutions
in the International markets outpaced (+79%) growth rates
in the United States (+64%). Europe and Asia/Pacific realized the strongest growth internationally. General on-site
consulting services for load testing remained very strong
throughout the industry, resulting in more widespread market education about successful load testing strategies and
techniques. Lastly, no company shined more than marketleader Mercury Interactive, increasing their overall market
share for load testing tools and services in distributed environments by an impressive 14 percent, and widening the gap
between themselves and everyone else in the load testing
tools and services space.

Another reason for the drop in average price is that a broader range of companies many smaller in size than the market has previously experienced are adopting load testing
solutions as they move their business critical applications to
the Web. Thus, based on a broader market appeal for load
testing solutions and based on the current performance/scalability requirements for a greater number of small to midsize companies, the average VU license has decreased. The

last factor that may have influenced the decrease, but to a

lesser extent, is the challenging economic climate, which
caused IT spending to be scrutinized a bit more carefully
toward year-end.

time the dust settled at the end of the year, hosted/remote

load testing services had accounted for slightly less than 3
percent of the overall market. Hosted load testing services
leverage the Internet to conduct load tests over the wire.
With this model, load tests can be driven from one to many
geographic locations around the world to stress a Web application and infrastructure from outside of its firewall.
Vendors competing in this space report that on average their
customers choose to have user loads driven from three different geographic locations and that it takes an average of
four test runs to complete most service engagements. The
main benefit to this approach is its ability to test the firewall
and ensure that appropriate ISP bandwidth is allocated and

The core strength of the load testing market still resides with
the handful of traditional players, and, of those players,
companies offering a wide array of testing products and
services that extend beyond load testing, such as test management, functional testing, component testing and performance monitoring and management solutions, realized
the strongest growth in 2000. We believe they will also be
the strongest performers in 2001. Mercury Interactive
remains the dominant market leader
and the one vendor, which all others
aim to compete against.

Worldwide Load Testing Market Share

Distributed Enviroments, 2000

Given the steady upward trend in

growth, and the clear business recognition for the need and importance of
Total $359.3 Million
measuring and managing application
(product & related services, including hosted/remote services)
figure 2
performance and scalability, we
believe the market will sustain some
Other 7%
Compuware 9%
new vendors, but only those with the
Segue 7%
Cyrano 3%
strongest and most aggressive competitive strategies. These new venEmpirix 6%
dors will thus broaden and further
enhance the overall growth potential
of the entire market. However, it is
Rational 11%
important to note that new players in
this space have entered a challenging
market when going up against strong
and highly aggressive traditional
RadView 3%
players such as market-leader
Mercury Interactive. As it stands
currently, the top five vendors in the
Mercury
load testing tools and related serInteractive 54%
vices space including Mercury
Interactive, Rational Software,
Source: Newport Group, Inc. 2001
Compuware Corporation, Segue
Software and Empirix account for a full 87 percent of the
that routers, switches and peering relationships will support
total revenue in 2000. And, consider that toward the end of
anticipated user loads. A hosted load testing service prothe year 2000 and into 2001, some shakeout and consolidavides a user-centric perspective of what will typically be
tion among vendors has occurred. For example, service-only
experienced in the live environment.
performance testing and management vendor iSharp closed
its doors and Keynote Systems purchased technology assets
Of the newer, hosted load testing services-only vendors,
from Envive Corporation.
Keynote Systems was the strongest performer with its
KeyReadiness service. Of the traditional testing tool companies now enhancing their current offerings with
HOSTED/REMOTE LOAD TESTING SERVICES
remote/hosted load testing services, Mercury Interactive and
Compuware Corporation were the strongest performers,
In 2000, multiple vendors both new and traditional, introwith Mercury Interactive being responsible for better than
duced remote or hosted load testing services, and we
half of the revenue generated from these new services. True
watched as these types of services began to take root. By the

Worldwide Load Testing Tool Market Share

Distributed Enviroments, 2000
Total $288.7 Million (Product Only)
figure 3

Other 5%
Segue 7%

Compuware 7%
Cyrano 2%

Empirix 7%

Rational 11%

testing plays in todays technologydriven economy. The worldwide market for product and related services
(inclusive of remote/hosted load testing services) for distributed environments reached $359.3 million, up 68
percent from its 1999 level of $214
million. Newport Group defines related services as any consulting services
that are sold to an end-user client,
which are over and above any services
that may be sold as part of the tool purchase. We factor in consulting services
provided by the tool vendors and their
direct channel partners.

Once again, Mercury Interactive

retains the leadership position in this
market space. In fact, the company significantly strengthened their leadership
Mercury
Interactive 58% position in 2000, capturing a full 54
percent of the worldwide market, an
Source: Newport Group, Inc. 2001
impressive 14 percent gain in market
share over 1999. Rational Software
once again captured the second place
to their market leadership position, Mercury Interactive was
position with 11 percent market share and Compuware with
first-to-market to deliver hosted / remote load testing serv9 percent, Segue with 7 percent and Empirix with 6 percent
ices.
rounded out the top five vendors in this category (see figure
2, Worldwide Load Testing Market Share, Distributed
Better than 70 percent of customers for hosted/remote load
Environments, product and related services, 2000).
testing services were new to the discipline of load testing
and on average these types of services were able to deliver
If we look at market share for distributed environments
3X performance gains and an average of 10 load testingbased on load testing product only, which reached $288.7
focused consulting hours to each customer. Newport Group
million in 2000, Mercury Interactive is again the market
expects that the percentage of companies adopting these
leader with 58 percent market share. Rational Software
services, who are new to load testing, will decrease in 2001
remains in second with 11 percent market share, Empirix,
as more IT teams that have load testing products in-house
Segue Software and Compuware Corporation share the third
choose to supplement their internal efforts with these types
place spot, each holding 7 percent market share (see figure
of service offerings. Traditional testing tool companies such
3, Worldwide Load Testing Market Share, Distributed
as Mercury Interactive, Compuware Corporation and Segue
Environments, product only, 2000).
Software should really benefit by leveraging their large base
of existing customers. Also noteworthy is that for compaWeb load testing revenue for products and related services
nies like Mercury Interactive, Compuware Corporation and
in 2000 totaled $263.8 million, a 114 percent increase over
Empirix that have broad product and service offerings, host1999. Second to Mercury Interactives 57 percent market
ed/remote load testing services work to enhance product
share in this catagory, Segue Software posted the most revsales. On average, 42 percent of customers signing on for
enue in the Web load testing space and holds 9 percent marhosted/remote services went on to purchase product for inket share. Not far behind are Compuware and Empirix sharhouse use in 2000.
ing third place, each with 8 percent market share, Rational
Software with 6 percent and RadView Software with 4 perHEALTHY INCREASES CONTINUED IN 2000
cent (see figure 4, Worldwide Load Testing Market Share,
Web Environments, product and related services, 2000).
Sustaining the momentum gained in 1999, healthy increases
in load testing revenue spilled into the year 2000. The largIf we look at market share for Web load testing based on
er than forecasted growth validates the importance load
product only, which increased 121 percent to reach $216
RadView 3%

million in 2000, Mercury Interactive

remains the market leader with a remarkable 63 percent market share. Empirix and
Segue share second place in this category,
each with 9 percent market share (see figure 5, Worldwide Load Testing Market
Share, Web Environments, product only,
2000).
For year over year percentage growth in
Web environments, Mercury Interactive
and Empirix both outpaced the 114 percent
growth of the entire market. Mercury
Interactive grew revenue in the Web load
testing tools and related services space an
impressive 169 percent. Empirix was not
far behind, growing their overall revenue
in the Web space by 164 percent (see figure 6, Market Share Growth for Load
Testing, Leading Vendors in Distributed
Environments, Web). RadView Software
was also nearly on par with overall market
growth, increasing their revenues in the
Web load testing space by 112 percent.

Worldwide Load Testing Market Share

Distributed Enviroments, 2000
Total $263.8 Million
(product & related services, including hosted/remote services)
figure 4

Other 6%
Segue 9%

Compuware 8%
Cyrano 2%

Empirix 8%

Rational 6%

RadView 4%

Mercury Interactive 57%

Source: Newport Group, Inc. 2001

accelerated and tighter integration among the two product

suites has occurred at a more rapid pace. This should
strengthen their competitive position in 2001.

THE INTERNATIONAL EFFECT

In 2000, 70.5 percent of the total load testing market resided
in the United States. From 1999 to 2000, the United States
experienced growth of 64 percent ($154.7M to $253.3M)
and the International markets experienced growth of 79 percent ($59.3M to $106M). For individual vendors, Paris,
France-based CYRANO recorded the greatest percentage of
their total revenue, 47 percent, as being derived from
International markets. Compuware followed with 37 percent and Mercury Interactive attributed 30 percent of their
total load testing revenues to International markets (see figure 7, Worldwide Load Testing Revenue, Distributed
Environments, 2000: U.S. vs. International Markets).

For load testing, Compuware offers QACenter Performance

Edition. Performance Edition provides a script development
workbench consisting of Easyscript modules, which are customized to record application traffic and convert it into
editable load testing scripts. Tool support for multiple Web
and application server protocols includes HTTP/HTTPS,
SSL and digital certificates, FTP, SMTP, POP3, TCP/IP,
UDP and LDAP. Tool support for multiple CORBA-based
middlewares, specifically Inprise, Visibroker, Iona
OrbixWeb and others supported through IIOP as well as
support for Microsoft technologies including DCOM, SQL
Server and MS Exchange are available. For distributed environment development, Compuware supports Oracle (Oracle
SQL*Net and Net8), Sybase (DBLIB and CTLIB) and
TUXEDO (6.3 or higher) as well as support for testing
enterprise Java and wireless applications. For ERP testing,
SAP R/3, Peoplesoft and Oracle implementations can be
tested using Compuware solutions. Additionally,
Compuware provides support for legacy character-based
applications with Telnet VT100-520 support and support for
custom middleware implementations through Windows
Sockets.

VENDOR PROFILES
The following sections provide brief summaries on the load
testing offerings from leading vendors. Each section begins
with one or several company highlights from 2000.
Compuware Corporation (Farmington Hills, MI)
Year 2000 Company Highlight: In 2000, Compuware
merged the development labs for their EcoSYSTEMS product line (production monitoring/management) with the
development labs for their QACenter product line (application testing). As a result, development initiatives have

The load testing technology in the Performance Edition (formally sold individually as QALoad) relies on multi-thread-

Via the PointForward service, Compuware engineers leverage the QACenter load testing technology to drive loads of
hundreds or thousands over the Internet to a customer site
and measure performance and scalability. With
PointForward, customers can opt to have user loads driven
from multiple geographic locations worldwide.

ed virtual users and the use of compiled C scripts to improve

throughput, reduce overheads and make effective use of
machine resources, thus reducing the amount of hardware
needed to run large tests. Performance Edition includes features such as automatic discovery of remote virtual users,
automatic configuration and remote downloading of files
(data, runtime). Any information or file downloading is
done at the beginning of the test to reduce network overheads and avoid impacting the load test itself. Performance
Edition can take activity into account, such as throughput
rates, connection speeds and user think times.

Compuware entered the market for distributed functional

and load testing in 1996 via the acquisition of UK-based
Direct Technologies. They strengthened their position in the
ASQ space in 1997 with the acquisition of NuMega
Technologies. NuMega offers a tool set aimed at defect
management, error detection and code coverage analysis for
Windows. A positive byproduct of Compuwares performance solutions is that they can be used in and across multiple environments. To aid customers who may require outsourced services, Compuware maintains a substantial professional services team of 11,000 plus, about 7 percent of
which focus on Compuwares performance testing and
management solutions for distributed environments.
Compuware services for load testing range from traditional
on-site consulting to recently added hosted/remote services.
With an effective launch of their hosted services in the second half of 2000 and quick uptick in attracting customers,
Compuware should continue to grow its remote load testing
service, PointForward in 2001.

In the QACenter Performance Edition, Compuware has bundled its load testing technology with server analysis agents
and file and data management (File-AID). The server analysis agents monitor each tier of the application infrastructure
during the load test. The technology, derived from
Compuwares production monitoring solution EcoSystems,
provides detailed performance data that is automatically
correlated to load testing results. The combination of load
testing, resource utilization, data management and user
experience monitoring tools work to identify the origin of
performance problems and then provide the drill-down
detail needed to locate and resolve those problems.
In September of 2000, Compuware launched its remote load
testing service, PointForward. It should be noted that in
addition to scalability testing, PointForward includes reliability testing, integrity testing and performance monitoring.

Compuware is publicly traded on the NASDAQ under the

symbol CPWR. A sample of Compuware clients include
Home Depot, IBM, PepsiCo, PlanetRx,
Kmart,
Kohls,
Lycos,
Lante,
CustomTracks, Sandbox.com, Dell UK,
Little Tykes, Xactware, the Detroit
Worldwide Load Testing Tool Market Share
Newspaper Agency, Campbell Soup, Bell
Web Enviroments, 2000
Atlantic, EMC Insurance, Fidelity, Glaxo
Total $216.0 Million (Product Only)
Welcome, NCR, Mass Mutual, and others.
figure 5
Key Compuware business partners
include Microsoft, IBM, Oracle, Dell,
Other 3% Compuware 6%
SAP, Peoplsoft, Baan, Seibel, Citrix,
Segue 9%
Cyrano 1%
Compaq, Sun, and Hewlett Packard
among others. In business since 1973,
Empirix 9%
more information can be viewed at
www.compuware.com.

Rational 5%

CYRANO (Paris, France)

RadView 4%
Mercury
Interactive 63%
Source: Newport Group, Inc. 2001

Year 2000 Company Highlight: The

product is free, the model is open source
and service contracts are reasonably
priced. In 2000, CYRANO is the one vendor that underwent the most dramatic
change to their business model. CYRANO
positioned itself as the first testing vendor

to provide their technology via the

open source business model.
CYRANOs open source solution
offers a core architecture of load
testing and production monitoring
technology to organizations at no
cost.

Market Share Growth for Load Testing

Leading Vendors in Distributed Environments Web

160
In 2000, CYRANO positioned itself
as the first testing vendor to provide
140
their technology via the open source
120
business model. CYRANOs open
source solution, OpenSTA, offers a
100
core architecture of load testing and
production monitoring technology
80
to organizations at no cost. By pro60
viding software for Web performance testing (OpenSTA), database
40
+103%
performance testing and analysis
(Impact and Workbench), database
20
monitoring (Production), network
0 Compuware
and firewall vulnerability testing
Corporation
(NV e-secure), legacy understanding
and testing (Wincap and Test) and
strategic consulting and education
offerings, CYRANO is an end-to-end quality assurance
provider to its customers. CYRANO offers integrated solutions, service and support to companies that want to minimize risk, benchmark Service Level Agreements and enable
Capacity Planning for their IT infrastructures.

figure 6

1998

1999

2000

+169%

+29%

+164%

+54%

Rational
Software

Empirix

Segue
Software

+112%

-8%
Cyrano, Inc

Mercury
Interactive

RadView
Software

Source: Newport Group, Inc. 2001

among the top five vendors in the market space for load testing solutions. In 2000, the company worked aggressively to
expand their testing offerings and added hosted load testing
services, component testing tools and performance monitoring services through a partnership with Internet quality
measurement firm, Gomez, Inc. Based on their performance
and forward-looking company strategies, Empirix should
continue to strengthen their competitive position in 2001.

The United States office of CYRANO, based in

Newburyport, Massachusetts, continues to feel the effects of
re-focusing their sales efforts in the States. As a result, key
executive positions have been shifted from the Paris, France
headquarters to the United States. This effort coincides with
CYRANOs goal to focus on their pervious testing strength,
a strategy they had moved away from several years back.
CYRANO is a public company listed on the EuroNM of the
Paris Bourse (Reuters: CYRA.LN, Sicovam 3922). Formed
in 1989 and publicly traded since 1998, CYRANO is headquartered in Paris, France, with regional headquarters in the
UK and USA. CYRANOs Web site address is www.cyrano.com.
Empirix (Waltham, MA)

Empirix (formerly RSW Software) offers e-Load as part of

its integrated e-TEST suite. e-TEST suite handles functional testing with e-Tester, load/performance testing with eLoad, production monitoring with e-Monitor, advanced test
data collection with ServerStats, analysis and reporting with
e-Reporter and WebReporter, test scheduling/management
with e-Manager and defect tracking with d-Tracker. When
used together, they provide end-to-end use of a single set of
test assets, which ensures consistency in testing throughout
development and deployment of Web applications. The
multiple components that comprise e-TEST suite are tightly
integrated and work well together; they also can be licensed
and used on a stand-alone basis.

Year 2000 Company Highlight: Empirix became an independent company by merging RSW Software with Hammer
Technologies (testing in the communications space), two
former independent business units of Teradyne. The combined technologies from each respective company uniquely
positions Empirix to serve the testing needs of individual
and/or integrated Web and voice applications. Empirix is

Requiring no programming, e-Load utilizes automaticallygenerated Visual Scripts and supports data-driven user loads
for hundreds or thousands of simultaneous virtual users.
The e-Load Scenario Manager lets you define custom load,
stress, and performance scenarios that can be changed on the
fly using different sets of scripts to represent a range of business functions. e-Load leverages TrueLoad technology,

Worldwide Load Testing Revenue

Distributed Environments, 2000
United States ($253.3M) vs. International Markets ($106M)
figure 7

Other
International
Segue
U.S.
Empirix
Rational
RadView
Mercury

based, NSAPI-based, APACHE, NTSE,

as well as all UNIX (including Linux)
and Microsoft Windows platforms. In
addition, ServerStats supports the collection of application server metrics
from BEA WebLogic, IBM WebSphere,
COM+, Sun/AOL/Netscape iPlanet,
Oracle, BroadVision, Allaire Cold
Fusion,
ATG
Dynamo,
Suns
NetDynamics, Apples WebObjects, and
Microsoft Active Server Pages (ASP).
Database support includes Microsoft
SQL Server and Oracle.

Powered by e-Load, e-LoadExpert is a

hosted testing service designed to verify
Cyrano
the performance of Web applications
Compuware
over the Internet. Customers of eIn Millions
LoadExpert can watch test engineers
0
50
100
150
from Empirix design and execute their
From 1999 to 2000, the U.S experienced 64% growth for load testing solutions ($154.7M to $253.3M)
and the International markets experienced growth of 79% ($59.3M to $106M)
load tests online and then work with
them to interpret and analyze results.
Source: Newport Group, Inc. 2001
Empirix launched the eLoadExpert servness functions. e-Load leverages TrueLoad technology, an
ice in October of 2000.
object-level recording method that insulates the user from
minor changes to the HTTP, to emulate real-world browser
For load testing at the component level, Empirix offers
traffic. This tool is sophisticated in its ability to handle freBean-test, a browser-based tool for testing the scalability
quently changing session IDs typical in Web environments,
and integrity of middle-tier applications built with
therefore not producing a host of false test failures that can
Enterprise JavaBeans (EJB). Optimized for leading applihappen as a result of continuously changing site information
cation servers from BEA (WebLogic), IBM (WebSphere)
for dynamic web applications. e-Load also features autoand Bluestone, Bean-test generates multi-threaded Java test
matic content validation so the user is assured that the appliclients to measure performance, scalability and functionalication under test provides the correct responses under load,
ty of EJB component-based applications as workloads
without the need for additional programming.
increase. Most recently, Empirix announced FirstACT, an
application component testing solution that performs load
e-Load allows QA engineers to review snapshots of virtual
and functional testing of Web Services and COM compouser interaction with the application in real time to aid in
nents.
identifying and correcting any scalability problems. e-Load
supports real-time graphing and reporting. Test reporting
Also from Empirix, the Hammer IT Contact Center Test
capabilities (common for e-Load and e-Monitor) allow the
System is an automated solution for acceptance testing and
tool user to layer and view multiple test runs on a single
load testing of contact center applications and systems. The
graph. Reports that display server-side metrics in correlaHammer IT Contact Center Test System works by generattion with client-side metrics are available, along with the
ing thousands of test calls for CTI, IVR and ACD installaability to export test results into Excel style graphs and
tions. Hammer IT duplicates realistic customer calling concharts using a reporting wizard. Summary reports are autoditions, dialing into the systems under test, entering account
matically generated in HTML format.
numbers and PIN numbers, listening to ensure that the right
prompt responses are being played, and then measuring syse-Load ServerStats collects metrics that are tied to specific
tem and network response times throughout each test call. If
components of a complete Web infrastructure. The
response times exceed pre-established thresholds or if any
ServerStat component of e-TEST suite supports collection
other problem is found, the Hammer IT logs the error and its
of metrics from multiple Web, application and database
location in the call flow to a database and displays test
server environments running multiple operating systems. It
results in real-time. Hammer IT provides standard alerting
supports the collection of Web server metrics from ISAPIcapabilities.

into their environments. Additionally, LoadRunner is integrated with Mercury Interactives application performance
monitoring and management technologies, allowing
LoadRunner scripts to be re-used to monitor application performance once an application has been deployed into production. In addition to LoadRunner, Mercury Interactive
also offers Astra LoadTest as part of the Astra suite of testing tools. Based on Visual Basic, Astra LoadTest is a lighter
and subsequently less expensive load testing tool, requiring less coding for script development and focused on Web
applications only. For either tool, load scenarios and results
can be saved to Mercury Interactives TestDirector repository for ongoing scenario reuse, test management and defect
tracking.

A sample of Empirix customers includes American Airlines,

British Telecom, Countrywide, Sprint PCS, MCI Worldcom,
Computer Sciences Corporation, Peoples Bank, Manulife
Financial, H&R Block, and Royal Bank of Canada. Partners
include BEA, ATG, BroadVision, Compaq, Microsoft,
PWC, AnswerThink and Sapient, among others. Empirix is
the combination of RSW Software and Hammer
Technologies, both formerly units of Teradyne, Inc., which
became an independent company in December, 2000. More
information can be viewed at www.empirix.com.
Mercury Interactive (Sunnyvale, CA)
Year 2000 Company Highlight: Mercury Interactive significantly increased their market share for load testing,
gaining 14 percent in distributed environment testing. The
company was added to the NASDAQ-100 index and cemented a strategic partnership with Akamai, a leading provider
of distributed application and content delivery services.
Mercury Interactive strengthened their performance monitoring/management offerings, part of which included the
acquisition of Freshwater Software in 2001. Based on their
impressive performance, Mercury Interactive continues to
shape and dominate this market space.

Mercury Interactives overall market strength has moved

companies such as Compaq, Dell, Data General, IBM SPC,
IBM AS/400 Benchmark Center, Informix, Intel, HP,
Microsoft, Oracle, Peoplesoft, Sequent, Sybase, and Unisys
to rely on LoadRunner in their benchmark and competency
centers. Further, e-business vendors such as BroadVision,
I2, Vignette, CacheFlow, Allaire Corporation, Ariba, Inc.,
Bluestone Software, Calico Commerce, Inc., Gemstone
Systems, Inc., nCipher, Microsoft, SilverStream, Symantec
and Vision Software have standardized on LoadRunner in
their benchmark and performance centers.

Primarily addressing the testing needs of distributed environments, Mercury Interactives LoadRunner is an enterprise level load testing tool that measures and predicts system behavior and performance. The tool exercises an entire
enterprise infrastructure by emulating thousands of users to
isolate problems, optimize performance and accelerate
deployment. LoadRunners Controller acts as a central point
of control in order to organize, execute and manage load test
scenarios. The Controller executes load tests by running and
synchronizing combinations of business processes with
multiple types of virtual users concurrently. As the virtual
users run and emulate the application clients, the Controller
uses its integrated real-time monitors to provide performance information on all parts of the infrastructure under load
in order to help isolate the bottlenecks. Performance monitors are available for the most popular web servers, application servers, databases, network components and firewalls.
For instance, an enterprise application can be tested with
hundreds or thousands of desktop, terminal, Web and Java
clients to determine how Java users affect a system as they
access an ERP application over the Web.

LoadRunner offers support for HTTP, XML, SOAP, WAP, iMode, CORBA, DCOM, plus many more protocols and
technologies. LoadRunners virtual users can run on several different platforms including Windows NT and 2000,
Solaris, AIX, HP-UX and Linux. In addition, Mercury
Interactives LoadRunner includes online performance monitors, allowing identification and isolation of performance
problems during the load test. Supported performance monitors are available for Apache, IIS, iPlanet, WebLogic,
WebSphere, SilverStream, Ariba, ATG Dynamo, Oracle,
SQLServer, DB2 and more.
Mercury Interactive also offers ActiveTest a hosted-load
testing service. ActiveTest is driven by Mercury
Interactives LoadRunner, and is designed to verify the performance of Web applications over the Internet. Leveraging
the current hosted infrastructure, the ActiveTest Service can
drive transaction load simulations of over 2 million concurrent users or an approximate Web server load of three billion
hits per day. ActiveTest results reveal which areas of the
application are in need of tuning to achieve better performance. Based on customer observations and summary performance test reports generated by the ActiveTest Service,
customers can assess their application weaknesses and make
improvements to their systems.

Using LoadRunner, load tests can be run against any server,

regardless of platform or geographic location. LoadRunner
can be used to load test applications running over many different architectures including Web, Wireless, Streaming,
ERP and CRM applications. The tools open API allows
users and/or third-party vendors to integrate LoadRunner

Publicly traded on the NASDAQ under the symbol MERQ,

Mercury Interactive is listed on the Standard and Poor 500
index and the NASDAQ 100. A sample of Mercury
Interactives customers include Bank of America, E*Trade,
Gap Inc., Citibank, Barnsandnoble.com, PHH Vehicle
Management Services, Sabre Group, Ford Motor Company,
COVISINT, Lufthansa, JCrew.com, and Prudential among
thousands of others. A sample of Mercury Interactives partners include Andersen Consulting, Ernst & Young,
Microsoft, BEA, iPlanet, marchFIRST, BroadVision,
Allaire, IBM Global Services and many others. In business
since 1989, more information can be viewed at www.mercuryinteractive.com.
RadView Software (Burlington, MA)
Year 2000 Company Highlight: RadView
Software experienced a respectable growth
rate of 112 percent and put aggressive
plans in place to expand their product line
beyond load and integrity testing. In 2001,
they have begun to deliver new testing
solutions, specifically a functional testing
tool. RadView also opened offices in the
UK and Germany and forged partnerships
in an effort to extend their international
presence to Asia-Pacific.

Cruise Control, which is a predictive tool feature for automatically measuring performance thresholds and determining concurrent usage limits. WebLoad runs on any
Microsoft Windows workstation and/or server and on the
Linux and Sun Solaris platforms, Solaris version 2.6 or
higher.

In early 2000, RadView introduced WebLoad Resource

Manager. WebLoad Resource Manager is essentially the
productization of RadViews vision of bridging the gap in
responsibilities between different IT departments, such as
development and quality assurance, within an organization.
Acting as a central point of administration, WebLoad
Resource Manager is able to define and
administer testing activities, such as staThe overall market for
tus and scheduling across a single department or an entire enterprise. The high
load testing tools and
level benefit lies in the fact that the many
related services in distribstages and departments involved in the
uted environments
development life cycle standardize on a
reached $359.3 million in
single solution to promote an easy
exchange of information across depart2000 and Newport Group
mental and/or geographic lines.
estimates that number

will grow to $879.2 million

in 2004.

Formally in business since 1997, RadView continues to

grow their business due to their focus on testing Web applications. The Companys cornerstone product, WebLoad,
generates virtual user loads of hundreds or thousands to simulate real-world traffic and measure and monitor application
performance, scalability and integrity. Test scripts are generated in JavaScript and are customizable, which provides
flexibility for a range of simple to complex testing scripts.
Likewise, WebLoad requires its user to have experience in
JavaScript programming.
RadViews differentiator is their focus on combining load
testing with application integrity testing coupled with their
DOM-centric approach. To accomplish integrity testing,
RadView opted to focus their testing approach on the DOM
(Distributed Object Model). So in essence, WebLoad focuses on the integrity of data and is not focused on the presentation of the GUI interface. As a reminder, the DOM is a
W3C specification that evolved to bridge competing object
models from Netscape and Microsoft. The DOM works
toward ensuring that Web pages are presented in the same
manner, independent of the browser that displays them. In
addition to centering their approach to Web testing on the
DOM, WebLoad incorporates support for HTTP/HTTPS,
Java, ActiveX, WAP, XML, as well as a feature named

10

The Company raised $37 million through

an initial public offering completed in
August 2000. RadView has more than
1,000 customers worldwide including:
Fidelity, American Express, eBay, Anheuser-Busch,
Vanguard, Sun Microsystems, Autodesk, and Nortel
Networks. More information can be viewed at www.radview.com.
Rational Software (Cupertino, CA)
Year 2000 Company Highlight: Rational Software is
among the top five vendors in the load testing space once
again for 2000. Rational integrated their performance testing software, formerly Rational Suite PerformanceStudio,
into the Rational Suite TestStudio in order to offer customers
a wide range of testing solutions within a single product
suite.
Through tools, best practices and services, Rational helps its
customers overcome a dilemma they call the software paradox - the need to deliver higher quality software in less time.
Rational Suite TestStudio helps companies address the challenges of the e-software paradox by providing an integrated
automated software quality (ASQ) solution that tests all
dimensions of quality, including functionality, reliability
and performance.
Rational Suite TestStudio includes a complete set of auto-

ing, consulting and technical support services. Rational

Software is publicly traded on the NASDAQ under the symbol RATL. More information can be viewed at www.rational.com.

mated testing and defect tracking tools, as well as the

Rational Team Unifying Platform, to help teams work more
productively by providing common access to project
requirements, change requests, test assets, and more.
Rational Suite TestStudio is supported by Rational e-development services to help accelerate application development
and implementation.

Segue Software (Lexington, MA)

Year 2000 Company Highlight: Segue Software realized
respectable growth for load testing tools and related services. The Company released an updated version of
SilkPerformer in December of 2000, enabling it to support
Microsofts DCOM architecture and enhancing the products security and reporting features.

Rational Suite TestStudio provides the three levels of diagnostic information needed to isolate the cause of performance bottlenecks by analyzing the business transactions,
low-level client calls, and system resources that are causing
poor performance. For example, performance tests with
Rational Suite TestStudio helps users decide when to tune
the back-end server by increasing system memory or CPU
speed. It also identifies specific areas of client, middle-tier,
or server-side code that are causing performance problems.

Segue Software offers SilkPerformer, an integrated Web,

application and database server testing tool. By simulating
high-volumes of Internet and database traffic, SilkPerformer
can create realistic models to test Web, client/server, and
Tools within Rational Suite TestStudio include the followERP applications. With the tools e-business focus,
ing: Rational Robot for functional testing
SilkPerformer is designed to provide load
of the entire application; Rational
testing capabilities across a number of ePureCoverage to identify untested code
business-related technologies including,
Load testing solutions
and provide code-coverage analysis;
HTML, XML, Java, JavaScript, ActiveX
account for 43.6% of the
application profiling; Rational Purify to
and browsers. Additionally, SilkPerformer
overall market for autolocate hard-to-find run-time errors;
offers support for the many Web and appliRational Quantify to pinpoint performcation server protocols such as HTTP,
mated software quality
ance
bottlenecks;
and
Rational
(ASQ) solutions in distrib- HTTPS, LDAP, SMTP, POP3, FTP, IIOP
TestFactory to detect run-time errors withand WAP , as well as tool support to test
uted
environments.
out user assistance and generate optimal
streaming media.
scripts for regression testing.
SilkPerformer leverages its multi-machine
With Rational Suite TestStudio, testers are unified with the
controller technology to balance its simulated user loads
rest of the team by providing them use of the same tools,
across any number of load-driving machines, resulting in the
methods, data and metrics as used by developers. By being
ability to conduct very robust load tests. In addition,
integrated with the Rational Suite Team Unifying Platform,
SilkPerformer combines multiple transaction types and varRational Suite TestStudio helps teams work more producied user connection speeds to run against the web applicatively by providing common access to project requirements,
tion, while automatically randomizing user think times.
change requests and test assets.
With support for SSL, SilkPerformer is able to conduct
secure recording to gauge the impact that securing an eTools within the Rational Team Unifying Platform include
business system has on overall performance. SilkPerformer
Rational TestManager, for managing all test activities and
utilizes a Results Repository that can store load test inforartifacts; Rational Unified Process, providing browsermation for analysis of performance factors such as scalabilaccessible knowledge prompting the team to use developity, SQL structure execution, Web server throughput, and
ment best practices; Rational RequisitePro to enable teamconcurrency. By storing information in the Results
based requirements management; Rational ClearQuest to
Repository, performance factors can be modeled and anadeliver integrated defect tracking and change request manlyzed to assist in performance and business decisions.
agement; Rational ClearCase LT to control artifacts that
evolve over the course of development; and Rational SoDA
Segue offers a range of consulting services from
to automatically generate project documentation.
eConfidence programs and performance audits to full technical implementations of automated test services, including
All Rational products are supported by a worldwide service
newly added hosted load testing services. Drawing on over
and support organization, which offers professional educaa hundred e-business consulting engagements, Segues
tion and training, customized services and projects, mentoreConfidence Programs deliver the testing expertise that

11

helps companies to define and achieve their specific e-business reliability requirements. Segues eConfidence programs include: eConfidence Functionality, eConfidence
Performance and eConfidence Production. Consulting customers include Bank of America, British Airways, Chase
Manhattan Bank, Citibank, Compaq, Dell, Discover
Brokerage, eCircles.com, Fidelity Investments, Hotmail,
Infoseek, Morningstar, Netscape, Quote.com, Rueters and
Tickets.com among others. Most recently, Segue announced
eConfidenceScale (eCScale). Powered by SilkPerformer,
eCScale is an expanded set of hosted load testing services
designed to provide performance and scalability assessments of e-business applications. Hosted by Navisite,
Segues eCScale service can be delivered via a hosted load
testing service or through traditional on-site consultations.
Segue is publicly
traded on the
NASDAQ under
the symbol SEGU.
Key Segue clients
for SilkPerformer
include Fidelity
Investments, eBay,
Sun Microsystems,
Charles Schwab,
AOL/Netscape,
Dell, State Farm, Merrill Lynch, Sears.com and Telcordia.
Key partners include marchFIRST, PriceWaterhouse
Coopers, Razorfish, Systest Labs, StarQuality, BMC
Software and IBM Global Services, among others. In business since 1988, more information can be viewed at
www.segue.com.

In 2000, the market for load

testing tools and related services in distributed environments grew 68%. The subsegment of the market
focused on the Web realized
an increase of 114%.

NEW MARKET ENTRANTS

As the market for load testing expands, Newport Group continues to see emerging companies introduce solutions.
Some are existing companies that have added load testing to
their current offerings. Others are newly formed companies
aimed at catering to smaller businesses or touting new
approaches to the mass market. What they all have in common is that they are companies trying to capitalize on a
growing market. Market share for the following companies
is relatively small in comparison to other vendors profiled in
this report and therefore falls into the other category of the
market share graphs contained in this report.
Atesto Technologies, Inc. (Fremont, CA)
During 2000, hosted testing services gained increased
acceptance and Atesto Technologies formally entered the
automated testing market with a range of Web-focused test-

12

ing services. Organized as a service suite, Atesto offers

hosted testing services in the following areas:
Atesto Function Checker: Tests and validates complex,
static and dynamic Web applications.
Atesto Load Modeler: Facilitates capacity planning and
management by measuring the response of Web sites under
load conditions from a globally distributed test infrastructure.
Atesto Response Watch: Monitors the response of a live
Web site over time from the perspective of the end customer.
Automated alerts notify site administrators when customizable service level thresholds are compromised.
As it relates to the focus of this report, the Atesto Load
Testing service utilizes a Web-based interface to manage the
performance testing process. The Atesto ScriptRecorder
allows automatic generation of scripts that can be directly
executed from one-to-any of the fifteen geographic locations where Atesto servers are located. The recorder is a
Windows based thin client that captures the sequence of
browser clicks constituting a transaction or business process
and translates it into a test script. Scripts can be further customized to add functions like checkpoints, user think times
and variable specifications for data driven testing. The
Atesto Load Test system can recreate complex Web application
In 2000, the market for
usage
scenarios
load testing tools and
involving
multiple
related services grew by
types business transac64% in the United States
tions executing at
varying connection
and 79% Internationally.
speeds, incorporating
dynamic session variables, embedded frames, pop-up windows and more. The
load test process can also be customized for ramp-up and cool down patterns.
Once the script is written and uploaded to the Atesto servers,
load tests can be scheduled and executed (following routine
security checks) with hundreds or thousands of virtual users
via the Atesto Web site. Real time graphical and tabular
report formats provide a detailed view of server and application performance under load conditions. An executive
summary report provides a condensed view of the most
important test results. Detailed reports are available for
response times (for each script, location and access type
combination), transaction summaries (passes and failures
for each transaction), server metrics (page views, throughput and number of hits), and failure analysis (failure type
classification and raw failure data for every script). Detailed
data on location and the Internet backbone is also available.

Prior to test execution, WebAvalanche provides the functionality for users to define profiles of the types of simulated users that are to be generated. This solution provides
flexibility for profiling simulated users at the network level,
ensuring that tests do not operate from the assumption of
clean LAN traffic only. For example, connection profiles
can be specified to vary network latency, packet loss rates
and network noise for each simulated
user. The fashion in which simulated
users are to be routed through a web
million in overapplication infrastructure can be specall load test tool and related
ified. With WebAvalanche, simulated
web traffic is configured and generatservices revenue for distribed at an aggregate level in contrast to
uted environments, 73.4% or
detailed scripting at a per simulated
$263.8 million was gained as a user level, resulting in less overhead in
result of the need to test Web
test set-up and script maintenance.

Understanding that Atesto is new to the automated testing

space, company historical performance information is not
available. Nonetheless, Atesto recently (April 2001) closed
on $15 million of second round financing. As a result of this
funding, Atesto plans to explore partnership and growth
opportunities. Atesto is privately held, with venture funding
from the Sprout Group, Labrador Ventures, Draper
Richards, Argus Capital, Wilson Sonsini
Law Group and others. Additional information can be found on their Web site at
Of the $359.3
www.atesto.com.
Caw Networks, Inc.
(Mountain View, CA)

Founded in 1999, Caw Networks bills

itself an Internet infrastructure company.
Caw Networks builds appliances
application environments, a
Simulated users are drawn from a
designed to help users strengthen and
15% increase over 1999.
large number of source IP addresses
scale the Internet.
The companys
and user behaviors such as think times
WebAvalanche product produces largebetween page requests and HTTP
scale stress tests and executes them
aborts can be randomly varied and weighted.
against a complete Web application infrastructure. Caw
WebAvalanche has a graduated load feature that allows that
Networks differentiates itself by taking an appliance-based
load to be added incrementally to determine the effects on
approach to load testing, offering a sophisticated hardware
system performance. In addition, WebAvalanche automatiappliance able to generate very high volumes of Web users
cally invokes ramp-down intervals upon reaching the test
from a single machine. The WebAvalanche appliance stressduration limit or if a user manually stops the test. This autoes both the application and network layers of a Web applimatic ramp-down feature is important for monitoring
cation infrastructure simultaneously.
whether connections close appropriately, ensuring the most
efficient turnover for computing resources and thus a highBuilt with best-of-breed hardware, WebAvalanche can be
er level of system performance.
used to build and execute large-scale load tests to define
traffic handling capabilities and aid in capacity planning.
Upon test execution, WebAvalanche cycles through a list of
WebAvalanche is a rack-mounted 4U high hardware device
site URLs on a per simulated user basis. As simulated users
build primarily on Intel-based technologies. The
are generated, the load generator automatically handles both
WebAvalanche appliance sits just outside of the web infrasession and persistent-based cookies. WebAvalanche also
structure, which may consist of routers, firewalls, load balsupports SSL, HTML forms and HTTP POSTS, with
ancers, web, application and database servers, etc.
upcoming support for distributed mode operation and other
protocols. Finally, following test execution detailed reportThe Web Avalanche load generator consists of a high pering per URL and detailed TCP/HTTP analysis is delivered.
formance micro-kernel operating system, proprietary
More information on Caw Networks can be found on their
TCP/IP stack and an optimized network driver. A single
website at www.caw.com.
WebAvalanche appliance is able to maintain up to 1 million
simultaneous connections, simulating in excess of 10,000
HTTP GETs per second. WebAvalanche can serve as a comDieselpoint, Inc. (Chicago, IL)
plement to script-based regression testing tools. Regression
tools could be leveraged to run limited instances of scripts
As another new entrant, Dieselpoint, Inc., is the creator of
to test specific features and functions of an application while
Dieseltest, a low cost windows application that simulates
WebAvalanche stressed that same application under very
hundreds or thousands of users hitting a Web site. Designed
high loads. This combination would ensure that an applicato run on client machines, Dieseltest is priced at $99 per
tions functional integrity was not compromised under
CPU. More information, and a demo version designed to
extreme load conditions.
support 25 users, is available from their Web site at
www.dieseltest.com

13

users to automatically compile,

deploy, and test back-end programs
The average expenditure for an
and related output pages; prompting
automated load testing tool was
users to take advantage of testing early
Last year, Keynote made its foray into
$33,000,
down
33%
from
1999.
in order to spot critical problems
the load testing market with the acquibefore they become widespread.
sition of Velogic (May, 2000). As a
Integrating WebKing into the Web
compliment to Keynotes performance
application development process works to prevent site
management services, Velogics load testing technology was
errors, increase site reliability and improve the efficiency of
absorbed into the Keynote offering under the name
the Web development process. The Parasoft Web site can be
KeyReadiness. KeyReadiness is a hosted load testing and
found at www.parasoft.com.
performance management service. More specifically,
KeyReadiness was organized into 3 separate performance
softlight technologies, inc. (Hamilton, NJ)
levels. KeyReadiness SilverTest is a low cost introductory
version of the load testing service. It includes a standard
Founded in 1999, softlight delivers SiteTools 3.0, a suite of
testing plan, a load test of up to 6000 sessions per hour,
performance testing tools. In addition to a Web site monibasic performance reports and returned results within two
toring tool, the SiteTools Suite includes the SiteTools
days. The KeyReadiness GoldTest is more robust, including
Loader, a performance-testing tool designed to generate
a more comprehensive test plan, a load test of up to 30,000
load against a Web server by recording and running scripts
sessions per hour, and in-depth reports. Results for this
with any number of simulated users with differing data
level of service are returned within three to seven days.
parameters. Scripts are customizable with the use of an
Lastly, the Key-Readiness Platinum Test provides expanded
included editor.
load testing, geographic profiling of users, up to 60,000 sessions per hour, advanced reporting that includes bottleneck
The SiteTools Loader is comprised of a MainConsole,
analysis, session information and lost revenue analysis.
UserManager, DynamicUser, Recorder and an Editor. The
MainConsole allows for the loading and running of tests.
What is unique about the KeyReadiness service is that it can
All scripts are validated during the load process in order to
provide revenue projections vs. load projections to deterensure that any manual editing or script custimization has
mine lost revenue as a result of performance degradation. It
not affected the validity of the scripts. If any script fails valis understood that certain assumptions are made when these
idation, a detailed message is displayed showing the reason,
tests are designed, however, the projections still offer a
the script file, and the line number of the failure. A test can
unique performance perspective.
be stopped at any point and restarted if needed. Once a test
is completed, a comprehensive set of reports is created in
Recent Keynote headlines (July, 2001) announced the acquiHTML format. Users can also choose to create a database
sition of Envive Corporation. Envive, based in Mountain
that collates all user response information for reporting purView California, was essentially a competitor to Keynote
poses. The UserManager handles the management of all
that offered geographically dispersed self-serve load testing
simulated users on a machine. It is responsible for the creservices via the Management Service Provider business
ation of all users, as either a separate thread or a separate
model. With the acquisition, Keynote changed the name of
process. Besides user creation, it routes all commands and
the Envive service to Test Perspective. Based in San Mateo,
messages to the appropriate destination. The DynamicUser
California, more information can be found at
handles the actual user simulation. It runs all requests, valwww.keynote.com.
idates the response, and logs all relevant information about
the request.
Parasoft Corporation (Monrovia, CA)
Keynote Systems
(San Mateo, CA)

Founded in 1987, Parasoft released the load testing functionality of its WebKing 3.0 product in November of 2000
to compliment its error prevention and error detection suite
of tools. WebKing is a comprehensive development, testing, and management tool designed specially for dynamic
Web site developers and testers. WebKing automatically
exposes load, construction, functionality, presentation, content, and design problems and requires no script development. WebKing also provides an infrastructure that allows

14

The average virtual user license for a load

testing tool is 469, down 22% from 1999.

The Recorder allows query scripts to be recorded from user

actions performed on a Web site. The recorder scripts can
then be edited and saved. Scripts can be recorded in two
ways. The first uses the browser to record scripts by way of
the actual high-level calls made to the requested URL, links
called by redirection, or links called to fill a frame within a
frameset. Internal links to images, style sheets or other types
of internal links are not recorded and will be parsed out of
the returning HTML and called during playback.

WebmasterSolutions (San Diego, CA)

Recognized as a smaller player in the automated testing

market, Web Performance Inc. develops and markets Web
Performance Trainer, a cross-platform load testing tool.

Businesses are steadily increasing their investments into

quality improvement. Newport Groups annual research
into Web application integrity, conducted in
November/December of 2000, reveals that on average
Global 2000 businesses are investing $250,000 into testing
tools, $237,500 into the hardware necessary to support an
adequate test environment, $36,000 into outsourced testing
services and slightly more than $790,000 per year for staff
and training. Newport Group expects that these investments
will continue and that, specifically, the market for automated load testing solutions will continue to grow, but at a slower pace given continued market expansion.

WebmasterSolutions provides outsourced monitoring and

load testing services. For load testing, WebmasterSolutions
offers SiteStress on a subscription-basis. With the SiteStress
service, customers have the option to load test up to three
times within a 15-day window of engagement. Once tests
are complete, WebmasterSolutions engineers will analyze
the results to determine what might be causing failures or
bottlenecks. Comparing results to similar systems,
The second way to record a script
WebmasterSolutions engineers will then
Load testing solutions account
involves setting up the recorder to
make appropriate recommendations on
improving performance or fixing probact as a proxy. This will record all
for 35.2% of the overall market
HTTP traffic being sent to and from
lems. Current clients range from online
for automated testing solutions.
retailers to online bank. Imformation
the Web browser and the Web server. The SiteTools Recorder will record HTTP traffic sent by
about WebmasterSolutions and SiteStress can be found at
any ActiveX control or JAVA applet residing in the currentwww.webmastersolutions.com
ly displayed page. At this time, HTTP is the only protocol
that can be recorded through the proxy recorder. Queries
THE BOTTOM LINE
that use HTTPS must be recorded through the Recorder
browser.
As witnessed by market growth projections and the amount
of new players entering the market space, the demand for
SiteTools 3.0 is packaged and sold in three different ways.
solutions that can define the limits of distributed application
Customers can opt for the Developer Edition, which
infrastructures and work to maintain optimal performance is
includes a license for one console and 50 virtual users; the
on the rise. Businesses clearly recognize the value in reducEnterprise Edition, which includes a license for one console
ing the risk of failures and poor performance and are
and unlimited virtual users; or the SiteTools Suite, which
increasingly leveraging solutions that deliver them control
includes one license for SiteTools Enterprise Edition and ten
over the performance, efficiency and growth of their distriblicenses of SiteTools Developer Edition.
uted application infrastructures. Clearly the Web and
strengthening business focus on electronic commerce is
A downloadable trial version of SiteTools, and more infordriving market growth, amplifying the requirement to
mation, can be found at www.softlight.com.
employ solutions that define and continually manage both
the integrity and the performance handling capabilities of
Web-based application infrastructures.
Web Performance, Inc. (Research Triangle Park, NC)

Designed for medium and small development and testing

teams, Web Performance Trainer delivers ease of use, while
simulating up to 10,000 or more virtual users. It also provides features such as modem bandwidth simulation, load
balancing of virtual user engines, form data replacement,
SSL, etc. Web Performance Trainer has been shipping since
May of 2000, and is available on a trial basis as a download
from their site at www.webperformanceinc.com.

All company names and products mentioned herein are registered trademarks of their respective companies.

15

About Newport Group

Newport Group is an independent information technology research firm. Founded in 1997, Newport
Group was created with the intent and interest to provide detailed research services to major
corporations and software vendors that share a vital interest in information technology. Drawing on
years of IT research experience, Newport Group concentrates on selective IT research topics and
trends. The benefits of this exclusive research approach are passed on to the client base with
demonstrated expertise and insight.
The information contained in any and all final written materials by Newport Group will be based on accurate reliable information.
However, because Newport Group information is gathered through numerous sources, including third parties, we cannot warrant
that the final publication is complete and error-free. Market statistics reported in this document are based on Newport Group
estimates and analyst opinions. Newport Group disclaims all implied warranties, including, without limitation, warranties of
merchantability of fitness for a particular purpose. Newport Group or its affiliates shall have no liability for any direct, incidental,
special or consequential damages or lost profits.

Copyright 2001 Newport Group, Inc. Reproduction is forbidden unless authorized

Additional copies may be obtained by contacting 1-508-362-9799
All company names and products mentioned herein are registered trademarks of their respective companies.

3230 Main Street, P.O. Box 909, Barnstable, Massachusetts 02630

Telephone: 508-362-9799 www.newport-group-inc.com Fax: 508-362-5229