Debugging & testing

Testing is the precursor to debugging. Testing is commonly the

forte of programmers and advanced users, and occurs when a
product is new or is being updated and needs to be put through
its paces to eliminate potential problems. Testing identifies
bugs or imperfections so that they can be corrected in the
debugging process, before the [next] official release of the
product. These unofficial releases are known as beta releases
(e.g. 3.0b), and public volunteers are known as beta testers.

Beta testing is a valuable resource for software developers

because of the varied computer systems participating, combined
with the sheer number of hours and scenarios under which the
program is used. This flushes out unforeseen problems in a way
that can’t be effectively achieved using in house debuggers
only. The beta testing phase gives authors a good idea of the
readiness of a product for the public domain.

Debugging is the forte of programmers and developers, and

involves fixing the very code of the software to eliminate
errors or bugs. Developers attempt to replicate beta-reported
bugs on in house systems for the purpose of eliminating them.

While there are many types of debugging tools, a simple

example is a tool that allows the programmer to monitor
program code while manipulating it to execute various commands
and routines. One basic approach is to simplify the code as
much as possible at the suspected troubled spot, while still
replicating the problem, narrowing the focus to potential
problem lines. In reality, debugging is a complex process that
requires different approaches based on factors such as the
complexity and length of the software code itself, and the
language it’s written with.

2. Debugging involves testing and fixing the programming

errors. Testing is the first step for identifying the errors.
Debugging is testing, fixing and retesting till it performs
the designed task well.

Q) Performance testing

The goal of performance testing is not to find bugs, but to

eliminate bottlenecks and establish a baseline for future
regression testing. To conduct performance testing is to
engage in a carefully controlled process of measurement and
analysis. Ideally, the software under test is already stable
enough so that this process can proceed smoothly.

A clearly defined set of expectations is essential for

meaningful performance testing. If you don't know where you
want to go in terms of the performance of the system, then it
matters little which direction you take. For example, for a
Web application, you need to know at least two things:

* expected load in terms of concurrent users or HTTP

connections
* acceptable response time

Once you know where you want to be, you can start on your way
there by constantly increasing the load on the system while
looking for bottlenecks. To take again the example of a Web
application, these bottlenecks can exist at multiple levels,
and to pinpoint them you can use a variety of tools:

* At the application level, developers can use profilers

to spot inefficiencies in their code (for example poor search
algorithms)
* At the database level, developers and DBAs can use
database-specific profilers and query optimizers
* at the operating system level, system engineers can use
utilities such as top, vmstat, iostat (on Unix-type systems)
and PerfMon (on Windows) to monitor hardware resources such as
CPU, memory, swap, disk I/O; specialized kernel monitoring
software can also be used
* At the network level, network engineers can use packet
sniffers such as tcpdump, network protocol analyzers such as
ethereal, and various utilities such as netstat, MRTG, ntop,
mii-tool

From a testing point of view, the activities described above

all take a white-box approach, where the system is inspected
and monitored "from the inside out" and from a variety of
angles. Measurements are taken and analyzed, and as a result,
tuning is done.

However, testers also take a black-box approach in running the

load tests against the system under test. For a Web
application, testers will use tools that simulate concurrent
users/HTTP connections and measure response times. Some
lightweight open source tools I've used in the past for this
purpose are ab, siege, httperf. A more heavyweight tool I
haven't used yet is OpenSTA. I also haven't used The Grinder
yet, but it is high on my TODO list.
When the results of the load test indicate that performance of
the system does not meet its expected goals, it is time for
tuning, starting with the application and the database. You
want to make sure your code runs as efficiently as possible
and your database is optimized on a given OS/hardware
configurations. TDD practitioners will find very useful in
this context a framework such as Mike Clark's jUnitPerf, which
enhances existing unit test code with load test and timed test
functionality. Once a particular function or method has been
profiled and tuned, developers can then wrap its unit tests in
jUnitPerf and ensure that it meets performance requirements of
load and timing. Mike Clark calls this "continuous performance
testing". I should also mention that I've done an initial port
of jUnitPerf to Python -- I called it pyUnitPerf.

If, after tuning the application and the database, the system
still doesn't meet its expected goals in terms of performance,
a wide array of tuning procedures is available at the all the
levels discussed before. Here are some examples of things you
can do to enhance the performance of a Web application outside
of the application code per se:

* Use Web cache mechanisms, such as the one provided by

Squid
* Publish highly-requested Web pages statically, so that
they don't hit the database
* Scale the Web server farm horizontally via load
balancing
* Scale the database servers horizontally and split them
into read/write servers and read-only servers, then load
balance the read-only servers
* Scale the Web and database servers vertically, by adding
more hardware resources (CPU, RAM, disks)
* Increase the available network bandwidth

Performance tuning can sometimes be more art than science, due

to the sheer complexity of the systems involved in a modern
Web application. Care must be taken to modify one variable at
a time and redo the measurements, otherwise multiple changes
can have subtle interactions that are hard to qualify and
repeat.

In a standard test environment such as a test lab, it will not

always be possible to replicate the production server
configuration. In such cases, a staging environment is used
which is a subset of the production environment. The expected
performance of the system needs to be scaled down accordingly.
The cycle "run load test->measure performance->tune system" is
repeated until the system under test achieves the expected
levels of performance. At this point, testers have a baseline
for how the system behaves under normal conditions. This
baseline can then be used in regression tests to gauge how
well a new version of the software performs.

Another common goal of performance testing is to establish

benchmark numbers for the system under test. There are many
industry-standard benchmarks such as the ones published by
TPC, and many hardware/software vendors will fine-tune their
systems in such ways as to obtain a high ranking in the TCP
top-tens. It is common knowledge that one needs to be wary of
any performance claims that do not include a detailed
specification of all the hardware and software configurations
that were used in that particular test.

Load testing

We have already seen load testing as part of the process of

performance testing and tuning. In that context, it meant
constantly increasing the load on the system via automated
tools. For a Web application, the load is defined in terms of
concurrent users or HTTP connections.

In the testing literature, the term "load testing" is usually

defined as the process of exercising the system under test by
feeding it the largest tasks it can operate with. Load testing
is sometimes called volume testing, or longevity/endurance
testing.

Examples of volume testing:

* testing a word processor by editing a very large document.
* testing a printer by sending it a very large job.
* testing a mail server with thousands of users mailboxes.
* A specific case of volume testing is zero-volume testing,
where the system is fed empty tasks

Examples of longevity/endurance testing:

* testing a client-server application by running the client in

a loop against the server over an extended period of time.

Goals of load testing:

* expose bugs that do not surface in cursory testing, such as

memory management bugs, memory leaks, buffer overflows, etc.
* ensure that the application meets the performance baseline
established during performance testing. This is done by
running regression tests against the application at a
specified maximum load.

Although performance testing and load testing can seem

similar, their goals are different. On one hand, performance
testing uses load testing techniques and tools for measurement
and benchmarking purposes and uses various load levels. On the
other hand, load testing operates at a predefined load level,
usually the highest load that the system can accept while
still functioning properly. Note that load testing does not
aim to break the system by overwhelming it, but instead tries
to keep the system constantly humming like a well-oiled
machine.

In the context of load testing, many important bugs simply do

not surface unless you deal with very large entities such
thousands of users in repositories such as LDAP/NIS/Active
Directory, thousands of mail server mailboxes; multi-gigabyte
tables in databases, deep file/directory hierarchies on file
systems, etc. Testers obviously need automated tools to
generate these large data sets, but fortunately any good
scripting language worth its salt will do the job.

Stress testing

Stress testing tries to break the system under test by

overwhelming its resources or by taking resources away from it
(in which case it is sometimes called negative testing). The
main purpose behind this madness is to make sure that the
system fails and recovers gracefully - this quality is known
as recoverability.

Where performance testing demands a controlled environment and

repeatable measurements, stress testing joyfully induces chaos
and unpredictability. To take again the example of a Web
application, here are some ways in which stress can be applied
to the system:

* double the baseline number for concurrent users/HTTP

connections.
* Randomly shut down and restart ports on the network
switches/routers that connect the servers (via SNMP commands
for example)
* take the database offline, and then restart it.
* rebuild a RAID array while the system is running
* run processes that consume resources (CPU, memory, disk,
network) on the Web and database servers.

However, stress testing does not break the system purely for
the pleasure of breaking it, but instead it allows testers to
observe how the system reacts to failure. Does it save its
state or does it crash suddenly? Does it just hang and freeze
or does it fail gracefully? On restart, is it able to recover
from the last good state? Does it print out meaningful error
messages to the user, or does it merely display
incomprehensible hex codes? Is the security of the system
compromised because of unexpected failures? And the list goes
on.

Verification Techniques

There are many different verification techniques but they all

basically fall into 2 major categories - dynamic testing and
static testing.

Dynamic testing - Testing that involves the execution of a

system or component. Basically, a number of test cases are
chosen where each test case consists of test data. These input
test cases are used to determine output test results. Dynamic
testing can be further divided into three categories -
functional testing, structural testing, and random testing.

* Functional testing - Testing that involves identifying

and testing all the functions of the system as defined within
the requirements. This form of testing is an example of black-
box testing since it involves no knowledge of the
implementation of the system.
* Structural testing - Testing that has full knowledge of
the implementation of the system and is an example of white-
box testing. It uses the information from the internal
structure of a system to devise tests to check the operation
of individual components. Functional and structural testing
both chooses test cases that investigate a particular
characteristic of the system.
* Random testing - Testing that freely chooses test cases
among the set of all possible test cases. The use of randomly
determined inputs can detect faults that go undetected by
other systematic testing techniques. Exhaustive testing, where
the input test cases consists of every possible set of input
values, is a form of random testing. Although exhaustive
testing performed at every stage in the life cycle results in
a complete verification of the system, it is realistically
impossible to accomplish.
Static testing - Testing that does not involve the operation
of the system or component. Some of these techniques are
performed manually while others are automated. Static testing
can be further divided into 2 categories - techniques that
analyze consistency and techniques that measure some program
property.

* Consistency techniques - Techniques that are used to

insure program properties such as correct syntax, correct
parameter matching between procedures, correct typing, and
correct requirements and specifications translation.
* Measurement techniques - Techniques that measure
properties such as error proneness, understandability, and
well-structuredness.

Q)10 ways to reduce cost of software testing

List of suggestions...

1. Closely work with developers, do some parallel testing with

them as the product/feature is getting developed
2. Identify and eliminate non-testing activities that occur in
the name of process, documentation, management, metrics etc.
3. Analyze and profile every application under the portfolio
to determine “stable” and “well tested” areas of the
application. These areas should receive the least or no
testing effort.
4. Analyze the test scripts suite and remove redundant, worn
out ones. Aim to reduce scripted test repository as small as
you can.
5. Review and reduce “regression testing” on the basis of
“well tested/stable areas” of the application
6. Switch from resource intensive and highly scripted testing
approach to highly improvisational exploratory /rapid testing
approaches
7. Plan testing in small but frequent cycles (Session based
exploratory testing approach) – reduce planning and management
overheads
8. Analyze and reduce the usage of costly tool licenses -
especially those do not help in testing directly (test
management tools)
9. Cut down on lengthy test plans, testing reports, dashboards
– switch to simple but frequent test reporting.
10. Simplify defect management process – reduce defect life
cycle – resort to informal/quick defect communication.