Software Testing

Testing across the entire software development lifecyle

Software test engineer

Software test engineer is a professional who is in charge of one or more technical test activities, including
designing test inputs producing test case values running test scripts analyzing results reporting results to developers and managers

By Ammann and Offutt, 2008

Software test manager

Test manager is in charge of one or more test engineers. They
set test policies and processes monitors the processes interact with other managers on the project help the engineers do their assignments

By Ammann and Offutt, 2008

Activities of a software test engineer

Design tests by creating test requirements
requirements are transformed into actual values and scripts ready for execution executable test are run against the software the results are evaluated to determine if the tests find a fault

Powerful tool: formal coverage criterion

Provides test engineers ways to decide what test inputs to use during testing such that maximum possible faults can be found Also provides stopping rules

By Ammann and Offutt, 2008

The mind of a tester

Four different kinds of thinking shown by a good tester (by Kaner et. al)
Technical thinking: ability to model technology and understand causes and effects Creative thinking: ability to generate ideas and see possibilities Critical thinking: ability to evaluate ideas and make inferences Practical thinking: ability to put ideas into practice

An example of these kinds of thinking is found in a fable called The Kings Challenge

By Everett and McLeod Jr., 2007 and Kaner et. al, 2002

The mind of a tester (contd)

The Kings Challenge (a fable)
Once upon a time, a mighty king wanted to determine which of his three court wizards was the most powerful So he put the three court wizards in the castle dungeon and declared whoever escaped from his respective dungeon cell first was the most powerful wizard in all the kingdom

By Everett and McLeod Jr., 2007 and Kaner et. al, 2002

The mind of a tester (contd)

Here is what each court wizards has done
The first wizard immediately started chanting mystical poems to open his cell door The second wizard immediately started casting small polished stones and bits of bone on the floor to learn how he might open his cell door The third wizard sat down across from his cell door and thought about the situation for a minute. Then, he got up, walked over to the cell door and pulled on the door handle. The cell door swung open because it was closed but not locked Thus, the third wizard escaped his cell first and became known as the most powerful wizard in all the kingdom

What kinds of tester thinking did the third wizard exercise in solving the kings puzzle?
By Everett and McLeod Jr., 2007 and Kaner et. al, 2002

The mind of a tester (contd)

Answer:
Creative thinking: ability to generate ideas and see possibilities Practical thinking: ability to put ideas into practice

By Everett and McLeod Jr., 2007 and Kaner et. al, 2002

Non-software testing at the user level

Buying a car
Use the automobile industry to find non-computer testing examples that can be related to software testing Motivation for testing a car is to determine its quality or its functionality before buying one As a user, you are not interested in performing all possible kinds of test since you assume that manufacturer has done those tests for you Important to realize that you limit your testing some way Limited test is referred as test drive To better understand the testing limits, examine what you do NOT test followed by what you TEST

By Everett and McLeod Jr., 2007

Non-software testing at the user level

Objectives of a Test Drive are NOT
to break the car
Seek guarantees and warranties that car manufacturer has already proven that car is unbreakable under normal driving conditions for x thousand miles or y years (reliability)

to improve the cars design

If you want design changes in the car, simply shop for different model or different manufacturer

By Everett and McLeod Jr., 2007

Non-software testing at the user level (contd)

What you test is determined by your transportation needs. The needs become test drive objectives Test objectives are the measurable milestones in testing which indicate that the testing activities have achieved the desired goals Test drive objectives are translated into testing approaches that validate whether the car on the dealers lot meets your transportation objectives

By Everett and McLeod Jr., 2007

Non-software testing at the user level (contd)

Objectives of a Test Drive ARE
to validate affordability to validate attractiveness to validate comfort to validate usefulness to validate performance

Each of these objectives can be validated without trying to break the car or redesign it. Each of these objectives are personal and you need to prioritize them to in final your final decision

By Everett and McLeod Jr., 2007

Non-software testing at the user level (contd)

Testing approaches include
examining the sticker price and sale contract trying out radio, the air conditioner, and the lights trying acceleration, stopping, and cornering

These testing approaches are referred to by common terminology in the testing industry
examine = static testing
(observe, read, review without driving the car)

try out = functional and structural testing

(work different features of the car without driving the car)

try = performance testing

(work different features of the car by actually driving the car)
By Everett and McLeod Jr., 2007

Non-software testing at the developer level

Building a car Testing objectives of a new car to be built
validate design via scale models validate operation of prototypes validate mass assembly plans from prototypes

Starts with written requirements such as

Seats six Carries five suite cases Runs on regular gas Consumes gas at a rate of 35 miles per gallon in highway Has a top speed of 100 miles per hour

By Everett and McLeod Jr., 2007

Non-software testing at the developer level (contd)

These requirements are the nonnegotiable design and manufacturing boundaries set by groups other than the designers

The manufacturer is responsible to build a new car which satisfies all the requirements
New requirements, more understandable test objectives The auto design tester is responsible to validate the current state of the new car against its requirements
If the new car doesnt initially meet the requirements, then its the designer, not the tester who must improve the design for full compliance After the design changes are done, the tester needs to revalidate the revised design against the requirements

Design, test, correct, retest cycle continues until the new car meets its requirements and is completed before the car is manufactured Requirements are essential for testing validation at every stage of developing a new car
By Everett and McLeod Jr., 2007

Non-software testing at the developer level (contd)

Testing approaches used while building new cars
plan the tests based on requirements and design specifications examine blueprints and clay models perform and analyze wind tunnel tests perform and analyze safety tests perform and validate prototype features drive prototype and validate operations

Specifications (blue prints or models) are the designers interpretation of the requirements on how the design can be manufactured When specifications are validated against the requirements, all the subsequent physical car assembly validation can be performed against the specifications

By Everett and McLeod Jr., 2007

Non-software testing at the developer level (contd)

Similar to the test drive, the car builder testing approaches can be described by common testing terminology
examine = static testing
(observe, read, review without driving the car)

perform = functional and structural testing

(work different features of the car models, mock-ups, and manufactured car subassemblies)

drive = performance testing

(work different features of the car in the prototypes)

By Everett and McLeod Jr., 2007

Primary objectives of testing

Testing objective 1: Identify the magnitude and sources of development risk reducible by testing
Company prepares a business case including expected benefits, costs, and risks If the project is determined to be bad on investment, then the project does not start If the benefits outweigh the cost and its good return on investment, the benefits are compared to the risks
If the risk is high but the likelihood of the risk occurrence is low, the project is on If the risk and the likelihood of the risk occurrence is high, the following questions are asked:
Can this risk be reduced by testing? If the risk can be reduced, how much can testing reduce it?

If the risk factors are well known and quantifiable, its possible that testing can reduce the probability of the risk occurrence
By Everett and McLeod Jr., 2007

Primary objectives of testing (contd)

Testing objective 2: Perform testing to reduce identified risks
Test planning includes
positive testing (things work as required) negative testing (things that break)

Test planning highlights the risk areas such that the largest possible percentage of the test schedule and effort (both positive and negative testing) are allocated to reduce that risk Testing does not completely eliminate the risk since there are always more scenarios to test than the allotted time and resources to complete the tests

By Everett and McLeod Jr., 2007

Primary objectives of testing (contd)

Testing objective 3: Know when testing is completed
Since 100% testing is unrealistic, the tester must determine when to stop testing The determination should start with the positive test items in the test plan Complete as much of the risk-targeted testing as possible relative to cost benefit break-even point
$100,000 business risk, $50,000 testing cost to reduce the risk (NO!!!) A rule of thumb is 10-20% cost to benefit break-even point for testing
$100,000 business risk, $1000-2000 testing cost (YES!!!)

Tester must complete as many of the negative test items in the plan with the remaining the testing budget after positive testing and risk testing are completed
By Everett and McLeod Jr., 2007

Primary objectives of testing (contd)

Testing objective 4: Manage testing as a standard project within the development project
Often testing is treated as simple skill that anyone can do without planning, scheduling or resources Since business risk represents real dollar loss, real dollar testing is required to reduce the risk Real dollar testing means that personnel with testing experience should be become the testing team with access to management, resources and schedules necessary to plan and complete the test Observations:
The testing does not have to be a hit or miss Testers are a limited resource

By Everett and McLeod Jr., 2007

References
P. Ammann and Jeff Offutt, Introduction to software testing, Cambridge University Press, 2008 G. D. Everett and R. McLeod Jr., Software Testing, Wiley InterScience, 2007 C. Kaner, J. Bach, B. Pettichort, Lessons learned in software testing, Wiley Computer Publishing, Addison-Wesley, 2002, pp. 286