1

SOFTWARE TESTING & QUALITY ASSURANCE

Lecture 1: Introduction

Outline
2

Consequences of software errors, faults and failures Cost of software errors Software quality Software testing Why test? When to test? What to test? Prioritization of test Who test?

Consequences of software errors, faults and failures

3

Therac-25 (1985- 1987)

When operating in soft X-ray mode, the machine was designed to rotate three components into the path of the electron beam, in order to shape and moderate the power of the beam.
The accidents occurred when the high-energy electron-beam was activated without the components being rotated; the machine's software did not detect that this had occurred, and did not therefore determine that the patient was receiving a potentially lethal dose of radiation, or prevent this from occurring.

Consequences of software errors, faults and failures

4

The computer aided dispatcher disaster - October 1992, the London Ambulance Service suffered a disaster Brought their operations to a virtual standstill over 36 hours, and cost 20 lives in the process. It was discovered that the new computer aided dispatch (CAD) software was responsible for the crisis. The Intel pentium bug (1994) cost ~$ 475 mill. Testers foudna a Division error in the chip but managers decided it wont affect many people and did not inform.

Consequences of software errors, faults and failures

5

Military Aviation:
An

F-18 crashed because of a missing exception condition: if ... then ... without the else clause that was thought could not possibly arise.

Year Ambiguities

Mr. Blodgetts auto insurance rate tripled when he turned 101. He was the computer programs first driver over 100, and his age was interpreted as 1

Consequences of software errors, faults and failures

6

Ariane 5 flight 501 was destroyed 40 seconds after take off due to a crash in a software component controlling the rocket (1996) Explosion was the result of a software error:

Uncaught exception due to floatingpoint error: conversion from a 64-bit integer to a 16-bit signed integer applied to a larger than expected number. Module was re-used without proper testing from Ariane 4 .

Error was not supposed to happen with Ariane 4. No exception handler.

Consequences of software errors, faults and failures

7

Mars Climate Orbiter - September 23, 1999

Mars Climate Orbiter, disappeared as it began to orbit Mars. Cost about $US 125-million. Failure due to error in a transfer of information between a team in Colorado and a team in California.

One team used imperial units (e.g., inches, feet and pounds) while the other used metric units for a key spacecraft operation.

Consequences of software errors, faults and failures

8

Mars Polar Lander - December, 1999

Mars Polar Lander, disappeared during landing on Mars. Failure most likely due to unexpected setting of a single data bit.

Defect not caught by testing. Independent teams tested separate aspects.

Consequences of software errors, faults and failures

9

May 2005 - automaker Toyota announced a recall of 160,000 of its Prius hybrid vehicles following reports of vehicle warning lights illuminating for no reason, and cars' gasoline engines stalling unexpectedly. But unlike the large-scale auto recalls of past years, the root of the Prius issue wasn't a hardware problem -- it was a programming error in the smart car's embedded code. The Prius had a software bug.

Consequences of software errors, faults and failures

10

An Australian man purchased $104,500 worth of Sri Lankan Rupees. The next day he sold the Rupees to another bank for $440,258. The first banks software had displayed a bogus exchange rate in the Rupee position! A judge ruled that the man had acted without intended fraud and could keep the extra $335,758!

What do software faults cost?

11

Software faults can cause death or injury

Radiation

treatment kills patients Train driver killed Aircraft crashes

Software

faults cost money to organisation

Not critical but loss of life

Bank

system overdraft letters cause suicide

Sources of Problems
12

Requirements Definition: Erroneous, incomplete, Design: Fundamental design flaws in the software. Inadequate Testing of Software. Evolution: introduction of new flaws in attempts to fix old flaws

Software Testing
13

Testing: The process concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.
[International Software Testing Qualifications Board 2008]

Testing is the process of executing a program with the intent of finding errors
[Myers 1976]

Definition
14

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can be stated as the process of validating and verifying that a software program/application/product:
meets

the requirements that guided its design and development; works as expected; and can be implemented with the same

Principles of Software Testing

15

1.

A good test case is one that has a high probability of finding an undiscovered defect. So, the test cases (the program input) should be selected systematically and with care, both for correct and incorrect behavior.

2.

A successful test is one that uncovers an undiscovered defect. So, testing is psychologically destructive since it tries to demolish the software that has been constructed.
Testing can never completely identify all the defects within software

3.

Principles of Software Testing

16

4.

Testing cannot show the absence of defects, it can only show that they are present (Dijkstra). Testing is quite an ineffective method of quality assurance. [Though, usually the most applicable one.] Successful testing shall be followed by a separate debugging phase. Testing is also by itself a process that must be systematically managed (and assisted with special testing tools).

5.

6.

7.

Error - Fault - Failure

17

Error - Fault - Failure

18

Error: are defects in the human thought process made while trying to understand given information, to solve problems, or to use methods and tools. A fault occurs when a human error results in a mistake in some software product. For example, a developer might misunderstand a user interface requirement, and therefore create a design that includes the misunderstanding On the other hand, a failure is the departure of a system from its required behavior. It occurs when fault executes.

Verification vs Validation
19

Verification: "Are we building the product right.

Process of ensuring that the software correctly implements a specific function

Validation: "Are we building the right product.

The software should do what the user really requires.

20

Why do faults occur in software?

Software is written by human beings Who know something, but not everything Who have skills, but arent perfect Who do make mistakes (errors) Under increasing pressure to deliver to strict deadlines No time to check but assumptions may be wrong Systems may be incomplete

Why test?
21

According to several empirical studies, a (professionally produced commercial) software system contains 3 30 defects in every 1000 lines of code
The average debugging effort is 12 hours of working time for a single defect Maintenance costs about 50% of software development costs, mostly in error removal

Why test?
22

Build confidence Demonstrate conformance to requirements Find faults Reduce costs Show system meets user needs Assess the software quality

Confidence
Confidence Faults found Fault found

Time
No faults found = confidence?

A traditional testing approach

Show that the system:

Does

what it should Doesn't do what it shouldn't

Goal:

show working

Success: system works Fastest achievement: easy test cases Result: faults left in

A better testing approach

Show that the system:

Does

what it shouldn't Doesn't do what it should

Goal:

find faults

Success: system fails Fastest achievement: difficult test cases Result: fewer faults left in

26

Cost of Testing v Cost of Defects

Developers already spend approximately 80% of software development costs on identifying and correcting defects". Identifying and correcting defects" not necessarily the same thing as the cost of testing, but still...
[NIST report (2002): The Economic Impacts of Inadequate Infrastructure for Software Testing http://www.nist.gov/director/planning/upload/report02-3.pdf]

27

Costs of fixing defects found at different stages

Why not just "test everything"?

Avr. 4 menus 3 options / menu system has 20 screens Average: 10 fields / screen 2 types input / field (date as Jan 3 or 3/1) (number as integer or decimal) Around 100 possible values

Total for 'exhaustive' testing: 20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests If 1 second per test, 8000 mins, 133 hrs, 17.7 days (not counting finger trouble, faults or retest) 10 secs = 34 wks, 1 min = 4 yrs, 10 min = 40 yrs

How much testing?

29

It depends on RISK
Risk

of missing important faults Risk of incurring failure costs Risk of releasing untested or under-tested software Risk of losing credibility and market share Risk of missing a market window Risk of over-testing, ineffective testing

When to test
30

Software testing, can be implemented at any time in the development process. However, most of the test effort occurs after the requirements have been defined and the coding process has been completed. Different software development models will focus the test effort at different points in the development process. Newer development models, such as Agile, often employ test driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of

Who carry out the tests?

31

Software Developers-unit testing Testers- module testing, sub-system & system Clients End Users

Independent Testing Testing done by someone who has not been involved in the development of the software under test is likely to be better (i.e. find more faults) than testing done by someone who has been involved in the softwares development.

Levels of Independence
32

Tests designed by a different person other than the developper

Tests designed by someone from a different department or team (e.g. test team) Tests designed by someone from a different organisation (e.g. agency) Tests generated by a tool (but low quality tests)

Software Quality
33

Quality:
(1) The degree to which a system, component, or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations.
[Standard Glossary of Software Engineering Terminology ]

Software quality: Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.
[R.S. Pressman: Software Engineering A Practitioners Approach (5th ed) McGraw-Hill, 2000]

Software Quality
34

Application of sound technical methods and tools Formal technical reviews and inspections Software testing Enforcement of standards Documentation Control of change Extensive measurement Record keeping and reporting of the process