Chapter 2

Introduction and Examples

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Errors, Faults, Failures and Incidents

Fault
Spec
Resolution

fault
Isolate
Design
Fault

fault Classify
Coding
Fault

fault Incident
Testing

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Universe of Discourse—Program Behaviors

Program Behavoirs

S P
“Correct
” region

Specification Program
(expected) (Implemented)

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 About Correctness...

• Impossible to demonstrate
• A term from “classical” computer science
– “proofs” derived from code
– Not derived from specification
– Can only prove that the code does what it does!
• Better viewpoint: a relative term—program P is
correct with respect to specification S.
• Bottom Line: do the specification and the program
meet the customer/user's expectations?

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Testing and Program Behaviors

Program Behavoirs

Specification Program
(expected) S P (Implemented)
5 2 6
Spec-based 1
testing 4 3
Code-based
7 8 testing
T
Test Cases
(verified)

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Synonyms—Metaphors gone wild

• Specification-based testing Inputs Outputs

– Functional testing
– Black box testing

An engineer’s Black Box: a

system understood in terms
of its inputs and outputs

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Program Behavoirs

S P

Spec-based Code-based
Functional Structural
Black Box White/Clear Box
(establishes confidence) (seeks faults)

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Levels of Testing (as seen in the V-model)

Requirements System
Specification Testing

Preliminary Integration
Design Testing

Detailed Unit
Design Testing

Coding

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Content of a Test Case

• Test case identifier (usually a short name for test

management purposes)
• Name
• Purpose (e.g., a business rule)
• Pre-conditions (if any)
• Inputs
• Expected outputs
• Observed outputs
• Pass/fail?

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Testing as an Experiment

• With respect to the Error, Fault, Failure, Incident

framework, a test case is an experiment that...
– is designed to anticipate an error
– that is realized as a fault
– that causes a failure
– and is recognized by comparing expected and observed
outputs.
• In this framework, test cases are intended to reveal
incidents.

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 About Pseudo-code

• Language neutral
• Supports both procedural and
object-oriented code
• Easy to “translate” into your
favorite programming language

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Testing goal.
The goal of testing is to build knowledge about a product by uncovering
errors made when producing that product.

The goal of testing is to build confidence in the software by conducting testing

and fix things and eventually find fewer faults.

If testing was exhaustively(on every input in input domain) without us finding

any faults we would say the product is correct. This is not practically possible
for more than trivial problems.

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Testing versus Verification
Goal of testing Goal of Verification:
“Uncover errors” “Show there is no error for any input”,
“Build confidence in the software” “prove correctness”

https://www.youtube.com/watch?v=gp_D8r-2hwk
Age of Conan(c) 2011

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction
 Errors and Faults
An error(mistake) in thought or action
may lead to a
fault(defect, bug)
that may lead to
an observed behaviour,
that is not correct.
(according to some specification) -Mathur

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction
 Aim of writing tests
We want to write tests in a way that we
execute a program such that we find bugs.

“Execute code with a fault in such a way

that the bug infects the state to something
we can eventually observe.”

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Test case
1. System under 1. f(x, y) = add(x, y)
Test 2. x = 3, y = 2
2. Input 3. f(3, 2) = 5
3. Expected Output 4. f(3, 2) = 3
4. Actual Output Observed = 3.0
Observed Output

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Definitions
Input domain Correctness
“The set of all possible input to a program P is “A program is considered correct if it behaves
known as the input domain, or input space of as expected on each element of its input
P” domain”
-Mathur, p 12 - Mathur, p 13

Reliability (note that this is just one def of

many)
“The reliability of a program P is the probability
of its successful execution on a randomly
selected element from its input domain.”
-Mathur, p 16

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Testing Types
●
Black-box testing
●
White-box testing
●
Model based testing
●
User Interface
testing
●
Smoke testing
●
Random testing

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Black box testing
Examine
functionality from the
outside without
knowing anything of
the internals.
Tests are generated
from specification.

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 White(or. Glass) box testing
Testing with
knowledge of code
internal workings.
Behaviour
Ex. coverage.
Tests are generated
from source code.

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Model based testing

"Mbt-overview". Licensed under Public Domain via Wikipedia - https://en.wikipedia.org/wiki/File:Mbt-overview.png#/media/File:Mbt-

overview.png

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 User Interface testing

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Smoke and Random Testing
Built on that the system may crash when
in a faulty state.

May be designed to crash when in a

faulty state. Eg by adding: “In code
asserts”

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Exceptions
Smoke testing Error messages
Log-messages
Output: -1
Run as large part of ...

the system as Asserts

possible, look for No output

smoke! 100 % CPU, no

stop

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Random testing

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Static and dynamic Qualities
Static: Dynamic:
(offline) (When program is running)
●Structured, Maintainable Code ●Reliability
●Testable code ●Completeness (Availability of features)
●Compiles ●Consistency (Eg. same design)
●Correct and complete documentation ●Usability
●Code review ●Performance

Mathur p 9

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Levels of Testing Activities
● Unit Testing, when coding
● Integration Testing, during integration
● System Testing, during system
integration
● Regression Testing, after change
● Beta-Testing, Pre release delivery
● Acceptance Testing, during delivery
Mathur

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction

 Tasks:
Open Lecture by James Bach (Exploratory)
https://www.youtube.com/watch?v=ILkT_HV9DVU#t=14

Agile Testing, Google Tech Talks (2007)

https://www.youtube.com/watch?v=bqrOnIECCSg

Software Testing: A Craftsman’s Approach, 4th Edition Chapter 1 Introduction