Вы находитесь на странице: 1из 80

1

Chapter

Software Testing Foundations


MSTB-GTB
Certified Tester
Fundamentals of Testing

2011
Version 1.0

After this lecture you should.

Be able to use examples to describe the way in which a software defect can cause harm to
people, to the environment, or to a company;

Know what is meant by the terms deficiency, defect, failure, defect condition / defect and error
and explain this with examples and compare;

Be able to distinguish between the cause of a defect and its effects;

Derive by examples why testing is necessary;

Be able to describe why testing is part of quality assurance and give examples of how testing
contributes to a higher quality;

Know the quality characteristic according to ISO 9126;

Recall the general objectives and principles of testing;

Differentiate between testing and debugging;

Know how the fundamental test process looks like;

Describe how expected values are calculated during testing;

Know and characterize the psychological problems during the testing;

Differentiate between the different mindset of a tester and a developer;

Know the code of ethics of a software tester.

Chapter 1

Page 2

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Learning Objectives for Fundamentals of Testing


(according to the Certified Tester Foundation Level Syllabus, Version 2011)

Chapter 1

1.1 Why is Testing Necessary? (K2)


LO-1.1.1

Describe, with examples, the way in which a defect in software can cause
harm to a person, to the environment or to a company (K2)

LO-1.1.2

Distinguish between the root cause of a defect and its effects (K2)

LO-1.1.3

Give reasons why testing is necessary by giving examples (K2)

LO-1.1.4

Describe why testing is part of quality assurance and give examples of how
testing contributes to higher quality (K2)

LO-1.1.5

Explain and compare the terms error, defect, fault, failure and the
corresponding terms mistake and bug, using examples (K2)

1.2 What is Testing? (K2)


LO-1.2.1

Recall the common objectives of testing (K1)

LO-1.2.2

Provide examples for the objectives of testing in different phases of the


software life cycle (K2)

LO-1.2.3

Differentiate testing from debugging (K2)

Page 3

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Learning Objectives for Fundamentals of Testing

(according to the Certified Tester Foundation Level Syllabus, Version 2011)

1.3 Seven Testing Principles (K2)


LO-1.3.1

1.4 Fundamental Test Process (K1)


LO-1.4.1

Chapter 1

Explain the seven principles in testing (K2)

Recall the five fundamental test activities and respective tasks from planning
to closure (K1)

1.5 The Psychology of Testing (K2)


LO-1.5.1

Recall the psychological factors that influence the success of testing (K1)

LO-1.5.2

Contrast the mindset of a tester and of a developer (K2)

Page 4

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Incident

Review

Test Condition
Independence

Terms you should be familiar with

Chapter 1

Page 5

Software Testing Foundations Certified Tester

Test Basis

Quality
Risk

Test Objective

Test Plan
Testware

Master Test Plan

Test Design
Test Run Log

Quality Assurance
Error cause
Confirmation Test

Exhaustive Test
Test Coverage

Software Test

Debugging

Fundamental Test Process

Test Summary Report


Exit Criteria
Test Data

Test Policy

Failure

Error
Regression Test Requirement
Error Test Execution
Test Case Error Guessing
Defect Testing Test Suite

Copyright 2011 to MSTB/GTB


V 1.0

Chapter

Fundamentals
of Testing

Chapter 1

Page 6

Terms and Motivation


Principles of Testingda
Fundamental Test Process Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

In General: What is a Defect or a Deficiency?


A situation can only be classified as defective if it is pre-defined, how
the expected, correct and therefore the non-defective situation should
look like.
Defect
Is the inability to fulfill a specific requirement. It is a discrepancy
between the actual behaviour (during the execution of the tests or
identified operation) and the expected behaviour (in the test
specification or the stated requirements).
Deficiency
Non-fulfilment of a requirement related to an intended or specified
use. A deficiency is for example the impairment of usability, while
meeting performance or failure to perform any reasonable
expectation.
Based on: DIN EN ISO 9000:2005
Quality Management Systems
Fundamentals and Terms (German)
Beuth Verlag, Berlin, 2005
Chapter 1

Page 7

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Connection:
Error - Fault/Defect - Failure
People make mistakes; they commit errors.
Example:
A programmer commits
an error by re-using a
a piece of software which is
not intended in the context
of the current project
(the Ariane 5 satellite
launching rocket)

Error

ISTQB Glossary:
Error: Human action that produces an incorrect result [after IEEE 610].

Chapter 1

Page 8

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Error Other Definitions


Other defintions:
1. Human action (of the developer) that
produces an error condition in the software
2. Human action (of the user) that produces
an unwanted result (failure) as a
consequence (misoperation)
3. Unknowingly, inadvertently or intentionally
performed act, or omission which leads under given
circumstances (task, environment) to an impairment of the
required function of a product

Chapter 1

Page 9

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Connection:
Error - Fault/Defect - Failure
An error made by a person
(e.g. a developer) can result
in a defect in the sofware.
(Defect and Fault are
synonymous terms)
Example: After the
programmer has
re-used the piece of
code in the wrong
context, the software
contains a defect/fault.

Defect/Fault

Error

in a program

ISTQB Glossary:
Defect: A flaw in a component or system that can cause the component
or system to fail to perform its required function, e.g. an incorrect statement or
data definition A(the
defect,
if encountered
execution,
may cause a
full definition
followsduring
on page
12)
failure of the component or system.
Chapter 1

Page 10

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Connection:
Error - Fault/Defect - Failure
A failure is the effect of
a defect when executing
a program (test object)
which appears to the
outside.

Failure

Defect/Fault

Error

Example:
Boom
The Ariane 5
rocket crashes
on its first
mission.
Cost ~ $7 billion
ISTQB Glossary:
Deviation of the component or
system from its expected delivery,
service or result [after Fenton].
Chapter 1

Page 11

Software Testing Foundations Certified Tester

in a program

which appears

Copyright 2011 to MSTB/GTB


V 1.0

Fault/Defect Failure
Additional comments
Defect (full definition from ISTQB glossary): A flaw in a
component or system that can cause the component or system to
fail to perform its required function, e.g. an incorrect statement or
data definition. A defect, if encountered during execution, may
cause a failure of the component or system.
Defect masking: An occurrence in which one defect prevents the
detection of another [after IEEE 610]
A failure
can also be called a malfunction or an external defect
can also (but rarely) be caused by cosmic radiation, electromagnetic fields or hardware failure. Such failures can cause
slow execution, incorrect output or the termination of execution.
Debugging is the development activity that finds, analyzes and
removes the cause of the failure.

Chapter 1

Page 12

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Testing Definition and Goals


The process consisting of all life cycle activities, both static and
dynamic, concerned with planning, preparation and evaluation of
software products and related work products to
determine that they satisfy specified requirements
demonstrate that they are fit for purpose
detect defects
Source: [ISTQB Glossary]

Chapter 1

Page 13

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Validation and Verification - Definitions


Validation
Confirmation by examination and through provision of
objective evidence that the requirements for a specific
intended use or application have been fulfilled .
[ISO 9000] and [ISTQB Glossary]
Did we implement the right system?
Verification
Confirmation by examination and through the provision
of objective evidence that specified requirements have
been fulfilled [ISO 9000] and [ISTQB Glossary]
Did we implement the system right?
Chapter 1

Page 14

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

What is Software Quality?


A feature or characteristic that affects an item's quality [IEEE 610].
A set of attributes of a software product by which its quality is
described and evaluated. A software quality characteristic may be
refined into multiple levels of sub-characteristics [ISO 9126]*.
Quality attributes relate to requirements
Functional requirements (capabilities, interfaces,
professionalism, ...)
Non-functional requirements (quality and implementation
requirements, project-specific requirements, ...)
*Remark:
The description of product quality attributes provided in ISO9126 is
used as a guide to describing software quality attributes. Other
standards, such as the ISO/IEC 25000 series, may also be of use.

Chapter 1

Page 15

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Software Quality according to ISO/IEC 9126 (1)


Software Quality

External and
Internal Quality

Quality in Use

Effectiveness

Productivity

Security

Satisfaction

Fulfillment of tasks
within the
accuracy and
completeness
limits

Fulfillment of tasks
within the
expenditure limits
for users (time,
costs, ...)

Fulfillment of tasks
within the limits of
risk (life and
health,
business...)

Subjective user
satisfaction within
the context of use

ISO/IEC 9126: Rate of software products, quality characteristics


and guidance on usage
Chapter 1

Page 16

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Software Quality according to ISO/IEC 9126 (2)


Software Quality

Quality in Use

Functionality

Reliability

Suitability

Maturity

Accuracy
Interoperability

Fault tolerance

(Data) Security
Compliance

Recoverability
Compliance

Usability

Understandability
Learnability
Operability
Attractiveness
Compliance

External and
Internal Quality

Efficency

Time behavior /
Performance
Resource
utilization
Compliance

Maintainability

Portability

Analyzability

Adaptivity

Changeability

Installability

Stability

Co-Existence

Testability

Replaceability

Compliance

Compliance

ISO/IEC 9126: Evaluation of software products, quality characteristics and guidance on usage
Chapter 1

Page 17

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

External Quality Attribute: Functionality (1)


Existence of functions with specified attributes.
These functions meet the specified or implied requirements.
sub-attributes of the quality attribute functionality are described
below and on the next page.
Suitability
The capability of the software product to provide an appropriate set
of functions for specified tasks and user objectives [ISO 9126].
Accuracy
The capability of the software product to provide the right or agreed
results or effects with the needed degree of precision [ISO 9126].
(Further sub-attributes of the quality attribute functionality follow)

Chapter 1

Page 18

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

External Quality Attribute: Functionality (2)


(Further sub-attributes of the quality attribute functionality)
Interoperability
The capability of the software to interact with one or more specified
components or systems [after ISO 9126].
(Data) Security
Attributes of software that bear on its ability to prevent
unauthorised access, whether accidental or deliberate, to programs
and data [after ISO 9126].
Compliance
Attributes of the software which demonstrate that the software
fulfills application-specific standards or agreements or statutory
directives or similar regulations. [after ISO 9126].
Note: this also applies to all other quality attributes (e.g. reliability)

Chapter 1

Page 19

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

External Quality Attribute: Reliability


Attributes that relate to the ability of the software to maintain a specified
level of performance under given conditions and for a specified time period.
Maturity
The capability of the software product to avoid failure as a result of
defects in the software [ISO 9126].
Fault tolerance
The capability of the software product to maintain a specified level of
performance in cases of software faults (defects) or of infringement
of its specified interface [ISO 9126].
Recoverability
The capability of the software product to re-establish a specified level
of performance and recover the data directly affected in case of failure
[ISO 9126].

Chapter 1

Page 20

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

External Quality Attribute Usability


Attributes that relate to the effort required to use, and on the individual
assessment of such use by a specific or implied group of users.
Understandability
The capability of the software product to enable the user to
understand whether the software is suitable, and how it can be used
for particular tasks and conditions of use [ISO 9126].
Learnability
The capability of the software product to enable the user to learn its
application [ISO 9126].
Operability
The capability of the software product to enable the user to operate
and control it [ISO 9126].

Chapter 1

Page 21

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

External Quality Attribute: Efficiency


Attributes that relate to the relationship between the performance
levels of the software and the amount of equipment used under
specified conditions.
Time behaviour / Performance
The degree to which a system or component accomplishes its
designated functions within given constraints regarding processing
time and throughput rate [after IEEE 610].
Resource utilization
The capability of the software product to use appropriate amounts
and types of resources, for example the amounts of main and
secondary memory used by the program and the sizes of required
temporary or overflow files, when the software performs its function
under stated conditions [after ISO 9126].

Chapter 1

Page 22

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Internal Quality Attribute: Maintainability (1)


Attributes that relate to the effort that is necessary to carry out software
changes.
Analyzability
The capability of the software product to be diagnosed for deficiencies
or causes of failures in the software, or for the parts to be modified to
be identified [ISO 9126].
Changeability
The capability of the software product to enable specified modifications
to be implemented [ISO 9126].

(Further sub-attributes of the quality attribute maintainability follow)

Chapter 1

Page 23

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Internal Quality Attribute: Maintainability (2)


(Further sub-attributes of the quality attribute maintainability)

Stability
The capability of the software product to avoid unexpected effects
from modifications in the software [ISO 9126].
Testability
The capability of the software product to enable modified software to
be tested [ISO 9126].

Chapter 1

Page 24

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Internal Quality Attribute Portability (1)


Attributes that relate to the ability of software to be transferred from
one environment to another.
Adaptability
The capability of the software product to be adapted for different
specified environments without applying actions or means other
than those provided for this purpose for the software considered
[ISO 9126].
Installability
The capability of the software product to be installed in a specified
environment [ISO 9126].
(Further sub-attributes of the quality attribute portability follow)

Chapter 1

Page 25

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Internal Quality Attribute: Portability(2)


(Further sub-attributes of the quality attribute portability)

Co-Existence
The capability of the software product to co-exist with other
independent software in a common environment sharing common
resources [ISO 9126].
Replaceability
The capability of the software product to be used in place of
another specified software product for the same purpose in the
same environment [ISO 9126].

Chapter 1

Page 26

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Quality Requirements
Quality requirements state what quality attributes the product
should have at what quality level.
The sum of all quality attributes and the specific variants
Not all quality attributes can be achieved equally well.
E.g. efficiency may come at the expense of maintainability
Set priorities
In close consultation with clients and users.
Quality requirements (with the exception of functionality) are
part of the non-functional requirements in the specifications.

Chapter 1

Page 27

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Testing and Quality


Testing measures the quality e.g. based on the number of located
failures and defects.
Testing increases the quality indirectly, as defects are detected
which can be corrected prior to delivery.
Testing increases the process quality indirectly, as defects are
documented which can then be analysed and help prevent similar
errors from taking place in the future.
Testing increases the confidence in the quality of the system when
few or no failures and defects are found.

Chapter 1

Page 28

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Quality Assurance - QA
Quality Assurance
Analytic QA

Artefacts

Constructive QA

Processes
Audits

Software, etc.
Dynamic
Testing:
White-Box,
Black-Box
Chapter 1

Page 29

Software, Documentation,
etc.

Norms, standards,
project management,
software techniques,
exchange of experiences,
training
...

Static Testing:
Reviews,
Static Analysis
Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Chapter

Fundamentals
of Testing

Chapter 1

Page 30

Terms and Motivation


Principles of Testing
Fundamental Test Process Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Seven Principles of Testing (1)


In the previous 40 years, the following seven testing principles have
emerged and can serve as guidelines:
Principle 1: Testing shows presence of defects
Testing can show that defects are present, but cannot prove that
there are no defects.
Program testing can be used to show the presence of bugs,
but never to show their absence! Edger W. Dijkstra, 1970
Principle 2: Exhaustive testing is impossible
Complete / exhaustive testing is not feasible except for trivial
programs (see slides in chapter 1 on exhaustive testing)
(continued)

Chapter 1

Page 31

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Seven Principles of Testing (2)


Principle 3: Early testing
Testing is not a phase in the software development that happens at
the end; it shall be started as early as possible. Through early testing
(e.g. reviews) which take place parallel to the development activities,
defects are detected earlier and thus cost less to fix.
Principle 4: Defect clustering
Testing effort shall be focused proportionally to the expected and the
observed defect density of modules. A small number of modules often
contains most of the defects discovered during pre-release testing, or
is responsible for most of the operational failures.
(continued)

Chapter 1

Page 32

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Seven Principles of Testing (3)


Principle 5: Repetitions have no effects - Pesticide Paradox
Just repeating tests brings no new insights. Test cases need to be
regularly reviewed, revised and modified.
Principle 6: Testing is context dependent
Testing is depending on context. For example, safety-critical software
systems are tested differently (more intensely and using other
techniques) from commercial applications.
Principle 7: Absence-of-errors fallacy
A system without failures does not imply that the system will meet the
users expectations.

Chapter 1

Page 33

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Testing Effort?
Testing is economically useful, as long as the costs of finding and
fixing a defect in testing are lower than the costs that are
associated with the occurrence of a defect when used.
M. Pol, T. Koomen, A. Spillner:
Management and Optimization of the Test Process
(German). dpunkt-Verlag, 2. Edition, 2002

A good test is like a liability insurance: it costs big money, but


allows the project manager and the customer to sleep peacefully. A
good sleep includes a good insurance policy that covers all
possible risks. The trust in a software product includes a good test
that fully covers the reality of production.
Siedersleben, J. (Hrsg):
Software methodology: Practical Knowledge for
Software Engineers, (German). 2. Edition, Hanser,
2003

Successful testing (detection of failures) reduces costs


Chapter 1

Page 34

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

More regarding Testing (1)


Testing effort in practice:
25% to 50% of the development effort
Test intensity and scope depends on risks and the criticality of the
project, the system and the domain
Defects can result in huge costs:
Estimated losses due to software defects in medium and large
companies in Germany are approximately 84.4 billion Euros per
annum
Productivity losses due to computer failures caused by software
defects are approximately 2.6% of sales - 70 billion Euros per
annum

Study by LOT Consulting, Karlsruhe


IT-Services 3/2001, P. 31 (German)

Chapter 1

Page 35

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

More regarding Testing (2)


Testing is always subject to limited resources
(e.g. time, test personal)

Particularly important:
Select appropriate test procedures which are compatible with
the test objectives and quality objectives
Avoid unnecessary tests which provide no new information
Take positive and negative test conditions into account
It can also be useful to test for functionality that has not been
requested

Chapter 1

Page 36

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Prioritization of Tests
Limited resources (time and personnel) require that the most important
test cases are carried out first!
Criteria for prioritization:
Expected impact (the severity of the damage a defect would cause)
Probability of occurrence of a failure
Combined consideration of impact and likelihood of occurrence
( Risk = Likelihood * Impact)
Perception of the failure
Prioritization of requirements by the customer
Importance of quality characteristics by the customer
Priority of test cases from the perspective of development (serious
consequences and / or complex cases first)
High project risk
Where many defects are found, more defects are likely to be
discovered! A change in the priority must be possible!
Chapter 1

Page 37

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Chapter

Fundamentals
of Testing

Chapter 1

Page 38

Terms and Motivation


Principles of Testing
Fundamental Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

SW-Development Models: Waterfall-Model - 1970

Royce, W. W.:
Managing the Development of Large Software
Systems: Concepts and Techniques
Proc. WESCON, IEEE Computer Society Press, Los
Alamitos, CA, 1970.
Reprinted at the ICSE'87, Monterey, California, USA.
March 30 - April 2, 1987

Chapter 1

Page 39

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

SW-Development Models: Waterfall-Model - 1981

Boehm, B.:
Software Engineering Economics
Prentice-Hall Inc., London, 1981

Chapter 1

Page 40

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

V-Model (B. Boehm, 1979)

Barry W. Boehm:
Guidelines for Verifying and Validating Software
Requirements and Design Specification.
EURO IFIP 79, P.A. Samet (eds.) North-Holland, IFIP
1979, 711-719

Chapter 1

Page 41

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

General V-Model

Definition of Requirements

Acceptance Testing

Functional
System Design

System Testing

Technical
System Design

Integration Testing

Component
Specification

Component Testing

Programming
Key
Test cases based on
corresponding documents
Chapter 1

Page 42

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Test Process

Is closely linked with software development

Is, however, a separate, independent process

A test plan is necessary for every test level (level test plan)

Testing cannot be considerd as a single activity (e.g. test execution)

Many individual tasks are performed within testing

The test process represents these individual testing activities in a coherent


process

Chapter 1

Page 43

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Activities of the Fundamental Test Process


Start

Test Planning and Control


Test Analysis and Design

Planning and

Test Implementation and Execution


Evaluating Exit Criteria and
Reporting
Test Closure Activities
These activities may overlap or take
place concurrently.
The Fundamental Test Process is to
be tailored to each test level (e.g.
module test, system test)

Chapter 1

Page 44

Software Testing Foundations Certified Tester

Analysis and Design

Implementation and
Execution

Control

Evaluation and
Report

Closure

Finish

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Planning and Control (1)


Specification of Test Plan
Determine test objectives, amount and
risks of testing.
Specify test approach (techniques*,
test objects, coverage, teams who
participate in testing, testware).

Planning and

Analysis and
Design
Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Details of Test plan


Finish
Test resources (e.g. staff, test environment, PCs).
How intensive should particular parts of the system be tested?
Which test techniques should be used?
Determine exit criteria. What level coverage should be reached?
Prioritizing of test (taking defect risks into consideration).
Planning for tool support (use, acquisition, training).
Test schedule and test strategy are to be recorded in the test plan
*refer to chapters 4 and 5
Chapter 1

Page 45

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Planning and Control (2)


Drawing up a rough schedule
Schedule tasks for test analysis and test
specification.
Schedule test implementation, test
execution and test evaluation.
Test Control
Measuring and analysing results.
Monitoring and recording of test progress,
achieved test coverage and exit criteria.
Initiate corrective actions.
Adapting time and resource planning.
Making decisions.
Refer to chapter 6 for more details

Chapter 1

Page 46

Software Testing Foundations Certified Tester

Planning and

Analysis and
Design
Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Analysis and Design (1)

Planning and

Analysis and
Design

General testing objectives are detailled to specific


Implementation
test requirements and criteria.
Control
and Execution
Basis are all documents showing the
Evaluation and
Report
requirements for the software (test basis).
Reviewing the test basis (requirements,
Test Closure
architecture, design, interface, ...).
Finish
Evaluating testability of requirements and the system.
Identifying test conditions / test requirements.
Designing the test environment (infrastructure, tools, ...).
Designing test cases using the chosen test design techniques.
Distinction to be made between high level test cases and detailed
test cases.
Creating bi-directional traceability between test basis and test
cases

Chapter 1

Page 47

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Analysis and Design (2)

Planning and

Analysis and
Design

Test cases contain more that just test data!


Implementation
Criteria and constraints that have to be met
and Execution
before execution (e.g. condition of test
Evaluation and
object, data, network status)
Report
Before test execution determine which result
Test Closure
or behavior is expected and the post-conditions,
that have to be fulfilled after the test, e.g. condition
Finish
of the test object and the database.
Refer to the next section in this chapter for further details

Control

Test cases are partly linked together to create test suites (test
procedure specification) for test execution
Each test case should only contain one or a few elementary
process steps or system calls
Pay attention in the test design to the post-conditions of the
test cases. These are the pre-conditions of next test cases.

Chapter 1

Page 48

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Implementation

Planning and

Analysis and
Design

Specifying and prioritizing test cases.


Creating test data and test scenarios.
Creating test suites.
Preparing test harnesses and (possibly) writing
automated test scripts.

Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

Verifying that the test system and test environment has been set
up correctly.
Before test execution: Verifying completeness of parts to be tested.

Chapter 1

Page 49

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Execution (1)

Planning and

Analysis and
Design

First check main functions. If any incidents


are found here, continuing with deeper tests
may not make much sense!
Execute test cases either manually or
by using test execution tools according to the
test procedure specification

Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

Record the test execution accurately and completely (versions of


test object, test tools and testware ) in the test log.

Chapter 1

Page 50

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Execution (2)

Planning and

Analysis and
Design

Compare actual results with expected results.


Report discrepancies as incidents and analyze
them. If a discrepancy of actual and expected
result is detected, it may not always mean
a software defect has been detected.
Check if any of the following aspects is true:

Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

incorrect expected result


test environment not set up properly
the test case specification is incorrect
Determine defect severity and priority for fixing
Repeat test activities as a result of action taken to resolve each
incident/defect (Re-Testing).

Chapter 1

Page 51

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Execution (3)

Planning and

Analysis and
Design

Create a test log

Implementation
and Execution

The test log must contain details on which


parts were tested, when, by whom,
how intensive and with what result.
On the one hand the description of test
execution in the test log has to be
comprehensive for those not directly involved
(e.g. clients)

Control

Evaluation and
Report

Test Closure

Finish

On the other hand it has to be traceable if the planned test strategy


has really been applied

Chapter 1

Page 52

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Evaluating Exit Criteria


Compare results of test execution with the
defined objectives of the test.
Checking test logs against the exit criteria
specified in test planning.
Reached test end?
Achieved coverage?
(Problem: unreachable program code)

Planning and

Analysis and
Design
Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

Deciding if more tests are needed or if the specified exit criteria


should be changed (only after consulting stakeholders!)
Further effort justified?
Addition practical factors for ending the test:
No more time
No more money

Chapter 1

Page 53

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Reporting (1)

Planning and

Analysis and
Design

Write a test summary report for


stakeholders.

Implementation
and Execution

The test summary report may have two


purposes:

Control

Evaluation and
Report

Test Closure

Communicate test status

Finish

Report on test completion


Refer to chapter 6 for more details on test reporting

Chapter 1

Page 54

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Reporting (2)

Planning and

Analysis and
Design

Example for exit criteria:


Found defects in order of failure severity

Implementation
and Execution

Control

Evaluation and
Report

New / Testing Hour


3

Test Closure

2,5

Finish

Failure Severity

1,5

low
medium

high

0,5

critical
0
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11

Chapter 1

Page 55

Software Testing Foundations Certified Tester

Wx: Calendar Week

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Reporting (3)

Planning and

Analysis and
Design

Example for exit criteria:


Comparison of found failures and corrected defects

Implementation
and Execution

Control

Evaluation and
Report

60

Test Closure

50
Finish

40
30
New failures
20
Corrected defects
10

In Process

0
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11
Chapter 1

Page 56

Software Testing Foundations Certified Tester

Wx: Calendar Week

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Closure Activities (1)

Planning and

Analysis and
Design

Collect data from completed test phases and


consolidate experience, testware, facts, metrics
etc.
Check which planned deliverables have been
delivered
Close incident reports or raise change requests
for any that remain open

Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

Document the acceptance of the system

Chapter 1

Page 57

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Start

Testing Tasks: Test Closure Activities (2)

Planning and

Analysis and
Design

Finalize and archive testware, the


test environment and the test infrastructure
Preserve and hand over the testware to the
maintenance organization everything should
be reusable during maintenance, so it has to be
portable and updatable

Implementation
and Execution

Control

Evaluation and
Report

Test Closure

Finish

Analyse and document any lessons learned


for later projects and to improve test maturity
Evaluation of test process
Assessment of test process
Recognizing potential for improvement
Improve the test process by applying the
findings

Chapter 1

Page 58

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Chapter

Fundamentals
of Testing

Chapter 1

Page 59

Terms and Motivation


Principles of Testing
Fundamental Test Process
Test Cases, Expected Results, Test Oraclesch\
Psychology of Testing
Ethics of Testing

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Criteria for Test Cases


Test cases for verification of specified results and of test object
delivered results and reactions
Positive test; expected input / operation (normal)).
Test cases that verify the specified handling of exceptional
situations and defects also need to be considered
Negative test; expected false input / operation (exceptional).
Remark: It is often difficult to create the conditions necessary
for execution of negative test cases (e.g. the overload of a
network connection).
Test cases for verification of reactions of the test object for invalid
and unexpected inputs or constraints, for which there was no
exception handling specified
Negative test / robustness test; unexpected erroneous inputs /
operations (catastrophic).

Chapter 1

Page 60

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Test Specification
High Level and Specific Test Cases (1)
Example:
A company has ordered a program that should calculate the
Christmas bonus of the staff in relation to the time they have been
working there. In the description of the requirements you find the
following text:
Staff who have been with the company for more that three years will
receive 50 % of the monthly salary as Christmas bonus. Staff who
have been with the company for more than five years will receive 75
%. Staff who have been with the company for more than eight years
will receive 100 % of their monthly salary.
How do the test cases look like?

Chapter 1

Page 61

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Test Specification
High Level and Specific Test Cases (2)
From the text you can set up the following relationship between the
allowance of the bonus and the time working for the company:

Chapter 1

years with the company <= 3

equals bonus = 0 %

3<

years with the company <= 5

equals bonus = 50 %

5<

years with the company <= 8

equals bonus = 75 %

years with the company

equals bonus = 100 %

Page 62

>8

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Test Specification
High Level and Specific Test Cases (3)
High Level (logical)
Test Case

x<=3

3<x<=5

5<x<=8

x>8

Expected result
(bonus in %)

50

75

100

Specific Test Case

Input value x
(years with the company)

12

Expected result
(bonus in %)

50

75

100

Input value x
(years with the company)

Remarks:
no pre- and post-conditions or constraints are considered
the test cases were not derived systematically
only positive tests with expected results

Chapter 1

Page 63

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Expected Results and Test Oracle


After each excecuted test case it must be decided whether there is
a failure or not.
For this decision the monitored result (actual result/behavior) has
to be compared with the expected result (expected
result/behavior).
The expected behavior of the test object has to be determined in
advance for each test case.
The tester must obtain this information from appropriate sources
when specifying a test case.
In this connection we also speak of an oracle or test oracle that
can be consulted and that predicts the expected results.

Chapter 1

Page 64

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Expected Results and Test Oracle


The expected results are to be deducted of the specification.
Following three possibilities:
The tester derives the expected date from the input date on the basis
of the specification of the test object. This is the common practise.
If a formal specification is available, an executable prototype can be
created with tool support. The results of the prototype can be used
as a test oracle for testing the actual program.
Parallel designs of the software program may be created by different
development groups. Each version of the program will then be tested
against another version using the same test data. If there are
different results in the two versions there must be a failure in one of
the versions. Only those failures that have the same effect in all the
versions remain undetected. This procedure is called Back-to-back
test.

Chapter 1

Page 65

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Other Test Oracles


Making use of user manuals
The system itself (create operational profile)
This is the worst possibility
(Tested!) previous versions
Programs with similar functionality
Exact values cannot always be predicted or calculated. Determine
range of tolerance, verify plausibility
Experience is important

Chapter 1

Page 66

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Chapter

Fundamentals
of Testing

Chapter 1

Page 67

Terms and Motivation


Principles of Testing
Fundamental Test Process
Test cases, Expected Results, Test Oracle
Psychology of Testing
Ethics of Testing

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Psychology of Testing
Making mistakes is human
- but who likes admitting that?
Development = constructive
Test = destructive?
Testing is an extremely creative
and intellectually challenging task

Myers, Glenford J.:


Systematic Testing of Programs
Oldenbourg, 2001 (7. Edition)

Is it possible for developer to test


his own program?
What to you think?

Chapter 1

Page 68

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Psychology of Testing Developer Test


Being blind to see own mistakes
If the developer has implemented a basic design error, e.g.
because he has misunderdstood the task, he is not able to
detect this through his own tests.
He will not be able to specify appropriate test cases.
No familiarisation
The developer knows his test object

Chapter 1

Page 69

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Psycholoy of Testing - Independent Test Team

An independent test team is unbiased


It his not his/their product
The possible assumptions and misunderstandings of the developer
are not necessarily the assumptions and misunderstandings of the
tester.

Need for familiarization


To create test cases the tester needs to gain knowledge about the test
object. This requires time.

Test Know-how
The tester should have test know-how
A developer probably does not have this or first has to gain it (often
with not enough time available)
Even better, the know-how has already been acquired at university

Chapter 1

Page 70

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Psychology of Testing Possible Levels


Tests are carried out with different levels of independence
By the developer himself
By colleagues of the developer in the same project
By persons of other departments
By persons of other organizations.
Tools can be used to support all possibilities.
The selection depends on the product as well as the project.
It is important to get the right mix and balance between
independent testing and tests performed by the developer.

Chapter 1

Page 71

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Psychology of Testing Communication of Defects


Communication of identified defects or incidents to the developer
and/or the management requires
neutral, fact-focused and constructive way of communication
Undisturbed, open communication.
Proving another persons error is not an easy task. It requires good
interpersonal skills.
Reproduceability of defects is important!
Recording the test environment
Differences to the development environment
Clear requirements, precise specification
Its not a bug, its a feature!
Mutual understanding between testers and developers
Collaboration rather than battles!

Chapter 1

Page 72

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Chapter

Fundamentals
of Testing

Chapter 1

Page 73

Terms and Motivation


Principles of Testing
Fundamental Test Process
Test cases, Expected Results, Test Oracle
Psychology of Testing
Ethics of Testing

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Ethics of Testing (1)

Software testers often have access to confidential and legally privileged


information. This must not be used inappropriately.

Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB
states the following code of ethics:

1. PUBLIC
Certified software testers shall act consistently with the public interest.
2. CLIENT AND EMPLOYER
Certified software testers shall act in a manner that is in the best interests
of their client and employer, consistent with the public interest.

Chapter 1

Page 74

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Ethics of Testing (2)


3. PRODUCT
Certified software testers shall ensure that the deliverables they provide
(on the products and systems they test) meet the highest professional
standards possible.
4. JUDGMENT
Certified software testers shall maintain integrity and independence in
their professional judgment.
5. MANAGEMENT
Certified software test managers and leaders shall subscribe to and
promote an ethical approach to the management of software testing.

Chapter 1

Page 75

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Ethics of Testing (3)


6. PROFESSION
Certified software testers shall advance the integrity and reputation of
the profession consistent with the public interest.
7. COLLEAGUES
Certified software testers shall be fair to and supportive of their
colleagues, and promote cooperation with software developers.
8. SELF
Certified software testers shall participate in lifelong learning regarding
the practise of their profession and shall promote an ethical approach to
the practice of the profession.

Chapter 1

Page 76

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Summary
Failure Not fulfilling the requirements!
Describing this situation as failure
cause: fault/defect in the software
that has been caused by an error made by a person
Tests are important measures for securing the quality criteria
Functionality, reliability, usability, efficiency, maintainability und
portability [according to ISO 9126]
Exhaustive testing is not possible. Tests are always random
checks, therefore they can detect failures only with a certain
probability
The intensity and amount of testing depends on the expected risk
of faulty behavior of the program
Tests should be prioritized

Chapter 1

Page 77

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Summary

The fundamental test process consists of the main activities


Test Planning and Control
Test Analysis and Design
Test Implementation and Execution
Evaluating Exit Criteria and Reporting
Test Closure Activities

A test case consists of


an input value and
an expected result as well as
preconditions and constraints for the execution of the test case, and
postconditions that have to be fulfilled after the test case.

While executing the test case, the test object shows its actual behavior. If there is a discrepancy
between expected and actual behavior there is a failure.

A test oracle determines the expected values for each test case before test execution

Humans make mistakes but they do not like admitting them!

Chapter 1

Page 78

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

By now, you should be able to answer the following


??? questions
Define the terms software failure, defect, error.
What is defect masking?
Explain the difference between testing and debugging.
Define the terms verification and validation.
Explain why tests cannot be exhaustive.
Name the main software quality attributes according to ISO 9126.
Define the term reliability of a system.
Explain the main activities of the fundamental test process.
What is a test oracle?
Why should developers not test their own programs?

Chapter 1

Page 79

Software Testing Foundations Certified Tester

Copyright 2011 to MSTB/GTB


V 1.0

Finally

The process cannot be stopped. The process cannot be stopped.

Installation
Copy file:

Cancel

Sort results

1 entry was found.


Would you like to sort the results?

Yes

Chapter 1

Page 80

Software Testing Foundations Certified Tester

No

Copyright 2011 to MSTB/GTB


V 1.0