Вы находитесь на странице: 1из 415

ISEB Foundation Certificate in Software Testing

Testing Terminology

© SIM Group Ltd., SQS Group AG, 2002


Testing Terminology

‰ IT is an industry riddled with mysterious words, terms and


TLA’s

‰ Software testing is no exception

‰ Historically there has not been a generally accepted set of


testing definitions

© SIM Group Ltd., SQS Group AG, 2002


Testing Terminology

‰ To resolve this issue, the BCS SIGIST created a Standard


Glossary of Testing Terms
z British Computer Society
z Special Interest Group In Software Testing

‰ This is now a British Standard


z BS7925-1

© SIM Group Ltd., SQS Group AG, 2002


Testing Terminology

‰ You will need to read the document and start to learn the
terminology used in software testing

‰ You will be exposed to the terminology and customers will


assume you know what they mean

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Why Testing is Necessary

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ In this session we will

‰ Understand what testing is and why it is necessary

‰ Define some of the words used in software testing

‰ Look at the cause and cost of errors

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Why Test?
To find bugs

What is a bug?

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

What is Testing?

“…the process of executing a program with the intent to certify its


Quality”
Mills

“…the process of executing a program with the intent of finding


failures / faults”
Myers

“…the process of exercising software to detect bugs & to verify that


it satisfies specified functional & non-functional requirements”

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Why is Testing becoming more important?

‰ The Y2K problem

‰ EMU

‰ e Commerce

‰ Increased user base

‰ Increased complexity

‰ Speed to Market

© SIM Group Ltd., SQS Group AG, 2002


Definitions

© SIM Group Ltd., SQS Group AG, 2002


Definitions

ERROR
“A human action that produces an incorrect result”

FAULT
“A manifestation of an ERROR in software”

© SIM Group Ltd., SQS Group AG, 2002


Definitions

FAULT
A Fault, if encountered, may cause a failure

FAILURE
“A deviation of the software from its expected delivery or
service”

© SIM Group Ltd., SQS Group AG, 2002


Definitions

DEFECT
“The departure of a quality characteristic from its specified value
that results in a product or service not satisfying its normal
usage requirements”

© SIM Group Ltd., SQS Group AG, 2002


Definitions

RELIABILITY
“The probability that software will not cause the failure of a
system for a specified period of time under specified
conditions”

© SIM Group Ltd., SQS Group AG, 2002


Definitions

QUALITY
“The totality of the characteristics of an entity that bear on its
ability to satisfy stated or implied needs”

© SIM Group Ltd., SQS Group AG, 2002


Testing & Quality

„ Does testing improve quality?

‰ Testing does not build Quality into the software

‰ Testing is a means of determining the Quality of the Software


under Test

© SIM Group Ltd., SQS Group AG, 2002


Definitions

QUALITY ASSURANCE

“All those planned actions used to fulfil the requirements for


quality”

© SIM Group Ltd., SQS Group AG, 2002


Why do errors occur?

© SIM Group Ltd., SQS Group AG, 2002


Why do errors occur?

„ How Errors Occur

‰ No one is perfect!
We all make mistakes or omissions

‰ The more pressure we are under the more likely we are to


make mistakes

‰ In IT development we have time and budgetary deadlines to


meet

‰ Poor Training

© SIM Group Ltd., SQS Group AG, 2002


Why do errors occur?

„ How Errors Occur

‰ Poor Communication

‰ Requirements not clearly defined

‰ Requirements change & are not properly documented

‰ Data specifications not complete

‰ ASSUMPTIONS!

© SIM Group Ltd., SQS Group AG, 2002


Why do errors occur?

„ The Cost Of Errors

‰ A single failure may incur little cost - or millions

‰ Report layouts may be wrong - little true cost

‰ Or a significant error may cost millions…


Ariane V, Venus Probe, Mars Explorer and Polar Lander

© SIM Group Ltd., SQS Group AG, 2002


Why do errors occur?

„ The Cost Of Errors

‰ In extreme cases a software or systems error may cost


LIVES

‰ Usually safety critical systems are tested exhaustively


Aeroplane, railway, nuclear power systems etc

‰ Unfortunately there are exceptions to this


London Ambulance Service

© SIM Group Ltd., SQS Group AG, 2002


Why do errors occur?

„ The Cost Of Errors

‰ The cost of defects increases proportionately (tenfold?) with


the passing of each successive stage in the system
development process before they are detected

‰ To correct a problem at requirements stage may cost £1

‰ To correct the problem post-implementation may cost £000’s

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Why Test?

‰ Identifies faults
Reduces live defects
Improves the quality of the users application
Increases reliability
To help ensure live failures don’t impact costs & profitability
To help ensure requirements are satisfied
To ensure legal requirements are met
To help maintain the organisation’s reputation

‰ Provides a measure of quality

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Quality Measurements

‰ Quality can be measured by testing for

correctness
reliability
usability
maintainability
testability
reusability

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ How much is enough?

‰ How do we know when to stop?


There are many factors to consider

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Exhaustive Testing

‰ Find all faults by testing everything?

‰ To test everything is rarely possible

‰ The time required makes it impractical

‰ The resource commitment required makes it impractical

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ If we can’t test everything, what can we do?

‰ Managing and reducing Risk

‰ Carry out a Risk Analysis of the application

‰ Prioritise tests to focus on the main areas of risk

‰ Apportion time relative to the degree of risk involved

‰ Understand the risk to the business of the software not


functioning correctly

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Should you stop testing when the product goes live?

‰ Continuing testing beyond the implementation date should be


considered

‰ Better that you find the errors than the users

© SIM Group Ltd., SQS Group AG, 2002


Why Testing is Necessary

„ Summary

‰ The purpose of testing is to find faults


Faults can be fixed, making better software
Better software is more reliable, less prone to failures
Establish the relationship between the software and its’
specification

‰ Testing enables us to measure the quality of the software

‰ This enables us to understand & manage the risk to the


business

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

The Test Process

© SIM Group Ltd., SQS Group AG, 2002


The Test Process

„ In this session we will

‰ Look at the steps involved in the test process

‰ Look in detail at each step

© SIM Group Ltd., SQS Group AG, 2002


The Test Process

„ What is the objective of a test?

‰ A “successful” test is one that does detect a fault

© SIM Group Ltd., SQS Group AG, 2002


The Test Process

TEST MANAGEMENT

SPECIFICATION

COMPLETION
RECORDING
EXECUTION
PLANNING

TEST ASSET MANAGEMENT


TEST ENVIRONMENT MANAGEMENT

© SIM Group Ltd., SQS Group AG, 2002


The Test Process

„ Five steps in the Test Process are

‰ Test Planning
‰ Test Analysis and Specification
‰ Test Execution
‰ Test Recording (Verification)
‰ Checking for Completion

© SIM Group Ltd., SQS Group AG, 2002


Test Planning

„ The Test Plan describes how


the Test Strategy is
implemented

© SIM Group Ltd., SQS Group AG, 2002


Test Planning

„ Contents of a Test Plan include

‰ Background ‰ Resources
‰ Reference documents ‰ Dependencies
‰ Approach ‰ Reporting
‰ Method ‰ Test Asset Identification
‰ Timetable ‰ Exit Criteria

© SIM Group Ltd., SQS Group AG, 2002


Test Planning

„ The most critical stage of the


process

„ Effort spent now will be


rewarded later

„ The foundation on which


testing is built

© SIM Group Ltd., SQS Group AG, 2002


Test Specification

„ Three step process

‰ Preparation & analysis

‰ Building test cases

‰ Define expected results

© SIM Group Ltd., SQS Group AG, 2002


Test Specification

„ Test preparation

‰ Analyse the Application


‰ Identify good test conditions
‰ Identify test cases
‰ Document thoroughly
‰ Cross-referencing

© SIM Group Ltd., SQS Group AG, 2002


Test Specification

„ Build Test Cases

‰ Test cases comprise:


z Standing Data
z Transaction Data
z Actions
z Expected Results

© SIM Group Ltd., SQS Group AG, 2002


Test Specification

„ Expected Results

‰ The outcome of each action

‰ The state of the application


during & after

‰ The state of the data during


& after

© SIM Group Ltd., SQS Group AG, 2002


Test Specification

„ Cross-Referencing &
Classification

‰ Enables maintainability of
Test Assets

‰ Allows testing to be
performed in a focussed
manner directed at specific
areas

© SIM Group Ltd., SQS Group AG, 2002


Test Execution

„ Test Execution checklist

‰ Test execution schedule / log


‰ Identify which tests are to be run
‰ Test environment primed & ready
‰ Resources ready, willing & able
‰ Back-up & recovery procedures in place
‰ Batch runs planned & scheduled

„ Then we are ready to run the tests

© SIM Group Ltd., SQS Group AG, 2002


Test Recording

„ Test verification

‰ If planning and preparation is sufficiently detailed this is the


easy part of testing

‰ The test is run to verify the application under test

‰ The test itself either passes or fails!

© SIM Group Ltd., SQS Group AG, 2002


Test Recording

„ The test log should record

‰ Software and test versions


‰ Specifications used as test base
‰ Test timings
‰ Test results
z Actual results
z Expected results

‰ Defect details for erroneous tests

© SIM Group Ltd., SQS Group AG, 2002


Test Completion

„ Test Exit Criteria

„ Used to determine when to


implement the software

‰ Key Functionality tested


‰ Test coverage
‰ Budget used?
‰ Defect detection rate
‰ Performance satisfactory

© SIM Group Ltd., SQS Group AG, 2002


The Test Process

„ Summary

‰ There are five steps in the Test Process

z Test Planning
z Test Analysis and Specification
z Test Execution
z Test Recording (Verification)
z Checking for Completion

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

The Psychology Of Testing

© SIM Group Ltd., SQS Group AG, 2002


The Psychology Of Testing

„ In this session we will

‰ Understand what qualities make good testers

‰ Look at a testers’ relationship with developers

‰ Look at a testers’ relationship with management

‰ Understand the issues with testing independence

© SIM Group Ltd., SQS Group AG, 2002


What makes a Tester?

„ Testing is primarily to find faults

‰ Can be regarded as ‘destructive’


‰ Development is ‘constructive’
‰ Testing asks questions
‰ Testers need to ask questions

„ A tester needs many qualities...

© SIM Group Ltd., SQS Group AG, 2002


What makes a Tester?

„ Intellectual qualities

‰ Can absorb incomplete facts


‰ Can work with incomplete facts
‰ Can learn quickly on many levels
‰ Good verbal communication
‰ Good written communication
‰ Ability to prioritise
‰ Self-organisation

© SIM Group Ltd., SQS Group AG, 2002


What makes a Tester?

„ Knowledge

‰ How projects work


‰ How computer systems and business needs interact
‰ What makes IT tick - technology
‰ What makes IT tick - commercial aspects
‰ Testing techniques
‰ Testing best practice
‰ To be able to think inside and outside of a system
specification

© SIM Group Ltd., SQS Group AG, 2002


What makes a Tester?

„ More skills to acquire

‰ How to find bugs - planning, preparation & execution


‰ How to understand systems
‰ How to read specifications
‰ How to extract testable functionality
‰ How to work efficiently
‰ How to focus on essentials

© SIM Group Ltd., SQS Group AG, 2002


Reporting Defects

„ Defects need to be reported to

‰ Developers to enable them to fix them


‰ Management so they can track progress

„ Communication to both groups is vital

© SIM Group Ltd., SQS Group AG, 2002


Communication with Developers

„ A good relationship is vital

‰ Developers need to keep testers up to date with changes to


the application

‰ Testers need to inform developers of defects to allow fixes to


be applied

© SIM Group Ltd., SQS Group AG, 2002


Communication with Management

„ Managers need progress reports

„ The best way is through metrics


‰ Number of tests planned & prepared
‰ Number of tests executed to date
‰ Number of defects raised & fixed
‰ How long planning, preparation and execution stages take

© SIM Group Ltd., SQS Group AG, 2002


The Psychology of Testing

Testing Independence

© SIM Group Ltd., SQS Group AG, 2002


Testing Independence

„ It is important that testing is separate from


development

‰ The developer is likely to confirm adherence not deviation

‰ The developer will make assumptions - the same when


testing as developing

© SIM Group Ltd., SQS Group AG, 2002


Testing Independence

„ Levels of Independence

‰ Low - Developers write their own tests

‰ Medium - Tests are written by another developer

‰ High - Tests written by an independent body


z Tests written by another section
z Tests written by another organisation

‰ Utopia - Tests generated automatically

© SIM Group Ltd., SQS Group AG, 2002


The Psychology Of Testing

„ Summary

‰ Testers require a particular set of skills


z The desire to break things
z The desire to explore and experiment
z Communication
z Questioning

‰ Testing requires a different mentality to development


z “Destroying” things rather than creating them
z Testing should be separate from development

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Re-Testing & Regression Testing

© SIM Group Ltd., SQS Group AG, 2002


Re-Testing & Regression Testing

„ In this session we will

‰ Understand how fault fixing and re-testing are key to the


overall testing process

‰ Look at how test repeatability helps

‰ Understand what regression testing is and where it fits in

‰ Understand how to select test cases for regression testing

© SIM Group Ltd., SQS Group AG, 2002


Re-Testing

„ Testing is directed at finding faults

‰ These faults will need to be corrected and a new version of


the software released

‰ The tests will need to be run again to prove that the fault is
fixed

‰ This is known as Re-Testing

© SIM Group Ltd., SQS Group AG, 2002


Re-Testing

„ The need for re-testing needs to be planned & designed


for

‰ Schedules need to allow for re-testing

‰ Tests need to be easily re-run

‰ The environment needs to be easily restored

© SIM Group Ltd., SQS Group AG, 2002


Regression Testing

“Re-testing of a previously tested program following


modification to ensure that FAULTS have not been
introduced or uncovered as a result of the changes
made”

© SIM Group Ltd., SQS Group AG, 2002


Regression Testing

8
Module 93 v1.3.5
How can you fix this
but break that? ~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
9
9
~~~~~~~~~~~~~~~~
Module 14 v1.4.2 ~~~~~~~~~~~~~~~~
Print (x);
Module 14 v1.4.3
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~ 9 ~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Declare String x =Date();
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Global String x = Date();
~~~~~~~~~~~~~~~~

9
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~ Module 41 v1.8.9
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
Declare Integer x = 0;
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
8
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
For x = 1 to 5
~~~~~~~~~~~~~~~~

© SIM Group Ltd., SQS Group AG, 2002


Regression Testing

„ Tests will also need to be re-run when checking


software upgrades

‰ Regression tests should be run whenever there is a change


to the software or the environment

‰ Regression tests are executed to prove aspects of a system


have not changed

‰ Regression Testing is a vital testing technique

© SIM Group Ltd., SQS Group AG, 2002


Regression Testing

„ Selecting cases for regression:

‰ Tests that cover safety or business critical functions

‰ Tests that are repetitive

‰ Tests that need accurate data

‰ Tests for areas that change regularly

‰ Tests of functions that have a high level of defects

© SIM Group Ltd., SQS Group AG, 2002


Regression Testing

„ Regression Testing is the ideal foundation for


Automation

‰ Selecting test cases is vital, and requires a degree of


knowledge of the applications and their expected evolution

© SIM Group Ltd., SQS Group AG, 2002


Re-Testing & Regression Testing

„ Summary

‰ Once a fault has been fixed the software MUST be re-tested


z New faults may have been introduced by the fix
z Existing faults may have been uncovered by the fix
z Tests need to be written to enable their re-use

‰ Re-testing is the rerunning of failed tests once a fix has been


implemented to check that the fix has worked

‰ Regression testing is running of a wider test suite to check for


unexpected errors in unchanged code

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Expected Results

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ In this session we will

‰ Understand the need to define expected results

‰ Understand where expected results can be found

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ Expected results = expected


outcomes

„ Identifying required behaviour

„ If the expected outcome of a


test is not defined the actual
output may be misinterpreted

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ Running a test with only a


general concept of the
outcome is fatal

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ It is vital the Expected Results


are defined with the tests,
before they are used

„ You cannot decide whether a


test has passed just because
it looks right

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ It may be possible to identify


the expected result of a test
based on knowledge or
experience of the
application… but not the code

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ Consult the Oracle!

© SIM Group Ltd., SQS Group AG, 2002


Expected Results

„ Summary

‰ Without defining expected results how do you know if a test


has passed or failed?

‰ Expected results can be found in the system specification and


by asking experienced business users

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Prioritisation of Tests

© SIM Group Ltd., SQS Group AG, 2002


Prioritisation of Tests

„ In this session we will

‰ Understand why we need to prioritise tests

‰ Understand how we decide the priority of individual tests

© SIM Group Ltd., SQS Group AG, 2002


Prioritisation of tests

„ It is not possible to test


everything

‰ Therefore faults will get


through to the live system

‰ We must do the best testing


possible in the time available

‰ This means we must


prioritise and focus testing
on the priorities

© SIM Group Ltd., SQS Group AG, 2002


Aspects to Consider

„ Severity „ Frequency of Change

„ Probability „ Vulnerability to error

„ Visibility „ Technical criticality

„ Priority of Requirements „ Complexity

„ Customer Requirements „ Resource availability

© SIM Group Ltd., SQS Group AG, 2002


Business Criticality

„ What elements of the


application are essential to
the success of the
organisation?

© SIM Group Ltd., SQS Group AG, 2002


Customer factors

„ How visible would a failure


be?

„ What does the customer


want?

© SIM Group Ltd., SQS Group AG, 2002


Technical Factors

„ How complex is it?

„ How likely is an error?

„ How often does this change?

© SIM Group Ltd., SQS Group AG, 2002


Prioritisation of Tests

„ Summary

‰ Due to the last minute nature of testing, it is necessary to


prioritise tests

‰ There will never be enough time to complete all tests

‰ Therefore those tests that cover those areas deemed most


important (to the business, highest risk etc) must be tested
first

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Models for Testing

© SIM Group Ltd., SQS Group AG, 2002


Models for Testing

„ In this session we will

‰ Look at two of the most commonly accepted testing models

© SIM Group Ltd., SQS Group AG, 2002


Testing Models

„ There are many approaches to testing

‰ Of these 2 are widely used & referred to:


z V,V&T
z The V-Model

‰ There are various other models


z They are not part of this course

© SIM Group Ltd., SQS Group AG, 2002


V,V&T

‰ Verification
z “The process of evaluating a system or component to determine
whether the products of the given development phase satisfies
the conditions imposed at the start of that phase”

‰ Validation
z “Determination of the correctness of the products of software
development with respect to the user needs and requirements”

‰ Testing
z “process of exercising software to verify that it satisfies specified
requirements and to detect faults”

© SIM Group Ltd., SQS Group AG, 2002


V-Model

„ The V-Model is the most commonly used model in


testing

‰ It represents the Software Development Life Cycle

‰ Shows the various stages in development and testing

‰ Shows the relationships between the various stages

© SIM Group Ltd., SQS Group AG, 2002


A simple V-Model

high level

T
requirements UAT
T
design system
T test
code unit
test
low level time

© SIM Group Ltd., SQS Group AG, 2002


A typical development life-cycle V-Model

Business Service Level


Requirements Testing

Operational User Acceptance


Requirements Testing

Design Integration
Specifications Testing (large)
Functional System
Specification Testing
level
Program Component
Specification Testing
time

© SIM Group Ltd., SQS Group AG, 2002


Models for Testing

V-Model - where do we start testing?

Requirements T UAT

Design T System Test

Code T Unit Test


level

time

© SIM Group Ltd., SQS Group AG, 2002


Models for Testing

V-Model - Start Testing Early

R PR R T T T
Reqs
UAT
RPR T T
Spec System Test
R P T
Code Unit Test
level

time

© SIM Group Ltd., SQS Group AG, 2002


Models for Testing

„ Summary

‰ The V-Model is the most common approaches model for


testing

‰ There are other models used for testing, but these are less
widely used
z Inc. V,V & T
z Other models have been created

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Economics of Testing

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ In this session we will

‰ Look at the cost of testing (or the cost of not testing)

‰ Appreciate the value of good, timely testing

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ The earlier a fault is found, the cheaper it is to remedy

‰ Faults in the requirements can lead to major re-engineering of


the entire system

‰ Many faults can be found using reviews


z Reviews of documentation and / or code

‰ Allows testing to find more ‘substantial’ faults

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ The average cost of fixing a defect increases tenfold


with every step of the development process

‰ In code reviews you will find and fix defects in an average of


1 - 2 minutes

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ In initial testing, defect fix times will average between


10 to 20 minutes

„ In integration testing each defect can cost an hour or


more

„ In system test each defect can cost 10 to 40 or more


engineer hours

Watts Humphrey

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Early test design can prevent fault multiplication

‰ Analysis of specification during test preparation often brings


faults in the specification to light

‰ If faults in documentation are not found then the system may


be developed incorrectly

© SIM Group Ltd., SQS Group AG, 2002


Economics of testing

An example

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in the requirements

‰ Change requirements document

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in the specifications

‰ Change requirements document

‰ Change specifications

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in the code

‰ Change requirements document

‰ Change specifications

‰ Change the code

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in system testing

‰ Change requirements document

‰ Change specifications

‰ Change the code

‰ Change the tests, re-test & regression test

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in UAT

‰ Change requirements document

‰ Change specifications

‰ Change the code

‰ Change the tests, re-test & regression test

‰ Re-do UAT

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in Live

‰ Change requirements document

‰ Change specifications

‰ Change the code

‰ Change the tests, re-test & regression test

‰ Re-do UAT
Plus...

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Finding the error in Live

‰ User to report and log fault

‰ Users must be advised of fix, work-around etc

‰ The live database may need to be fixed

‰ Deploying the new build may take hours

‰ Loss of business

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Additionally

‰ Faults found in coding or later require changes to the code


library

‰ Documentation and help files will have to be changed

‰ Every fault requires investigating and logging

‰ Administration costs

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ The cost of not Testing

‰ What is the cost of testing?

‰ Which is cheaper?
z Testing and finding faults before release
z Not testing and finding faults once the system is live

‰ Companies do not typically have figures for either

© SIM Group Ltd., SQS Group AG, 2002


Economics of Testing

„ Summary

‰ The purpose of testing is to find faults

‰ The earlier a fault is found the cheaper (and easier) it is to fix

‰ Starting testing early will help find faults quicker

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

High Level Test Planning

© SIM Group Ltd., SQS Group AG, 2002


High Level Test Planning

„ In this session we will

‰ Look at how a test plan is put together

‰ Understand how it should be used and maintained

‰ Understand why they are so important to a testing project

© SIM Group Ltd., SQS Group AG, 2002


High Level Test Planning

„ What is a test plan?

‰ A project plan for testing

‰ Covering all aspects of testing

© SIM Group Ltd., SQS Group AG, 2002


High Level Test Planning

„ Before you plan

‰ A test strategy must be in place

© SIM Group Ltd., SQS Group AG, 2002


The Different Sections of a Test Plan

‰ Test plan identifier ‰ Test deliverables

‰ Introduction ‰ Testing tasks

‰ Test items ‰ Environment

‰ Features to be tested ‰ Responsibilities


‰ Staffing and training needs
‰ Features not to be tested
‰ Schedules
‰ Approach
‰ Risks and contingencies
‰ Item pass / fail criteria
‰ Approvals
‰ Suspension criteria &
resumption criteria

Based upon IEEE 829-1998 standard for


software test documentation

© SIM Group Ltd., SQS Group AG, 2002


Test Plan Identifier

„ A unique identifier for the test plan

© SIM Group Ltd., SQS Group AG, 2002


Introduction

„ The introduction should

‰ Give an overview of the plan


z A summary of the requirements
z Discuss what needs to be achieved
z Detail why testing is needed

‰ Reference to other documents


z Quality assurance and configuration management

© SIM Group Ltd., SQS Group AG, 2002


Test Items

„ The various items to be used in the tests

‰ The software items

‰ Their versions numbers / identifiers

‰ How they will be handed over to testing

‰ References to relevant documents

© SIM Group Ltd., SQS Group AG, 2002


Features to Be Tested

„ List all features of the SUT that will be tested under this
plan

© SIM Group Ltd., SQS Group AG, 2002


Features Not to Be Tested

„ List all features of the SUT that will not be tested under
this plan

Between the test items, features to be tested and features not to be


tested we have scope of the project

© SIM Group Ltd., SQS Group AG, 2002


Approach

„ Describes the approach to testing the SUT

‰ This should be high level, but sufficient to estimate the time


and resources required
z What this approach will achieve
z Specify major activities
z Testing techniques
z Testing tools / aids
z Constraints to testing
z Support required - environment & staffing

© SIM Group Ltd., SQS Group AG, 2002


Item Pass / Fail Criteria

„ How to judge whether a test item has passed

‰ Expected vs. Actual results

‰ Certain % of tests pass

‰ Number of faults remaining (known and estimated)

‰ Should be defined for each test item

© SIM Group Ltd., SQS Group AG, 2002


Suspension criteria & resumption requirements

„ Reasons that would cause testing to be suspended

‰ Steps necessary to resume testing

© SIM Group Ltd., SQS Group AG, 2002


Test Deliverables

„ Everything that goes to make up the tests

‰ All documentation
z e.g. Specification, test plans, procedures, reports

‰ Code releases

‰ Testing tools
z Test management tools, automation tools, excel, word etc

‰ Test systems
z Manual and automated test cases

© SIM Group Ltd., SQS Group AG, 2002


Testing Tasks

‰ Preparation to perform testing


z Test case identification
z Test case design
z Test data storage
z Baseline application

‰ Special skills needed


z Spreadsheet skills, test analysis, automation etc

‰ Inter-dependencies

© SIM Group Ltd., SQS Group AG, 2002


Environment

„ Requirements for test environment

‰ Hardware & software


z PCs, servers, routers etc
z SUT, interfaced applications, databases

‰ Configuration
z Maybe operating systems or middleware to test against

‰ Facilities
z Office space, desks, internet access

© SIM Group Ltd., SQS Group AG, 2002


Responsibilities

„ Who is responsible?

‰ For which activities

‰ For which deliverables

‰ For the environment

© SIM Group Ltd., SQS Group AG, 2002


Staffing and Training Needs

‰ Staff required
z Test managers, team leaders, testers, test analysts

‰ Skill levels required


z Automation experience
z Spreadsheet skills, etc

‰ Training requirements
z Tools specific training
z Refresher courses, etc.
z Overall project resource requirements

© SIM Group Ltd., SQS Group AG, 2002


Schedule

„ Timescales, dates and milestones

‰ Resources required to meet milestones

‰ Availability of software and environment

‰ Deliverables

© SIM Group Ltd., SQS Group AG, 2002


Risks and Contingencies

„ What might go wrong?

‰ Actions for minimising impact on testing should things go


wrong

© SIM Group Ltd., SQS Group AG, 2002


Approvals

„ Who has approved the test plan

‰ Names and dates of approval

‰ Why is it so important
z Evidence that the document has been viewed
z Shows that the approach has been agreed and has the backing
of those who matter
z You have commitment, now make them stick to it!

© SIM Group Ltd., SQS Group AG, 2002


High Level Test Planning

„ Summary

‰ Test plans are created to ensure that the requirements are


understood

‰ To ensure that maximum test coverage is achieved

‰ To identify all items that will form the test

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Acceptance Testing

© SIM Group Ltd., SQS Group AG, 2002


Acceptance Testing

„ In this session we will

‰ Understand what acceptance testing is


z Why you would want to do it
z How you would plan it and prepare for it
z What you need to actually do it

‰ Understand the different types of acceptance testing

© SIM Group Ltd., SQS Group AG, 2002


Introduction

„ What is Acceptance Testing ?

"Formal testing conducted to enable a user, customer or other


authorised entity to determine whether to accept a system or
component."

© SIM Group Ltd., SQS Group AG, 2002


User Acceptance Testing

„ What is User Acceptance Testing (UAT)?

‰ Exactly what it says it is!!

‰ Users of end product conducting the tests

‰ The set-up will represent a working Environment

‰ Covers all areas of a project, not just the system

‰ A.K.A Business Acceptance Testing or Business Process


Testing

© SIM Group Ltd., SQS Group AG, 2002


Where does it fit in?

Unit
Test

Systems
Test

UAT

Alpha /
Beta

© SIM Group Ltd., SQS Group AG, 2002


A typical development life-cycle V-Model

Business Service Level


Requirements Testing

Operational User Acceptance


Requirements Testing

Design Integration
Specifications Testing (large)
Functional System
Specification Testing
level
Program Component
Specification Testing
time

© SIM Group Ltd., SQS Group AG, 2002


Planning UAT

„ Why plan?

„ Things to consider

„ Preparing Tests
‰ Manual
‰ Automated

© SIM Group Ltd., SQS Group AG, 2002


Why Plan?

„ If you don’t then how do you know you have achieved


what you set out to do!

‰ Avoids repetition

‰ Test according to code releases

‰ Makes efficient and effective use of time and resources

© SIM Group Ltd., SQS Group AG, 2002


Things to consider

„ Timescales

„ Resources

„ Availability

© SIM Group Ltd., SQS Group AG, 2002


Preparing your tests

„ Take a logical approach

‰ Identify the business processes

‰ Separate into everyday business scenarios

© SIM Group Ltd., SQS Group AG, 2002


How the business fits together...

The Business

Process Process Process

Task Task Task Task Task Task Task Task Task

© SIM Group Ltd., SQS Group AG, 2002


Data Requirements

„ Copied Environments

„ Created Environments

© SIM Group Ltd., SQS Group AG, 2002


Running the Tests

„ Order of Tests

„ Confidence Checks

„ Automated and Manual test runs

© SIM Group Ltd., SQS Group AG, 2002


Contract Acceptance

„ “A demonstration of the acceptance criteria”

‰ Acceptance criteria will have been defined in the contract

‰ Before the software is accepted it is necessary to show that


its matches its’ specification as defined in the criteria

© SIM Group Ltd., SQS Group AG, 2002


Alpha Testing

„ Letting your customers do your testing

‰ Requires a stable version of the software

‰ People who represent your target market use the product in


the same way(s) as if they had bought the finished version

‰ Alpha testing is conducted in-house (at the developers site)

© SIM Group Ltd., SQS Group AG, 2002


Beta Testing

„ As Alpha testing but users perform tests at their site

© SIM Group Ltd., SQS Group AG, 2002


Acceptance Testing

„ Summary

‰ You can UAT anything!

‰ Every Deliverable should pass through UAT

‰ User representation is VITAL

‰ If the product does not pass UAT then a decision about


implementation needs to be made

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Integration Testing in the Large

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ In this session we will

‰ Understand what Integration Testing is

‰ Understand what Integration Testing in the Large is

‰ Look at how & why we test system integration

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

‰ Integration testing
z “Testing performed to expose faults in the interfaces and in the
interaction between integrated components”

‰ Integration testing in the large


z Integration Testing in the Large is testing performed to expose
faults in the interfaces and in the interaction between systems

‰ Why do we need to do integration testing in the large


z Just because the SUT works in the test lab doesn’t mean it will
work in the real world

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ All components of a computer system work on only


one thing

DATA

‰ Data is manipulated within the SUT

‰ It may then passed to another system

‰ This is done to satisfy a business process requirement

© SIM Group Ltd., SQS Group AG, 2002


A typical system

Env Env Env


SUT SUT SUT

Data Model

Databases

Data Ext apps.

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ Integration Testing on a Large Scale

‰ Test the way the SUT interfaces with external systems

‰ The elements may run on the same and / or different


platforms

‰ They may be located in separate locations and use


communication protocols to pass data back & forth

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ Typical approach

‰ Test the system


z In a test environment
z Use stubs & drivers where external systems aren’t available

‰ Test the system “in situ”


z In a replica of the production / live environment
z Use stubs & drivers where external systems aren’t available

‰ Test the system and its’ interfaces with other systems


z In a replica of the production / live environment
z Access test versions of the external systems

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ Fundamentally we are still looking at

DATA

‰ & how that data is obtained, processed, manipulated, stored


and made available or transported for use elsewhere within or
outside the organisation

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ Additional challenges posed by large scale:

‰ Multiple Platforms
‰ Communications between platforms
‰ Management of the environments
‰ Coordination of changes
‰ Different technical skills required

© SIM Group Ltd., SQS Group AG, 2002


The approach used

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing

„ Planning testing

‰ We need to determine when and at what levels Integration


Testing will be performed

‰ Identify the boundaries

‰ A similar process will need to be followed for all elements


z Once these elements have been tested individually they are
further integrated to support the business process

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing

„ Approaches to integration testing

‰ Incremental
z Top down
z Bottom-up
z Functional

‰ Non-incremental
z “Big bang”

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Large

„ Summary

‰ Integration testing is needed to test how components interact


with each other and how the system under test works with
other systems

‰ Large scale integration testing is needed to test how the


system under test works with other systems

‰ Just because a system works in a test environment doesn’t


mean it will work in a live environment

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Non-Functional System Testing

© SIM Group Ltd., SQS Group AG, 2002


Non-Functional System Testing

„ In this session we will

‰ Understand what Non-Functional System Testing is

‰ Understand the need for non-functional system testing

‰ Look at the various types of non-functional system testing

© SIM Group Ltd., SQS Group AG, 2002


Non-Functional System Testing

„ Non-Functional System Testing is defined as:

“Testing Of Those requirements that do not relate to the


Functionality e.g. Performance, Load, usability, security etc”

‰ Users, generally, focus on the Functionality of the system

© SIM Group Ltd., SQS Group AG, 2002


Non-Functional System Testing

‰ Functional testing may show that the system performs as per


the requirements

‰ Will the system work if now deployed?

‰ There are other factors critical to the successful use of a


system

© SIM Group Ltd., SQS Group AG, 2002


Security Testing

“Testing whether the system meets specified security


requirements”

‰ How easy is it for an unauthorised user to access the


system?

‰ How easy is it for an unauthorised person to access the data?

© SIM Group Ltd., SQS Group AG, 2002


Usability Testing

“Testing the ease with which users can learn and use the
product”

‰ Employ people to use the application as a user and study


how they use it

‰ A beta-test may be a cheap way of doing Usability testing

‰ There is no simple way to examine HOW people use the


product

© SIM Group Ltd., SQS Group AG, 2002


Storage Testing

“Testing whether the system meets its specified storage


objectives”

‰ Study how memory and storage is used by the product

‰ Predict when extra capacity may be needed

© SIM Group Ltd., SQS Group AG, 2002


Installability Testing

“Testing concerned with the installation procedures for the


system”

‰ Does the installation work?

‰ Is it easy to carry out?

‰ Does it uninstall correctly?

© SIM Group Ltd., SQS Group AG, 2002


Documentation Testing

“Testing concerned with the accuracy of the documentation”

© SIM Group Ltd., SQS Group AG, 2002


Recovery Testing

“Testing aimed at verifying the system’s ability to recover from


varying degrees of failure”

‰ Should include recovery from system back-ups

© SIM Group Ltd., SQS Group AG, 2002


Load Testing

“Testing geared to assessing the applications ability to deal with


the expected throughput of data & users”

© SIM Group Ltd., SQS Group AG, 2002


Stress Testing

“Assesses individual components by exercising them to and


beyond the limits of expected use”

© SIM Group Ltd., SQS Group AG, 2002


Performance Testing

“Tests the efficiency of individual components of an application”

© SIM Group Ltd., SQS Group AG, 2002


Volume Testing

“Testing where the system is subjected to large volumes of


data”

© SIM Group Ltd., SQS Group AG, 2002


Non-Functional System Testing

„ Summary

‰ Just because a system’s functions have been tested doesn’t


mean that testing is complete

‰ There are a range of non-functional tests that need to be


performed upon a system

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Functional System Testing

© SIM Group Ltd., SQS Group AG, 2002


Functional System Testing

„ In this session we will

‰ Understand what functional system testing is

‰ Understand the benefits of functional system testing

© SIM Group Ltd., SQS Group AG, 2002


Functional System Testing

„ What is System Testing?

‰ Testing of the complete system


z May be the last step in integration testing in the small
z May be the first time that enough of the system has been put
together to make a working system

‰ Ideally done by an independent test team

‰ Two types - functional and non-functional

© SIM Group Ltd., SQS Group AG, 2002


Functional System Testing

„ A Functional Requirement is

“A requirement that specifies a function that a system or system


component must perform”

‰ Functional System testing is geared to checking the function


of the system against specification

‰ May be requirements based or business process based

© SIM Group Ltd., SQS Group AG, 2002


Functional System Testing

„ Testing Based on Requirements

‰ Requirements specification used to derive test cases

‰ System is tested to ensure the requirements can be met

© SIM Group Ltd., SQS Group AG, 2002


Functional System Testing

„ Testing carried out against Business processes

‰ Based on expected use of the system

‰ Builds use cases - test cases that reflect actual or expected


use of the system

© SIM Group Ltd., SQS Group AG, 2002


© SIM Group Ltd., SQS Group AG, 2002
Functional System Testing

„ Summary

‰ Functional system testing allows us to test the system against


the specifications, user requirements and business processes

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Integration Testing in the Small

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

„ In this session we will

‰ Understand what “Integration Testing in the Small” is

‰ Look at how & why we do “Integration Testing in the Small”

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

‰ Integration testing
z “Testing performed to expose faults in the interfaces and in the
interaction between integrated components”

‰ Integration testing in the small


z Testing performed to expose faults in the interfaces and in the
interaction between components

© SIM Group Ltd., SQS Group AG, 2002


A sample system

Databases

The SUT
- made up of 000’s of units

External
systems

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

„ All components of a computer application work on only


one thing

DATA

‰ Data is passed from one component to another

‰ Data is passed from one system to another

‰ Software components will be required to add, amend, &


delete information, validate the information and report on the
information

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

„ Integration testing

‰ Testing should start with the smallest, definable component

‰ Building tested components up to the next level of definition

‰ Eventually the complete business processes is tested ‘End to


End’

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

„ Testing components

‰ Each component needs to be tested individually to ensure it


processes the data as required

‰ This is Component Testing

‰ All possible options & paths should be tested

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

„ Once components have been tested we can link them


together to see if they can work together

‰ This is Integration Testing in the Small


z Linking components together to ensure that they communicate
correctly
z Gradually linking more and more components together
z Prove that the data is passed between components as required
z Steadily increase the number of components and create & test
sub-systems and then the complete system

© SIM Group Ltd., SQS Group AG, 2002


The approach used

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing

„ Planning testing

‰ We need to determine when and at what levels Integration


Testing will be performed

‰ Identify the boundaries

‰ A similar process will need to be followed for all elements


z Once these elements have been tested individually they are
further integrated to support the business process

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing

„ Stubs and drivers

‰ Used to emulate the “missing bits”


z External systems
z Sub-systems
z Missing components

‰ May need to be written specifically for the SUT

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing

„ Approaches to integration testing

‰ Incremental
z Top down
z Bottom-up
z Functional

‰ Non-incremental
z “Big bang”

© SIM Group Ltd., SQS Group AG, 2002


Integration Testing in the Small

„ Summary

‰ Integration testing is needed to test how components interact


with each other and how the system under test works with
other systems

‰ Integration testing in the small is concerned with the


interaction between the various components of a system

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Component Testing

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ In this session we will

‰ Understand what Component Testing is

‰ Look at the Component Testing Process

‰ Look at the myriad types of Component Testing

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ What is component testing?

‰ “The testing of an individual software component”

„ What is a component?

‰ “A minimal software item for which a separate specification is


available”

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ Component testing is the lowest form of testing

‰ At the bottom of the V-model

‰ Each component is tested in isolation


z Prior to being integrated into the system

‰ Involves testing the code itself


z Test the code in the greatest detail
z Usually done by the components developer

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ Component Test Process

‰ Component Test Planning


‰ Component Test Specification
‰ Component Test Execution
‰ Component Test Recording
‰ Checking for Component Test Completion

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ Testing techniques

‰ Equivalence Partitioning ‰ Data Flow Testing

‰ Boundary Value Analysis ‰ Branch Condition Testing


‰ Branch Condition
‰ State Transition Testing
Combination Testing
‰ Cause-Effect Graphing ‰ Modified Condition Decision
‰ Syntax Testing Testing
‰ LCSAJ Testing
‰ Statement Testing
‰ Random Testing
‰ Branch/Decision Testing
‰ Other Testing Techniques

For definitions see BS7925-


BS7925-2

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ What about “Playtime”?

‰ Often a short unscripted session with a component can be of


benefit

‰ More common at later stages (once formal test techniques


have been completed)

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ Not done too well?

„ Some basics would quickly improve it:


‰ Checklists
‰ Standards
‰ Procedures
‰ Code & version control
‰ Sign off

© SIM Group Ltd., SQS Group AG, 2002


Component Testing

„ Summary

‰ Component testing is intended to test a software component


before it is knitted into the system

‰ There are lots of different approaches

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Maintenance Testing

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ In this session we will

‰ Look at the challenges that face testers when an application


changes post implementation

‰ See how to ensure that maintenance applied to the system


does not cause failures

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ What is Maintenance Testing?

‰ Testing of changes to existing, established systems


z Where maintenance has been performed on the (old) code

‰ Checking that the fixes have been made and that the system
has not regressed

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ The Challenge

‰ Old Code

‰ No Documentation

‰ Out of Date Documentation (worse)

‰ How much testing to do and when to stop testing


z Mitigation of risk vs. lost revenue

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ Often applications will need maintenance

‰ In doing this minor changes will be required

‰ These need to be tested quickly & effectively to allow service


to be restored

‰ These systems may have been in place and working for


years
z They are likely to be vital to the running of the business
z They may be in use 24/7

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ You need to be able to test the changes quickly and


effectively

‰ As well as being able to prove that the change introduced


does not impact the other functions

‰ A Regression test is in order

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ A regression test is

‰ “Retesting of a previously tested program following


modification to ensure that faults have not been introduced or
uncovered as a result of the changes made”

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ Other Risks

‰ Reactive, high risk when changing

‰ Impact analysis is vital but is difficult

‰ Stopping testing too soon - errors may be missed

‰ Stopping testing too late - missed business through delayed


implementation

© SIM Group Ltd., SQS Group AG, 2002


Maintenance Testing

„ Summary

‰ All established systems need maintenance from time to time

‰ The changed code / function will need to be tested

‰ A regression test will need to be executed to ensure that the


change(s) have been made correctly and that they have not
affected some other part of the system

‰ Impact analysis is key, but difficult

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Black & White Box Testing

© SIM Group Ltd., SQS Group AG, 2002


Black & White Box Testing

„ In this session we will

‰ Understand the differences between Black & White Box


testing and where they feature in a testing lifecycle

‰ Understand how a systematic approach provides confidence

‰ Understand how tools can be used to improve and increase


productivity

© SIM Group Ltd., SQS Group AG, 2002


Black & White Box Testing

„ Black Box testing

‰ “Test Case selection that is based on an analysis of the


specification of a component without reference to its internal
workings”

„ White Box testing

‰ “Test Case selection based on an analysis of the internal


structure of a component”

© SIM Group Ltd., SQS Group AG, 2002


Black Box Testing

„ Concentrates on the Business Function

‰ Can be used throughout testing cycle

‰ Dominates the later stages of testing


z Although is relevant throughout the development life cycle

‰ Little / No knowledge of the underlying code needed

© SIM Group Ltd., SQS Group AG, 2002


Black Box Testing

„ We have all done it!

‰ What do you do when


changing a light bulb?

‰ `Remove Old Bulb

‰ Insert New Bulb

‰ Switch light / lamp on

‰ Switch light / lamp off

© SIM Group Ltd., SQS Group AG, 2002


White Box Testing

„ Testing at a detailed level

‰ A.K.A. “Glass Box testing” or “Structural testing”

‰ Focuses on Lines of Code

‰ Looks at specific conditions

‰ Looks at the mechanics of the Application

‰ Useful in the early stages of testing.

© SIM Group Ltd., SQS Group AG, 2002


White Box Testing

„ Requires a detailed knowledge of the code

‰ Business Function / Process not a prime consideration

‰ Aim to achieve code coverage, not just functional coverage

‰ Plays a lesser role later in the testing cycle

© SIM Group Ltd., SQS Group AG, 2002


White Box Testing

„ Code Coverage objective

‰ To devise tests that exercise each significant line of program


code at least once

‰ This does not mean every combination of paths and data -


which is normally unattainable

© SIM Group Ltd., SQS Group AG, 2002


Black & White Box Testing

„ Techniques and measurements

‰ Systematic techniques exist for black and white box testing


z Give us a systematic approach to testing
z Test cases are developed in a logical manner
z Enable us to have a sub-set of tests that have a high probability
of finding faults

‰ By using defined techniques we should be able to produce


corresponding measurements
z Means we can see how testing is progressing

© SIM Group Ltd., SQS Group AG, 2002


Black & White Box Testing

„ Software Testing Tools

‰ Are useful for both Black & White Box Testing

‰ Can increase productivity and quality of tests

‰ Particularly useful for white box testing

© SIM Group Ltd., SQS Group AG, 2002


Black & White Box Testing

„ Summary

‰ A systematic approach is needed for both

‰ Tests need to be planned, prepared, executed and verified


against the code

‰ Expected results need to be defined and understood

‰ Tools can help increase productivity and quality

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Black box test techniques

© SIM Group Ltd., SQS Group AG, 2002


Black box test techniques

„ In this session we will

‰ Understand what black box testing is

‰ Look at some of the different types of black box testing

© SIM Group Ltd., SQS Group AG, 2002


Black box test techniques

„ Black box testing is

‰ “Test Case selection that is based on an analysis of the


specification of a component without reference to its internal
workings”

© SIM Group Ltd., SQS Group AG, 2002


Black box test techniques

„ Why do we need black box test techniques?

‰ Exhaustive testing is not possible


z Due to the constraints of time, money and resources

‰ Therefore we must create a sub-set of tests


z These must have fewer tests, but should not reduce coverage

‰ We should also focus on areas of likely risk


z Those places where mistakes may occur

© SIM Group Ltd., SQS Group AG, 2002


Black box test techniques

„ Each black box test technique has

‰ A technique
z i.e. how to do it

‰ A test case design approach


z How to create test cases using the approach

‰ See BS7925-2 for detailed information

© SIM Group Ltd., SQS Group AG, 2002


Black box test techniques

„ BS7925-2 lists all the black box test techniques

‰ Equivalence Partitioning
‰ Boundary Value Analysis
‰ State Transition
‰ Cause & Effect Graphing
‰ Syntax Testing
‰ Random Testing

© SIM Group Ltd., SQS Group AG, 2002


Equivalence Partitioning

„ Uses a model of the component to

‰ Partition Input & Output values into sets such that each value
within a set can be reasonably expected to be treated in the
same manner

‰ Therefore only one example of that set needs to be input to or


resultant from the component

© SIM Group Ltd., SQS Group AG, 2002


Equivalence Partitioning

„ Test Case Design

‰ Inputs to the component

‰ Partitions exercised:
z Valid partitions
z Invalid partitions

‰ Expected results

© SIM Group Ltd., SQS Group AG, 2002


Equivalence Partitioning

„ Example

‰ Course grade

z Coursework - out of 25 marks


z Examination - out of 75 marks

z Grades A-D based on Coursework and Examination marks

z Identify Valid Partition and Invalid Partitions


z Define test cases including expected results

© SIM Group Ltd., SQS Group AG, 2002


Boundary Value Analysis

„ Uses a model of the component to

‰ Identify the values close to and on partition boundaries

‰ For input and output, valid & invalid data

‰ Chosen specifically to exercise the boundaries of the area


under test

© SIM Group Ltd., SQS Group AG, 2002


Boundary Value Analysis

„ Test Case Design

‰ Inputs to the component

‰ Boundaries to be exercised:
z Just below
z On
z Just above

‰ Expected results

© SIM Group Ltd., SQS Group AG, 2002


Boundary Value Analysis

„ Examples

‰ Using the grading system defined earlier

‰ The exam mark and the coursework mark both have two
boundaries

‰ Select values on, immediately above and immediately below


each boundary

© SIM Group Ltd., SQS Group AG, 2002


State Transition

„ Uses analysis of the specification of the component to


model its behaviour by state transitions

‰ Identifies
z The various states that the software may be in
z The transitions between these points
z The events that cause the transactions
z Actions resulting from the transactions

© SIM Group Ltd., SQS Group AG, 2002


State Transition

„ Test Case Design

‰ Identify possible transitions

‰ Identify initial state and the data / events that will trigger the
transition

‰ Identify the expected results

© SIM Group Ltd., SQS Group AG, 2002


State Transition

„ Example

‰ Using the course grade example

‰ There is a grade indicator on screen


z When coursework and examination marks are entered it
displays the persons grade (A-D)

‰ Select sets of data to test for each grade


z And extra sets of data to change from one grade to the next

© SIM Group Ltd., SQS Group AG, 2002


Other techniques

‰ Cause & Effect Graphing & Tabling


z Analysis of the specification to produce a graphical model of
causes and their effects
z Cases expressed as Boolean conditions in the form of a
decision table

‰ Syntax Testing
z Uses the syntax of the components’ inputs as the basis for the
design of the test case

‰ Random Testing
z Not as odd a concept as it sounds
z Not Ad-Hoc testing, but a means of testing functionality with a
randomly selected set of values

© SIM Group Ltd., SQS Group AG, 2002


Black Box Testing Techniques

„ Summary

‰ Black box testing concentrates on testing the features of the


system

‰ Techniques enable us to maximise testing


z Create an achievable set of tests that offer maximum coverage
z Ensure possible areas of risk are tested

‰ Black box testing is relevant throughout the testing process

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

White Box Test Techniques

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ In this session we will

‰ Understand what White Box testing is

‰ Look at some of the different types of White Box testing

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ White Box testing

‰ “Test Case selection based on an analysis of the internal


structure of a component”

‰ Also known as Glass Box testing, Clear Box testing

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Why do we need White Box


techniques?

‰ Provide formal structure to


testing code

‰ Enable us to measure how


much of a component has been
tested

‰ Example
z <100 lines of code
z 100,000,000,000,000 possible
paths
z At 1,000 tests per second
would still take 3,170 years to
test all paths

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ To plan and design effective cases requires a


knowledge of the

‰ Programming language used

‰ Databases used

‰ Operating system(s) used

‰ And ideally knowledge of the code itself

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ BS7925-2 lists all the white box test techniques

‰ Statement Testing
‰ Branch / Decision Testing
‰ Branch Condition Testing
‰ Branch Condition Combination Testing
‰ Modified Condition Decision Testing
‰ Linear Code Sequence & Jump
‰ Data Flow Testing

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Statement Testing

‰ “A test case design technique for a component in which test


cases are designed to execute statements”

‰ Test cases are designed and run with the intention of


executing every statement in a component

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Statement Testing example

a;
if (b)
{
c;
}
d;

Any test case with b TRUE will achieve full statement coverage

NOTE: Full statement coverage can be achieved without


exercising with b FALSE

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Branch / Decision testing

‰ A technique used to execute all branches the code may take


based on decisions made

‰ Test Cases designed to ensure all branches & decision points


are covered

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Branch / Decision testing example

a;
if (b)
{
c;
}
d;

100% statement coverage requires 1 test case (b = True)

100% branch / decision coverage requires 2 test cases


(b = True & b = False)

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Other techniques

‰ Branch Condition testing, Branch Condition Combination


testing & Modified Condition Decision testing
z Condition testing is based upon an analysis of the conditional
control flow within the component

‰ LCSAJ
z Linear Code Sequence And Jump

‰ Data flow testing


z Tests the flow of data through a component

© SIM Group Ltd., SQS Group AG, 2002


White Box Test Techniques

„ Summary

‰ White box testing can be done immediately after code is


written
z Doesn’t need the complete system
z Does need knowledge of the code

‰ A combination of all techniques are required for a successful


test
z Don’t rely on just one technique

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Error Guessing

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ In this session we will

‰ Understand how you can use experience to ‘predict’ where


errors may occur

‰ Understand how this can benefit testing

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ Used to guess where errors may be lurking

‰ Based on information about how the system has been put


together and previous experience of testing it

‰ Use to complement more systematic techniques not instead


of

‰ Not ad-hoc testing, but testing that targets certain parts of the
application

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ Based on the user's or tester's experience of the


commercial aspects of the system

‰ e.g.

z Table based calculations such as calculating benefits payments

z Where the user is allowed a high degree of flexibility in GUI


navigation using multiple windows

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ Experience of the developers or the development cycle

‰ e.g.

z Knowledge of an individual's ‘style’

z Knowledge of the development lifecycle and especially the


change management process

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ Experience of the operating systems used

‰ e.g.

z It is known that Windows NT is better at storage management


than Windows 95

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ Should still be planned

‰ Is part of the test process


z Should be used as a supplement to systematic techniques

‰ Ad-Hoc testing can be encouraged if


z The tester maintains a log of those tests performed & is able to
reasonably recollect what actions were performed

‰ If you have the time to run an extra (Ad-Hoc) test that passes
there is little loss
z If it finds a defect then that’s a bonus

© SIM Group Ltd., SQS Group AG, 2002


Error Guessing

„ Summary

‰ You can never do enough testing!


z The more tests that you can run the more errors you may find

‰ Error guessing can help ensure that areas where defects are
likely to occur are fully tested

‰ Uses experience and gut feeling to supplement systematic


testing techniques

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Reviews and the Test Process

© SIM Group Ltd., SQS Group AG, 2002


Static Testing

„ Over the next three sessions we will

‰ Discover the various approaches to Static Testing

‰ Discover the benefits of Static Testing

‰ Discover the issues with Static Testing

© SIM Group Ltd., SQS Group AG, 2002


Static Testing

„ What is Static Testing?

‰ “Testing of an object without execution on a computer”

© SIM Group Ltd., SQS Group AG, 2002


Static Testing

Session Topics

1. Reviews and the Test Process


2. Types of Review
3. Static Analysis

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ Session Objectives

‰ To identify what and when to Review

‰ To cover the Costs & Benefits of Reviews

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ Testing without data


‰ “Testing of an object without execution on a computer”

„ How is this done


‰ By reviewing the system deliverables

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ Why review

‰ To identify faults as soon as possible in the development


lifecycle

‰ Reviews offer the chance to find faults in the system


specifications

‰ This should lead to:


z Development productivity improvements
z Reduced development time-scales
z Lifetime cost reductions
z Reduced fault levels

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ When to review?

‰ As soon as an object is ready, before it is used as a product


or the basis for the next step in development

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ What to Review

‰ Anything and everything can be reviewed

‰ Requirements, System and Program Specifications should be


reviewed prior to publication

‰ System Design deliverables should be reviewed both in terms


of functionality & technical robustness

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ What should be reviewed?

‰ Program specifications should be reviewed before


construction

‰ Programs should be reviewed before execution

‰ Test Plans should be reviewed before execution

‰ Test Results should be reviewed before implementation

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ Reviews can be hazardous

‰ If misused they can lead to friction

‰ The errors & omissions found should be regarded as a good


thing

‰ The author(s) should not take errors & omissions personally

‰ It is a positive step to identify defects and issues before the


next stage

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ Building a Quality culture

‰ In order for them to work, Reviews must be regarded as a


positive step and must be properly organised

‰ This is often a part of the company’s Quality culture


z Define the review panel
z Define the roles of its’ members
z Define the review procedures
z “Sell” the quality culture to staff

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ There are other dangers in a regime of regular reviews

‰ Lack of Preparation

‰ “Familiarity breeds contempt”

‰ No follow up to ensure correction has been made

‰ The wrong people doing them

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ Systems development is very complex

‰ This means that it is difficult to build a system with all


elements completely & accurately specified

‰ By reviewing a deliverable, its deficiencies can be identified


and remedied earlier – saving time and money

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

What else can Testers review?


„ Test plans „ Causes of defect
„ Test cases „ Test metrics
„ Test results „ Development metrics
„ Defects „ Operational defects

We can measure how well we do

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ The costs of reviews

‰ On-going reviews cost approximately 15% of development


budget

‰ This includes activities such as the review itself, metric


analysis & process improvement

‰ Reviews are highly cost effective


z Finding and fixing faults in reviews is significantly cheaper than
finding and fixing faults in later stages of testing

© SIM Group Ltd., SQS Group AG, 2002


Reviews & the Test Process

„ The benefits of reviews

‰ Reviews save time and money

‰ Development productivity improvements


z People take greater care
z They have more pride in doing good work
z They have a better understanding of what they are required to
deliver

‰ Reduced development costs


z Reduced fault levels
z Reduced lifetime cost reductions

© SIM Group Ltd., SQS Group AG, 2002


Reviews and the Test Process

„ Summary

‰ Reviews enable us to test that the systems specification is


correct and relates to the users’ requirements

‰ Anything generated by a project can be reviewed

‰ In order for them to be effective, reviews must be well


managed

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Types of Review

© SIM Group Ltd., SQS Group AG, 2002


Static Testing

Session Topics

1. Reviews and the Test Process


2. Types of Review
3. Static Analysis

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ In this session we will

‰ Look at the types of review & the activities performed


‰ Look at goals & deliverables
‰ Look at roles & responsibilities
‰ Examine the suitability of each type of review
‰ Discover review pitfalls

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

‰ Author’s Review
‰ Informal Review
‰ Walkthroughs
‰ Presentations
‰ Prototypes
‰ Checklists
‰ Templates
‰ Technical Review
‰ Inspections

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Author’s review

‰ Before issuing a document, the author should ALWAYS


check
z Conformance to company standards
z Fitness for purpose
z Presentation
z Spelling

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Informal Review

‰ It is very helpful to ask a colleague to read through the


deliverable to
z Identify jargon to eliminate or translate
z Make sure it all makes sense
z Rationalise the sequence
z Identify where the author’s familiarity with the subject assumes
the reader knows more than he or she actually does

‰ This is the least effective form of review as there is no


recognised way of measuring it

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Walkthrough

‰ For some technical deliverables or complex processes it is


very useful for the author to walk through the document

‰ Reviewers are asked to comment and query the concepts


and processes that are to be delivered

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Presentations

‰ Used to explain to users your understanding of their


Requirements

‰ To explain how you will test those requirements i.e. your


approach

‰ To gain agreement for the next step

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Prototypes

‰ Allows the non-technician to see what is going to be built

‰ Enables the business to refine their business flows or adjust


sequences on input screens

‰ Is used to demonstrate and display features of the [proposed]


system
z GUI screens, Database designs, Network architecture etc
z Has been around for years in the form of mock ups and screen
layouts
z Much improved with the advent of GUI’s and tools to paint
screens quickly and effectively

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Checklists

‰ Extremely simple and useful method of ensuring that all


elements of a deliverable have been considered

‰ Tends to ensure conformity of content, sequence and style

‰ Can be devised for almost any activity

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Templates

‰ Very useful to standardise the style, sequence & content of


code and documents

‰ Saves time re-inventing

‰ Requires agreement by all who will use them

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Technical (Peer) Review

‰ Your Peers can also perform more formal technical reviews in


larger groups

‰ Often used by workgroups as a self-checking mechanism

‰ It’s format is defined, agreed and practised with documented


processes and outputs

‰ Requires technical expertise

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Inspections

‰ Focus on reviewing code or documents

‰ Formal process based on rules and checklists


z Chaired by Moderator
z Every attendee has a defined role
z Includes metrics and entry & exit criteria
z Training is required

‰ Reviewers must be prepared prior to attendance

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Fagan Inspections

‰ Are very effective - to the point where they have replaced


unit testing

‰ Defects are reported - no debate

‰ Contentious issues and queries are resolved outside the


inspection and the result reported back to the Moderator

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Goals

‰ Validation and verification against specifications and


standards

‰ Achieve consensus before moving on to next stage

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Activities

‰ Reviews need to be planned

‰ They may require overview meetings

‰ Preparation

‰ The review meeting

‰ Follow-up

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Roles & responsibilities

‰ For each type of review (especially the formal ones) roles and
responsibilities must be clearly defined

‰ Training may be required to understand and perform role

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Deliverables

‰ Any project deliverable can (and should) be reviewed

‰ Reviewing these items can lead to


z Product changes
z Source document changes
z Improvements (to both reviews and development)

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Pitfalls

‰ Lack of training
z And therefore understanding of role, responsibilities and
purpose of review

‰ Lack of documentation

‰ Lack of management support

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Review Management

‰ Formal reviews require proper management


z Allow people time to prepare
z Issue guidance notes (if not in Standards)
z Give plenty of notice
z Practice Presentations & Walkthroughs
z Sound out “contentious issues”
z ALWAYS document outcomes & Actions

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

Peer Technical
Walkthrough Inspections
Review Reviews
What Req.'s & Designs Everything Everything Specs & Code

Approach Informal Informal Formal Formal

Why / when Before detail Always* Before detail Before sign off

By Who Author(s) Peer Author(s) Moderator

To Whom IT & Business Author IT & Business Reviewers

Always* : can be done at all points when a document is released.

© SIM Group Ltd., SQS Group AG, 2002


Types of Review

„ Summary

‰ There are various types of reviews


z Companies must decide which one(s) are best for them

‰ In order to gain maximum benefit from them they must be


organised and implemented

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Static Analysis

© SIM Group Ltd., SQS Group AG, 2002


Static Testing

Session Topics

1. Reviews and the Test Process


2. Types of Review
3. Static Analysis

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ In this session we will:

‰ Understand what static analysis is

‰ Look at some of the elements of Static Analysis


z Compilers
z Static analysis tools
z Data flow analysis
z Control flow analysis
z Complexity analysis

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ What is Static Analysis?

‰ Analysis of the code without dynamic execution


z Examine the logic and syntax of the code

‰ Attempting to identify faults in the code

‰ Provides metrics on flow through the SUT and its complexity

‰ A form of automation - usually done with tools

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ What do we hope to find?

‰ Faults in the code


z Unreachable code
z Undeclared variables
z Parameter type mismatches
z Uncalled functions and procedures
z Possible array boundary violations

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ What else will we get?

‰ Information on the code


z Its complexity
z The flow of data through it
z The control flow through it

z Metrics and measurements

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ Compilers

‰ Convert program language code into Machine Code for faster


execution

‰ Will perform some basic Static Analysis


z Usually produces a source code list
z Often produces a memory map
z Often produces a cross-reference of labels and variables
z Will do a syntax check on the code prior to compilation

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ Static Analysis Tools

‰ A Static Analyser is used to parse the Source Code & identify


defects, such as

z Missing branches and links


z Non returnable “performs”
z Variables not initialised
z Standards not met
z Unreachable code

‰ Will also find any faults found by a Compiler

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ Data Flow Analysis

‰ Technique to follow the path that various specific items of


data take through the code of a program

‰ Looking for possible anomalies in the way the code handles


the data items
z Variables used without being first created
z Variables being “used” after they have been “killed”

‰ Consider a CRUD matrix to detail which components affect


which data items

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ Control Flow Analysis

‰ Examines the flow of control through the code

‰ Can identify
z Infinite loops
z Unreachable code
z Cyclomatic complexity and other metrics

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

Control Flow Graph

B1 B2 B3 B4 B8

B9 B5 B6

B7

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ Metrics

‰ Static Analysers generate metrics based on the code

‰ Cyclomatic Complexity
z Identifies the number of decision points in a control flow
z Can be calculated by counting the number of decision points in
a control flow graph and adding 1

‰ Other Complexity Metrics


z Number of Lines of Code
z Number of nested statements

© SIM Group Ltd., SQS Group AG, 2002


Static Analysis

„ Summary

‰ The inspection of program code and logic for errors

‰ The code is analysed without being executed by the tool

‰ Errors are highlighted and metrics are produced

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Test Organisation

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ In this session we will

‰ Gain an appreciation of the many different testing structures


in place

‰ Understand that it is likely that every organisation will have a


unique testing set-up

‰ Look at some of the more common, general set-ups and


appreciate the impact they may have on the success of the
testing carried out

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Testing may be the responsibility of the individual


developers

‰ Developers are only responsible for testing their own work

‰ Goes against the ideal that testing ought to be independent

‰ Tends to confirm the system works rather than detect errors

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Testing may be the responsibility of the developer


team

‰ A “Buddy system”

‰ Each developer is responsible for testing his / her colleagues


work

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Each team has a single tester

‰ Is not involved in the actual development, solely concentrates


on testing

‰ Able to work with the team throughout the development


process

‰ May be too close to the team to be objective

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Dedicated test teams may be in place

‰ These do no development whatsoever

‰ Take a more objective view of the system under test

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Internal Test Consultants

‰ Provide testing advice to project teams

‰ Involvement throughout the lifecycle

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Testing outsourced to specialist agencies

‰ Guarantees independence

‰ Empathy with the Business of the organisation?

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Multi-disciplined teams needed to cover

‰ Production of Testing ‰ Logging Results


Strategies
‰ Executing necessary re-tests
‰ Creation of Test Plans
‰ Automation expertise
‰ Production of Testing Scripts
‰ Technical support
‰ Proving of Testing Scripts z Database admin,
environment admin
‰ Execution of Testing Scripts
‰ User Interface expertise

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Testing teams will typically include

‰ Test analysts to prepare, execute and analyse tests


‰ Test Consultants prepare strategies and plans
‰ Test automation experts
‰ Database administrator or designer
‰ User interface experts
‰ Test environment management
‰ And others..

© SIM Group Ltd., SQS Group AG, 2002


Test Organisation

„ Summary

‰ Testing structures will vary between organisations

‰ Testing teams require a range of skills

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Configuration Management

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ In this session we will

‰ Gain an understanding of Configuration Management

‰ Look at the potential impact of poor Configuration


Management on testing

‰ And understand how introducing Configuration Management


helps address those problems

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ What is Configuration Management?

‰ Ensuring that no-one is able to change a part of the system


without proper procedures being in place

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Closely linked to Version Control

‰ Version Control looks at each component

‰ Holds the latest version of each component

‰ What versions of components works with others in a


configuration

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Without it, testing can be seriously impeded

‰ Developers may not be able to match source to object code

‰ Simultaneous changes may made to the same source


z Although this may be allowed

‰ Unable to identify source code changes made in a particular


version of the software

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Critical to successful testing

‰ It associates programs with specifications, with test plans,


with test data

‰ When a component is successfully tested test results can be


included

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ It enables you to understand what versions of


components work with each other

‰ It allows you to understand the relationship between test


cases, specifications, and components

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ If any configuration component is out of step - Results


are unpredictable

‰ This applies throughout the project lifecycle, and beyond

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

‘…is as old as the pyramids. It allows you at any point in


time to examine the product of your work. (a program,
requirements, tests etc.,) and know: Who, What When,
Where, Why & How’

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Everything intended to last must be subject to


Configuration Management

‰ Anything that isn’t doesn’t exist as part of the product

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Other Configuration Management terms

‰ Configuration identification
z Requires that all Configuration Items (CI’s) and their versions
are known

‰ Configuration control
z Maintenance of the Configuration Items in a library and
maintenance of records on how Configuration Items change
over time

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Other Configuration Management terms

‰ Status accounting
z The function recording and tracking problem reports, change
requests etc

‰ Configuration auditing
z The function to check on the contents of libraries etc for
standards compliance

© SIM Group Ltd., SQS Group AG, 2002


Configuration Management

„ Summary

‰ Configuration Management is a necessary part of any system


development

‰ All assets must be known and controlled

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Test Estimation, Monitoring & Control

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ In this session we will

‰ Understand how we estimate how long we need for testing

‰ Understand how we monitor the progress of testing once it


has started

‰ Understand what steps we take to ensure that the effort


progresses as smoothly as possible

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

Test Estimation

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Test Estimation

‰ The same as estimating for any project

‰ We need to identify the number of tasks (tests) to be


performed, the length of each test, the skills and resources
required and the various dependencies

‰ Testing has a high degree of dependency


z You cannot test something until it has been delivered
z Faults found need to be fixed and re-tested
z The environment must be available whenever a test is to be run

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Factors to Consider

‰ Risk of Failure

‰ Complexity of Code / Solution

‰ Criticality

‰ Coverage

‰ Stability Of SUT

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Testing is a tool to aid Risk Management

‰ Consider the cost to the business of failure of a feature

‰ Test the most important features as soon as possible

‰ Be prepared to repeat these tests

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Consider the complexity of the supporting code

‰ Estimate & plan tests for these ASAP

‰ These are likely to be re-run many times, allow for this in


plans

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Data is critical to testing

‰ You can in certain situations load the data for testing & test in
one pass

‰ These tests will be run first but only once… unless...

‰ The testing environment may need to be re-set before a test


can be re-run

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Use previous experience or best estimates to predict


the reliability of components

‰ Plan to run tests for the unreliable elements sooner rather


than later

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Allow time for re-testing

‰ Time must be included for identifying, investigating & logging


faults

‰ Faults that have been “fixed” must be re-tested

‰ Whenever a change is made to the SUT or the environment a


regression test should be run

‰ How do we build these factors into the estimation?

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

Test Monitoring

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Test Preparation

‰ Estimate number of tests needed

‰ Estimate time to prepare

‰ Refine these figures

‰ Track percentage of tests prepared

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Test Execution

‰ Assess the coverage achieved to date by testing


z Coverage - amount of tests completed, amount of code
exercised by tests, amount of features tested so far

‰ Estimate time required to run the test suite

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Whilst testing you can monitor

‰ The number of tests run

‰ The number of tests passed & failed

‰ The number of defects raised


z These can be categorised by Severity, Priority & Probability

‰ The number of re-tests - that pass & that fail

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ The status of the project should be regularly reported

‰ Any deviations from the schedule raised ASAP

‰ Any critical faults found should be raised immediately

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

Test Control

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Controlling measures

‰ Assign extra resource

‰ Re-allocate resource

‰ Adjust the test schedule

‰ Arrange for extra test environments

‰ Refine the completion criteria

© SIM Group Ltd., SQS Group AG, 2002


Test Estimation, Monitoring & Control

„ Summary

‰ Multiple factors must be considered when estimating the


length of time we need to perform testing

‰ Once testing has started it is necessary to monitor the


situation as it progresses

‰ Careful control must be kept to ensure project success

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Incident Management

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ In this session we shall

‰ Understand what an “incident” is

‰ Understand the impact they have on the testing process

‰ Understand how they are logged, and tracked

‰ Understand why & how they should be analysed to help


ensure they don’t happen again

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ An ‘incident’ is when testing cannot progress as


planned

‰ The application does not function as anticipated

‰ Components are missing etc

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ What is an incident?

‰ When actual results differ from expected results

‰ They may be raised against


z Documentation
z The code
z The System Under Test
z The Test Environment
z Or the tests themselves

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ The incident needs to be viewed in terms of its impact


on testing

‰ As testing progresses through its lifecycle priorities can


change

‰ The progress of the testing activity can be monitored by


analysing incidents

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ Top Priority must be given to any issues that prevent


testing from progressing

‰ At the second level we need to assign those that can be


worked around

‰ In the lower categories we log what are minor issues &


questions

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ As project deadlines get nearer what is considered


high priority may well change

‰ The emphasis will move towards the product and what will
compromise its success

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ When logging incidents testers should include

‰ Test ID, System ID, Testers ID

‰ Expected and Actual results

‰ Environment in use

‰ Severity or priority of the incident

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ Service levels should be set up for each category of


incident

‰ Progress monitored accordingly

‰ Involve Development, Testers and Users

‰ Issue regular status reports

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ STATUS should also be tracked

‰ This ensures that we are able to swiftly establish the situation


with an incident

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ A typical status flow is

‰ When first raised an incident is OPEN


‰ It then passes to development to fix - WITH DEVELOPMENT
‰ Development may question the incident - PENDING
‰ Or fix the problem and return it for - RETEST
‰ Should the retest prove successful the defect is CLOSED

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ Analysing incidents

‰ Incidents may be analysed to help monitor the test process

‰ And to aid in improvement of the test process

© SIM Group Ltd., SQS Group AG, 2002


Incident Management

„ Summary

‰ An incident is “any significant, unplanned event that occurs


during testing that requires subsequent investigation and / or
correction”

‰ Incidents should be recorded and tracked

‰ Analysis of incidents will enable us to see where problems


arose and to aid in test process improvement

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Standards for Testing

© SIM Group Ltd., SQS Group AG, 2002


Standards for Testing

„ In this session we will

‰ Understand the various standards that may influence testing

© SIM Group Ltd., SQS Group AG, 2002


Standards for Testing

„ There are three types of standards that affect testing

‰ Quality Assurance standards

‰ Industry specific standards

‰ Testing Standards

© SIM Group Ltd., SQS Group AG, 2002


Standards for Testing

„ Quality Assurance standards

‰ Only specify that testing should be done

z e.g. ISO 9000-3:1991

Quality management and quality assurance standards-Part 3:


Guidelines for the application of ISO 9001 to the development,
supply and maintenance of software

© SIM Group Ltd., SQS Group AG, 2002


Standards for Testing

„ Industry specific standards

‰ Specify what level of testing to perform

z e.g. railway, medical, insurance

ASTM F153-95 Standard Test Method for Determining the Yield of


Wide Inked Computer Ribbons

© SIM Group Ltd., SQS Group AG, 2002


Standards for Testing

„ Testing standards

‰ Specify how to perform testing

z e.g. BS7925-1 Software Testing vocabulary


BS7925-2 Software component testing
IEEE 829 Standard for Test Documentation

© SIM Group Ltd., SQS Group AG, 2002


Standards for Testing

„ Summary

‰ There are a range of standards that can influence testing

‰ These come from different sources and affect different areas


of testing

‰ Additionally, various industries have legal requirements that


will influence testing

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Types of CAST Tool

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ In this session we will

‰ Understand that help is at hand in the form of CAST Tools


z CAST - Computer Aided Software Testing

‰ Look at the range of tools available in the market today

‰ Identify the areas in which tools can best help the testing
process

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Why do we need / use them?

‰ To automate the most repetitious and time consuming tasks

‰ Tools are able to do some things that people cannot easily do

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Requirements testing „ Drivers & Simulators


„ Static Analysis „ Performance
„ Test Design „ Dynamic Analysis
„ Data Preparation „ Debugging
„ Character Based Test „ Comparison
Running „ Test Management
„ GUI Test Running „ Coverage Measurement
„ Test Harnesses

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Requirements Testing tools

‰ There are tools available to help verify and validate


requirements

‰ They use requirements models

‰ Can check consistency and animations

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Static Analysis tools

‰ There are tools on the market that can help with analysing
code

‰ These tools analyse the quality and complexity of the code


and paths through the code

‰ Many use McCabe, Halstead or McClure metrics

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Test Design tools

‰ Help with designing of test cases

‰ These generate test cases from a specification


z Normally held in a CASE tool repository
z Or from formally specified requirements held in the tool itself

‰ Some tools generate test cases from analysis of the code

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Data Preparation tools

‰ Help with data preparation

‰ These enable data to be selected from existing databases or


created, generated, manipulated and edited for use in tests

‰ May create data in a simple database or a complex


mainframe database

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Character Based Testing tools

‰ Used with character based applications


z Terminals on Mainframes, Unix and mid-range systems (System
36/38)
z Early PC (DOS) systems

‰ The tools simulate user entered keystrokes and capture


screen responses for comparison
z Normally used to automate regression and functional system
testing

‰ Many tools available to cover this

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ GUI Test Running tools

‰ Testing for GUI based applications

‰ Testing software running in graphical operating systems i.e.


Windows
z Most of our work in this area
z Mostly for functional and regression tests

‰ Tools must be able to identify and interact with the objects on


screen

‰ Tests scripted in a Test Script Language

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Web Testing tools

‰ Testing web based applications

‰ Increasingly tools are becoming available to help test E-


Commerce, web-based applications
z Existing tools that have “evolved” to include web testing
z Dedicated web testing tools

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool Test

„ Harnesses & Drivers

‰ These are key to building up integration tests

‰ Used to execute the software under test


z i.e. for systems that do not have a user interface

‰ These are small modules that drive data into the area under
test

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Simulators

‰ Simulate events or systems

‰ There will be times when it is necessary to test a system that


interacts with an event or system that does not physically
exist

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Performance Testing tools

‰ Measure what a system operates like in normal


circumstances
z Drive the SUT & monitor the response time for specified
transactions

‰ Load generation
z How the system performs when accessed by many users
z The tool simulates a number of users accessing the SUT

‰ The tools can simulate multiple sessions from one device


z LoadRunner, QALoad etc

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Dynamic Analysis tools

‰ Provide run time information on the state of the executing


software

‰ Typically used to monitor system resources

‰ Find errors that can be difficult to find “statically”

‰ i.e. use and de-allocation of system memory and flag memory


leaks

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Debugging tools

‰ Used mainly by programmers to reproduce bugs and


investigate the state of programs

‰ Debuggers enable programmers to execute programs line by


line, to halt the program at any statement and to set and
examine program variables

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Comparison tools

‰ Comparing two elements is a key activity in testing

‰ ‘Eyeballing’ information is labour intensive and prone to error

‰ Tools are available on many platforms to help here

‰ Comparison tools detect differences between files


z Typically actual and expected results files

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Test Management tools

‰ Enable testers to manage their tests

‰ Tests are created in the tool, linked together to form workflow,


tests executed and results collated

‰ Some of the better known tools


z i.e. TestDirector, SQA Manager, QADirector

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Coverage Measurement tools

‰ Report amount of code exercised by test executions

‰ The code is instrumented before compilation

‰ As the program is executed the instrument code records the


coverage data

‰ Reports generated can show the amount of code exercised


during tests
z The most common coverage measures are statement and
branch coverage

© SIM Group Ltd., SQS Group AG, 2002


Types of CAST Tool

„ Summary

‰ CAST Tools can help automate areas of the testing process

‰ There is a wide range of functions that can be automated

© SIM Group Ltd., SQS Group AG, 2002


ISEB Foundation Certificate in Software Testing

Tool Selection & Implementation

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ In the session we will

‰ Examine the processes involved in identifying where tools


can help the testing process

‰ Learn how to approach the selection and implementation of


those tools

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ As indicated tools can help many aspects of testing

‰ How do you select where you want to start introducing CAST


support?

‰ The selection and implementation of a tool is a major task


and should be regarded as a project in its own right

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ If testing is currently badly organised tools may not be


the immediate answer

‰ Automating chaos simply speeds the delivery of chaos

‰ The benefits of tools usually depend on a systematic and


disciplined test process being in place

‰ Address the test processes and introduce disciplines first

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Examine the test process in detail

‰ Identify the weaknesses, and the cause of those weaknesses

‰ Find tools that fit in with your processes and address the
weak areas
z This may be more important than selecting the tool with the
most number of features

‰ Prioritise the areas of greatest importance

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Pick a number of tools to trial in a certain area

‰ Define clear requirements & success criteria

‰ Understand each tools’ capabilities & weaknesses

‰ Run trials or small pilot exercises to evaluate the tools

‰ Review and select tool

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Examine the Test Environment(s)

‰ One of the most crucial areas for tool implementation

‰ Shared environments will not accept automation unless the


SUT allows the data to be entered fresh every time

‰ Automation may also fill an environment up and make it run


out of space as it allows a large amount of data to be entered
in a short space of time

‰ Test facilities may be (will be!!!) less than Production facilities


and use of the tools with the SUT can cause systems to run
out of memory

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Funding must be secured

‰ Software testing tools are not low cost items and there is
usually a large amount of man hours required as well

‰ It is necessary to budget for both the tool and the resources


required to implement

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Evaluate the results of the trial

‰ Select the best candidate - pilot project

‰ Plan the implementation & use of that tool

‰ Implement & use

‰ Broadcast the benefits

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Plan the roll-out of the tool

‰ Gain commitment from the tools users

‰ Ensure new projects are aware of the cost of introducing a


tool

© SIM Group Ltd., SQS Group AG, 2002


Tool Selection & Implementation

„ Summary

‰ Tools can improve the organisation and efficiency of a test


process

‰ Before tools can be implemented a sound testing process


must be in place

‰ Tool selection and implementation must be carefully


considered and managed

© SIM Group Ltd., SQS Group AG, 2002