6.

BASIC PRINCIPLES FOR

TEST PLANNING AND
CONTROL
 Table of Contents (1)
 Test Planning
 Test Prioritization
 Entry Criteria
 Exit Criteria
 Test Estimation
 Test Strategy, Test Approach

2
 Table of Contents (2)
 Risk and Testing – Main Concepts
 Product Risks
 Project Risks
 Risk-Based Testing
Risk Management
○ Risk Identification
○ Risk Analysis (Assessment)
○ Risk Control

3
Test Planning Activities
 The Purpose and Substance of Test Plans

 Why Do We Need Test Plans and How Can We Use

Them?
 Test Planning Activities (1)
 Defining the overall approach to and strategy for testing
 Deciding about the test environment
 Definition of the test levels
How are they going to cooperate
Integrating and coordinating the testing activities with other
project activities

6
 Test Planning Activities (2)
 Deciding how to evaluate the test results
 Selecting metrics
For monitoring and controlling test work
Defining test exit criteria
 Test documentation
How much documentation shall be prepared
What templates will be used

7
 Test Planning Activities (3)
 Writing the test plan
Deciding on what, who, when, and how much testing
 Estimating test effort and test costs
(Re)estimating and (re)planning the testing tasks

8
 Why Do We Write a Test Plan?
 Writing a test plan guides our thinking
If we can explain something in words, we understand it
○ Otherwise there is a good chance we don't
Forces us to confront the challenges that await us
○ Focus our thinking on important topics

9
 Why Do We Write a Test Plan? (2)
 Serves as a vehicle for communicating with other
members of the project team
Testers, peers, managers and other stakeholders
Using a test plan draft allows team members to leave their
notes
○ Becomes a record of previous discussions

10
 Why Do We Write a Test Plan? (3)
 Manage change
Written test plans give us a baseline
○ Against it we can measure revisions and changes made
Test plans should be updated at major milestones
○ Helps to keep testing aligned with project needs

11
 Test Plan Vs. Test Design
 Test Plan
Defines overall testing objectives and approach
Provides accurate test estimation
 Test Design
Defines what will be tested
Describes expected results
 Reasons for Planning Tests
 Repeatability
All testers should be able to execute the tests
Assures all critical elements are tested correctly
Parts can be executed
 Controllability
Knowledge of test data requirements, expected results, what
to run
 Coverage
Ensures adequate coverage
 Test Documentation Hierarchy
Test Plan
Direction for overall testing activity

Test Design Specification

Refines Approach, identifies features to be covered
Test Case Specification
Specific input/intended output values

Test Procedures Specification

Test execution steps

14
 Test Plan Templates
 Test plans can be made using templates
E.g., IEEE 829 test plan template

15
 Elements of a Test Plan
 Test Scope
Defines what will be tested
 Test Objectives
Description of expected (measurable) test result, priority
 Assumptions
Include skill level of testers, budget, starting state of
application, tools & equipment availability, etc.
 Risk Analysis
Things that could impact testing ability
 Elements of a Test Plan (2)
 Test Design
Identifies tests to run, stages to test, outlines sequence and
timing
 Test Schedule & Resources
Major test activities, sequence of tests, estimates,
dependence on other activities, people, tools, facilities
 Test Data Management
Methods for preparing test data, backup/rollback procedures,
data requirements/sources, any data conditioning/conversion,
data security
 Elements of a Test Plan (3)
 Test Environment
Version Control, HW/SW configurations, defect tracking tool,
Environment for each kind of testing
 Communication Approach
Meetings, processes, tools, techniques, contact lists
 Test Tools
Automation, performance, verification, defect tracking, test
planning, etc.
 Testing Concerns
 Not Enough Training
 Us Versus Them Mentality
 Lack of Test Tools
 Lack of Management Understanding/Support
 Lack of Customer and User Involvement
 Not Enough Time
 Over Reliance on Independent Testers
 Rapid Change
 Lose-Lose Situation
 Having to Say “No”

19
 Who? What? When? How?
 Planning resources is an essential part of test planning
Often this is related to hard decisions
Consensus across the team is needed

20
 What Else Is Important?
 A good test plan provides some more answers:
How precisely should testing be documented
What metrics should be used
Entry criteria
Exit criteria

21
Test Prioritization
 Why Should We Prioritize Tests?
 Time and budget are never enough
Not sufficient to execute all planned test cases
 Still - as many critical faults should be found as possible
 The prioritization rule:
A premature end of testing should still assure the best possible test
result at that actual point in time

23
 Prioritization Criteria
 Criteria for prioritization of test cases may be:
Usage frequency of a function / probability of failure
Risk of failure
Visibility of a failure
Priority of the requirements
Customer priorities
Code complexity
Project risk

24
 Entry Criteria
 When to start testing?
 Test Entry Criteria
 Test entry criteria define when to start testing
E.g., at the beginning of a test level or when a set of tests is
ready for execution
 Entry criteria may cover the following:
Test environment availability and readiness
 Test tool readiness in the test environment
Testable code availability
Test data availability

26
 Exit Criteria
 When to stop testing
 Test Exit Criteria
 What is test exit criteria?
A definition of when testing can be stopped (totally or within a
test level)

28
 Typical Test Exit Criteria
 Typically exit criteria may cover the following:
Thoroughness of measures
○ E.g., coverage of code, functionality or risk
Estimates of defect density or reliability measures
Cost

29
 Typical Test Exit Criteria (2)
 Typically exit criteria may cover the following:
Residual risks
○ E.g., defects not fixed
○ Lack of test coverage in certain areas
Schedules
○ E.g., time to market

30
Test Estimation
 Test Estimation
 What do we estimate?
What testing will involve?
What it will cost?

32
 Work-breakdown
 Test estimation could start with designing a work-
breakdown structure
Identifying the stages, activities and tasks for testing

33
 Phases of a Test Project
 A test project could be broken down into phases
Planning and control
Analysis and design
Implementation and execution
Evaluating exit criteria and reporting
Test closure

34
 Activities
 Within each phase we identify activities and tasks
within each activity
 This can be performed in two ways:
Working forward
○ "Now, what comes next?"
Working backward
○ "What activities and tasks are required in each stage to carry out
this testing?"

35
 Estimation Approaches
 Testing effort is usually estimated using two
approaches:
Metrics-based approach
○ Using metrics of former or similar projects
○ Using typical values
Expert-based approach
○ Consulting with experts and with people who will actually perform
the testing

36
 Co-ordination With the Management
 Even the best estimate must be negotiated with management
Different sides on the project can have different priorities
Effective negotiations are focused on finding the best balance
○ Between quality, schedule, budget and features

37
 Factors Affecting Testing Effort
 The testing effort may depend on a number of factors:
Complexity and size of the product
Life-cycle model used
Tools available
Product documentation available
How detailed test documentation needs to be done
Time pressure
People factors
Etc.

38
Test Strategy and
Test Approach
 Test Strategy
 What is a test strategy?
Defines the project's testing objectives and the means to
achieve them
Determines testing effort and costs
One of the key-responsibilities of the test manager

40
 What is the Point of Test Strategies?
 The main goal of the test strategy is to choose the best test
approach
Optimizing the relation between costs of testing and costs of defects

41
 Test Approach
 What is a test approach?
Implementation of the test strategy for a specific project
The testing strategy usually involves a combination of test
approaches

42
 Preventative vs. Reactive Approach
 Preventive approaches
Testers are involved from the beginning:
Test planning and design start as early as possible
 Reactive approaches
Testers are involved (too) late and a preventive approach cannot be
chosen
Test planning and design starts after the software or system has already
been produced

43
 Analytical vs. Heuristic Approach
 Analytical approach
Based on data and (mathematical) analysis of collected data
 Heuristic approach
Based on experience of experts and/or on rules of thumb
○ When no data are available
○ When mathematical modeling is too complicated
○ When know-how is missing

44
 Other Approaches
 In practice, approaches used are between the
extremes of analytical and heuristic – e.g.,:
Model-based testing
Risk-based testing
Reuse-oriented approaches
Checklist-based (methodical) approaches
Expert-oriented approaches

45
How do we select which strategies to pick or blend for
the best chance? (1)
 Risks
Risk management is very important during testing, so consider
the risks and the level of risk.
 Skills
Consider which skills your testers possess and lack because
strategies must not only be chosen, they must also be
executed
 Objectives:
Testing must satisfy the needs and requirements of
stakeholders to be successful.
How do we select which strategies to pick or blend for
the best chance? (2)
 Regulations
Sometimes you must satisfy not only stakeholders, but also
regulators
 Product
Some products like, weapons systems and contract-
development software tend to have well-specified requirements
 Business
Business considerations and business continuity are often
important.
Risk and Testing
 Risk
 Risk
The possibility of a negative or undesirable outcome or event
Any problem that may occur would decrease perceptions of
product quality or project success

49
 Types of Risk
 Two main types of risk are concerned
Product (quality) risks
○ The primary effect of a potential problem is on the product
quality
Project (planning) risks
○ The primary effect is on the project success
○ Factors relating to the way the work is carried out

50
 Levels of Risk
 Not all risks are equal in importance
 Factors for classifying the level of risk:
Likelihood of the problem occurring
○ Arises from technical considerations
○ E.g. programming languages used, bandwidth of connections,
etc.
Impact of the problem in case it occurs
○ Arises from business considerations
○ E.g. financial loss, number of users affected, etc.

51
Levels of Risk - Chart
RISK

Impact Likelihood
(Probability
(Probability of
of
(damage)
(damage) failure)
failure)

Use Lack of
frequency quality

52
Product Risks
 Product Risk
 What is a product risk?
The possibility that the system or software might fail to satisfy
some reasonable customer, user, or stakeholder expectation
Also referred to as "quality" risk

54
 Typical Product Risks
 What does "unsatisfactory software" mean?
Omitted key functionality
Unreliable and frequently fail to behave normally
Might cause financial or other damage to users
Poor software characteristics
○ Low security, usability, maintainability or performance
Poor data integrity and quality

55
Project Risks
 Typical Project Risks
 Organizational factors:
Skill, training and staff shortages
Complexity of the project team / organization
Inadequate expectations or improper attitude toward testing
○ E.g., not appreciating the value of testing

57
 Typical Project Risks (2)
 Technical issues:
Ambiguous, conflicting or non-prioritized requirements
Excessively large number of requirements
High system complexity
Quality problems with the design, the code or the tests
Insufficient or unrealistic test environments

58
 Typical Project Risks (3)
 Supplier issues:
Failure of a third party
Contractual issues

59
Risk-Based Testing
 Risk-Based Testing
 What is Risk-based testing?
An approach to testing that aims to:
○ Reduce the level of product risks
○ Inform stakeholders on their status
Starts in the initial stages of a project
Involves the identification of product risks and their use in
guiding the test process

61
Risk Management
 Primary Activities
 Risk management includes three primary activities:
Risk identification
Risk analysis
○ Assessing the level of risk
Risk control
○ Mitigation
○ Contingency
○ Transference
○ Acceptance

63
Risk Identification
 Risk Identification Techniques
 Product and quality risks can be identified
Expert interviews
Project retrospectives
Risk workshops and brainstorming
Checklists
Calling on past experience

65
 Include Stakeholders
 Include representatives of all (possible) stakeholders in
risk identification
The broadest range of stakeholders will yield the most
complete, accurate, and precise risk identification

66
 Downstream vs. Upstream
 Risk identification techniques can look in two
directions:
"Downstream"
○ Identify potential effects of the risk item if it becomes an actual
negative outcome
"Upstream"
○ Identify the source of the risk

67
 Risk Analysis
or Risk Assessment
 Risk Assessment
 Risk analysis (assessment) involves the study of the
identified risks
Categorize each risk item appropriately
○ Important for complex projects
Assign each risk item an appropriate level of risk
○ Involves likelihood and impact as key factors

69
 Technical Factors for Assessing Likelihood
 Complexity of technology and teams
 Personnel and training issues
 Supplier and vendor contractual problems
 Geographical distribution of the development organization
E.g., out-sourcing

70
 Technical Factors for Assessing Likelihood (2)
 Legacy (established) versus new designs and technologies
 The quality (or lack of quality) in the tools and technology used
 Bad managerial or technical leadership
 Time, resource, and management pressure
Especially when financial penalties apply

71
 Technical Factors for Assessing Likelihood (3)
 Lack of earlier testing and quality assurance tasks in the lifecycle
 High rates of requirements, design, and code changes in the
project
 High defect rates
 Complex interfacing and integration issues

72
 Business Factors for Assessing Impact
 Potential damage to image
 Loss of customers and business
 Potential financial, ecological, or social losses or liability

73
 Business Factors for Assessing Impact (2)
 Civil or criminal legal sanctions
 Loss of licenses, permits, etc.
 The lack of reasonable workarounds
 The visibility of failure and the associated negative publicity

74
 How Do We Determine the Level of Risk
 Quantitatively
Using numerical ratings for both:
○ Likelihood (usually percentage)
○ Impact (often a monetary quantity)
○ Both can be calculated to a common risk index
 Qualitatively
E.g., very high, high, medium, low, very low

75
Risk Control
 Risk Control
 Risk control has four main options:
Mitigation
○ Taking preventive measures to reduce the likelihood and/or the
impact of a risk
Contingency
○ Where we have a plan or perhaps multiple plans to reduce the
impact if a risk should it occur

77
 Risk Control (2)
 Risk control has four main options:
Transference
○ Getting another party to accept the consequences of a risk
should it occur
Accepting (ignoring) the risk
○ A final option

78
 Techniques for Risk Control
 Various techniques can be used for risk control:
Choosing an appropriate test design technique
Reviews and inspection
Reviews of test design

79
 Techniques for Risk Control (2)
 Various techniques can be used for risk control:
Setting appropriate levels of independence
○ For the various levels of testing
Using the most experienced person on test tasks
Using strategies for confirmation testing (retesting) and
regression testing

80