ISYS6264 – Testing and System Implementation

Test Management – Planning

 Learning Outcomes

• LO 2: Design the testing management plan for a software

 References

• Black, Rex. (2009). Managing the testing process : practical

tools and techniques for managing hardware and software
testing. 03. Wiley. Indianapolis. ISBN: 9780470404157.

• Burnstein, Ilene. (2003). Practical Software Testing. Springer.

New York. ISBN: 0-387-95131-8

• Homès, Bernard. (2012).Fundamentals of Software Testing. ISTE

– Wiley. London – Hoboken. ISBN: 978-1-84821-324-1

• Black, Rex & Mitchell, Jamie. (2011). Advanced Software Testing.

3. Rocky Nook. Santa Barbara, USA. ISBN: 978-1-933952-39-0
 Sub Topics

• Test Planning and Estimation

• Test Preparation Activities
• Test Planning Activities
• Estimation Methods
• Test Plan
• Test Plan Template
• Entry and Exit Criteria for Test Activities
• Evaluating Exit Criteria
Test Planning and Estimation
 Planning and Estimation

• Test managers (project leaders, test leaders, test managers, etc.)

must estimate the test workload, and organize and plan the
activities over time. This means that the company’s test policy
needs to be put into practice, to identify the necessary resources
and justify their use or acquisition. This delicate and complex task
is applied to a constantly evolving target: the software is being
developed and the quality of the product and the effective end
date of the project is unknown.
 Planning and Estimation
(cont.)
• The following factors influence the workload:
– process-related factors: constantly executing tests, changing management,
process maturity, development and test processes, previous test phases,
planned and actual levels of defects and corrections;
– hardware factors: tools, system tests, test environment, project
documentation, similarity with other projects;
– human factors: tester abilities and expectations, support from development
teams, relationships between teams;
– other delaying factors: complexity of the software, large number of
stakeholders, too many new features, geographic distribution, delicate
logistics, fragile test data;
– the correct understanding of the estimation techniques and other aspects that
may influence results.
 Planning and Estimation
Example
• Let us imagine that we have a 6 week period to test the software. If we distribute
the testing activities uniformly over this period, 33% of the software will be tested
in 2 weeks. If, at the end of each week, we receive a corrected version of the
application, we will be forced to execute re-testing (confirmation tests) and non-
regression tests on this version of the software. This will take time (re-initializing
environments, impact analysis, re-execution of selected tests), during which we
will not be able to test new functionalities as initially planned. We will end up
accumulating delays that will impact future activities. This scenario will repeat for
each new delivery during these 6 weeks, generating an increased workload for the
test team, delays in planning, and a financial impact and stakeholder
dissatisfaction (hierarchy, development team, customers, and users) towards the
test team.

Source: Homès (2012, pg. 216)

 Planning and Estimation
Example (cont.)
• If we break down our 6 week period in three 2 week periods, we could process
60% of the software during the first 2 week period (week 1 and week 2), then we
could test an additional 30% of the functionalities during week 4 and the last 10%
during week 6. During the other weeks we will execute non-regression testing and
confirmation tests (re-test).

Source: Homès (2012, pg. 217)

 Planning Document

• Test plans for software projects are very complex and detailed
documents. The planner usually includes the following essential
high-level items.
– Overall test objectives. As testers, why are we testing, what is to be achieved
by the tests, and what are the risks associated with testing this product?
– What to test (scope of the tests). What items, features, procedures, functions,
objects, clusters, and subsystems will be tested?
– Who will test. Who are the personnel responsible for the tests?
– How to test. What strategies, methods, hardware, software tools, and
techniques are going to be applied? What test documents and deliverable
should be produced?
– When to test. What are the schedules for tests? What items need to be
available?
– When to stop testing. It is not economically feasible or practical to plan to test
until all defects have been revealed. This is a goal that testers can never be
sure they have reached. Because of budgets, scheduling, and customer
deadlines, specific conditions must be outlined in the test plan that allow
testers/managers to decide when testing is considered to be complete.
Test Preparation Activities
 Test Preparation Activities

• Identifying the testing scope, applicable constraints and test

objectives;

• Definition of the overall test approach, of the test levels, and their
entry and exit criteria;

• Identifying and prioritizing risks, anticipating tasks to mitigate

impacts and to limit the possible occurrence of these risks;

• Identifying the activities with which tests will interact, such as

development, acquisition, delivery, exploitation, and maintenance
activities;
 Test Preparation Activities
(cont.)
• Defining and ordering test basis analysis and design activities, by
making sure that adequate traceability is maintained from (and to)
the requirements;

• Defining and sequencing test implementation activities, selecting

test data (including expected results), sequencing test execution
and verification activities;

• Defining the level of detail, the volume, structure, and templates

for test documentation, both in terms of progress reports, periodic
status reports, and final deliverables.
 Another Tasks
to be Anticipated
• Deciding what will be tested (which quality characteristics), which
roles will execute which activities, when and how these activities
will be executed, how to evaluate the test results and when to
stop the tests (definition of the test exit criteria);

• Assigning resources to the different tasks that were defined and

taking into account the specialties of each individual, possible
issues, holidays, as well as the impact of disease on the limited
availability of resources;

• Selecting the means required to track and control test preparation

and execution, defect removal and risk limitation activities,
through the definition of metrics and other appropriate means;

• Defining the level of detail for test cases and test procedures, so
as to provide enough detail to allow the creation and execution of
reusable tests. This greatly depends on the knowledge of the
testers in charge of the actual test execution.
Test Planning Activities
 Test Planning Activities

Generation of the master test plan;

Workload estimation review;

Review of the technical support required;

Interface with the organizational and supporting processes;

Identification of test process improvement opportunities.

 Another Tasks
to be Anticipated
• Identification of the test scope;

• Risk analysis

• Integration and coordination of the test activities with development

activities: acquisition, delivery, development, operations, and
maintenance;

• Organization and scheduling of the analysis and test design activities,

creation of test cases and test data, test case execution, and evaluation
activities;

• Volume, level of detail, structure, and templates for test documentation;

• Identification of test control and monitoring activities (test metrics), to

prepare and execute tests, eliminate defects, and solve risk-related
issues;

• Volume, level of detail, and structure of the test procedures to allow

reproducible test preparation and test execution.
Estimation Methods
 Varieties
in Estimation Methods

Estimation in
Agile World

Experience-
based

Metrics-based
 Metrics-based Estimation

• There are many estimation methods based on metrics. The most

frequently mentioned are those based on lines of code, on
function points, on formulas such as COCOMOII and those based
on test points. This estimation method computes, from a number
of characteristics, a workload and a development time, as well as
a size in lines of code. From this estimation of the development
effort, it is possible to extrapolate a test workload. The
parameters used to compute the test workload will have to take
into account the different factors we have seen previously.

• Estimation methods based on software size in lines of code or on

development workload, often do not take into account that the
same functionality can be designed with more – or less – lines of
code, depending on the programming language used. Some
languages are wordier than others and some developers are
more concise than others.
 Experience-based Estimation

• Methods based on experience are not based on a series of

metrics, but on the experience of experts or those who will
execute these tasks.

• The first experience-based method requires that each individual

who will execute the planned tasks estimates the workload and
these individual tasks estimations are summed. The drawbacks of
this proposed solution is that individuals only have an
approximate idea of the workload (based on a number of
unconfirmed hypotheses), and that there is frequently a safety
factor that is added as a rounding factor (in hours or days).
 Experience-based Estimation
(cont.)
• The second experience-based method requires an experienced
tester to estimate the workload. The drawback of this method will
depend on the person who will effectively execute the work (are
the knowledge and expertise levels similar to that envisaged by
the expert tester during the initial estimation?). This drawback can
be bypassed by the three-point estimation with the aim of
identifying the hypotheses. It requires three estimations: an
optimistic one, a pessimistic one, and a third one that must be the
most likely.
 Experience-based Estimation
(cont.)
• A third method is based on the similarity between the project to
be estimated and previous projects that have been carried out.
The hypothesis here is that the context is similar, which is rarely
the case. If we test a software that we have already tested, our
understanding of the software will be better than before and
therefore efficiency should be increased. However, it could be
reduced if the test team is different or if the test objectives are
modified. If we test a project that is similar to a project we have
already executed, we could be tempted to estimate the same
workload, but are we in the same context, with the same
programming language, the same defect rate, and similar
objectives?
 Estimation in the Agile World

• In the agile world, there are similar estimation methods,

depending on how the teams are organized and on the
requirement definition.

• This is no longer an experience-based estimation, but an

estimation based on “story points”. The principle of “story point”
estimation consists of estimating each “user story”, the complexity
of its development and test, whenever the story is received.
Workload estimation for a story is influenced by the previous
workload estimations in the project. Any change of context from
the previous stories (such as increased or decreased complexity,
required refactoring, etc.) will impact the estimation.
 Estimation in the Agile World
(cont.)
• Scrum proposes estimation methods similar to the second
experience-based methods, with a periodical re-estimation of the
workloads based on the time spent carrying out the previous
tasks and the tasks that remain. This estimation is executed by all
members of the team and when a consensus is reached, this
value is taken as a reference. To make this estimation quicker or
more “agile”, estimation can be done by using cards, as seen
previously (poker estimations).
 Test Cost Estimation

Source: Burnstein (2003, pg. 211)

Test Plan
 Why Write Test Plans?

• Writing a test plan gives you a chance to collect your thoughts,

your ideas, and your memories. Undoubtedly you’ve learned a
great deal throughout the course of your career. Writing a
thorough test plan gives you a chance to crystallize that
knowledge into a concrete way of tackling the tasks ahead.

• Test plan will be used to communicate with test team,

development colleagues, and managers. In some organizations,
the test plan encompasses the entirety of the test effort, all the
way down to defining all the individual test cases — often called
the test set or the test suites— that the team will run.

• The difference between a test plan and a test suite is a matter of

strategy versus tactics. The test plan describes how we intend to
implement the test strategy on a particular project. The test cases
in the test suite provide the specific steps the testers will carry out
during test execution, which are the tactics necessary to
implement the plan.
 Test Plan Components

Source: Burnstein (2003, pg. 201)

Test Plan Template
 Test Plan Template

Source: Black (2009, pg. 52)

 Overview

• The Overview section of a test plan allows you to introduce

readers of the plan to your test project, including what we plan to
test and the general test approach. You’ve found that oftentimes
managers one or two levels above you don’t have a good idea of
what testing covers or how it works. In the overview, I present a
concise explanation of your goals, methodologies, and objectives.
You keep this section brief. You find it useful to include simple
pictures or charts. You might want to illustrate concepts such as
the architecture of the system under test, the decomposition or
segmentation of the system for component or integration testing,
or how this test effort fits into other test efforts that might precede,
run concurrently, or follow.
 Bounds

• In this section, you set boundaries for the test plan by discussing
what I will and will not test, by defining important terms and
acronyms related to the testing you plan to perform, and by
determining where and in what context the test efforts associated
with this test subproject will take place.
– Scope. Webster’s Dictionary defines scope, in the context of a project or an
operation, as the ‘‘extent of treatment, activity, or influence; [the] range of
operation’’.
– Definitions. Therefore, you include a table of definitions in my test plans. Such
a table can help to clarify terminology for those who are not experienced in
the field of testing, and can also help to ensure that everyone on the test
team is operating from the same set of definitions.
– Setting. This section of the test plan describes where I intend to perform the
testing and the way those organizations doing the testing relate to the rest of
the organization. The description might be as simple as ‘‘our test lab’’.
 Quality Risks

• You can summarize the quality risk documents you’ve prepared,

or simply reference them in the test plan. If you suspect that many
of your readers won’t look at the referenced documents, it makes
sense to summarize the quality risks here, given that your
purpose is to communicate as well as to plan.
 Proposed Schedule
of Milestones

Source: Black (2009, pg. 57)

 Transitions

• For each test phase, the system under test must satisfy a minimal
set of qualifications before the test organization can run tests
effectively and efficiently.
– For example, it makes little sense to start extensive user-scenario testing of
SpeedyWriter if the application cannot open or save a file or display text on
the screen.
– Likewise, the DataRocket server can’t undergo environmental testing—
especially thermal testing— if you don’t have even a prototype case.

• This section of the test plan should specify the criteria essential
for beginning and completing various test phases (and for
continuing an effective and efficient test process). I usually refer
to these as entry, continuation, and exit criteria, respectively, but
some test professionals use the terms entry,
suspension/resumption, and exit criteria or entry, stopping, and
exit criteria.
 Entry, Continuation, and Exit
Criteria

• what must happen to allow a system to

Entry Criteria move into a particular test phase.

Continuation • conditions and situations that must prevail

in the testing process to allow testing to
Criteria continue effectively and efficiently.

• issue of how to determine when the

Exit Criteria project has completed testing.
 Test Development

• In this section you’ll describe how my test team will create each of
various test objects, such as test cases, test tools, test
procedures, test suites, automated test scripts, and so forth.
 Test Configurations and
Environments
• This section of the test plan is where you document which
hardware, software, networks, and lab space you will use to
perform the testing. For these various test systems, you’ll
describe whatever important configuration details bear
mentioning, as well. For a PC application or utility, this task can
be as simple as listing the half-dozen or so test PCs, the two or
three test networks (assuming that networking is even an issue),
and the printers, external drives, and other accessories you might
require from time to time.
 Test Execution

• This portion of the test plan addresses important factors affecting

test execution. For example, in order to run tests, you often need
to receive items from the outside world, primarily resources and
systems to test. In the course of running tests, you will gather
data that you must track, analyze, and report to your team, your
peers, and your managers.
– Resources. In this section I identify the key participants in the test effort and
the role they’ll play in testing, along with any other resources not identified
elsewhere in the plan.
– Test Case and Bug Tracking. This section deals with the systems used to
manage and track test cases and bugs.
– Bug Isolation and Classification. This section of the test plan is where you
explain the degree that you intend to isolate bugs and to classify bug reports.
– Test Release Management. Every new release of a software or hardware
component into the test lab should have a release (or revision) number or
identifier attached.
– Test Cycles. By a test cycle, means running one, some, or all of the test
suites planned for a given test phase.
– Test Hours. You need to define the specific hours of testing.
 Risks and Contingencies

• However, like any other part of the project, testing is subject to

risks. These risks are possible outcomes or events that could
make the test plan difficult or impossible to carry out. It’s a good
idea to try to identify the key project risks that could affect testing
and to determine how you’ll deal with those risks. For any risk,
you have four options:
– Mitigation. Taking steps in advance that reduce the likelihood or impact of
the event or outcome.
– Contingency. Being ready to act, should the risk become an actual event or
outcome, to reduce its impact.
– Transfer. Getting another member of the project team or some other
stakeholder to accept the impact of the risk should it become an actual event
or outcome.
– Accept or ignore: Doing nothing.
 Changed History

• This part of the document records the changes and revisions that
have been made to the test plan itself to this point. Specifically,
you can assign a revision number and record who made the
changes, what those changes were, and when the revision was
released.
 Referenced Documents

• As a rule, a test plan refers to other documents such as design

specifications, requirements, the test suites, any quality risk
analysis documents, and other pertinent information. Listing these
documents in this section lets me avoid extensive repetition of
their contents (which can create complications when these
documents change).
 Frequently Asked Questions

• Many of these questions entail describing the importance of the

escalation process or some other delicate matter. You’ll find that a
frequently asked questions section is useful.

• That said, if you don’t need this section, don’t use it. As with the
Test Hours section, using it inappropriately can waste your time
and create problems. It can become a catch-all for any question
anyone ever asked about testing, bloating your test plans into
huge, unnavigable, and unmanageable documents.
Entry and Exit Criteria for Test Activities
 Entry Criteria
for Test Activities
• The entry criteria define when the activities can start, whether for
a test level or for activities within a single test level. Generally
these criteria define the availability of:
– Adequate documentation (requirements, design, operations manual, etc.)
allowing testers to determine the expected behavior of the component to be
tested;
– Test object (component, software, system) of an appropriate level of quality.
This means that the previous phases (test or design phases) were
successfully finished (the exit criteria for these previous phases have been
successfully reached);
– Test environment, test harness, drivers, and stubs necessary to execute the
component to be tested, in a format usable by the testers;
– Test resources (testers, hardware and software resources, etc.);
– Test tools and test scripts;
– Test data required for the test to be executed.
 Examples of Entry Criteria

• Availability of defect-tracking tools and test-execution tracking

tools;

• All components are managed with a configuration management

tool;

• The test environment – including all the hardware components –

is configured. The test team has obtained the required level of
access to these systems;

• All planned fixes for this version have been implemented by the
development team;

• Lower-level tests (component testing and integration testing)

have been executed for all functionalities and fixes provided in
this version;
 Examples of Entry Criteria
(cont.)
• There is no remaining open blocking or critical defect in this
version, and they are less than in other open defects for this
version;

• The development team has delivered the software to the test

team at least 3 days before the start of the system tests;

• The test team have executed 3 days of smoke tests and

acceptable results have been provided at this test phase kick-off
meeting;

• The management team, during the test phase kick-off meeting

(for this level) agrees to continue testing. The elements to cover
during this meeting are:
– Is the code complete?
– Is component testing complete and correct?
– Will the remaining identified fixed defects be delivered less than 1 week after
the start of the test phase?
Entry Criteria for SpeedyWriter

Source: Black (2009, pg. 59)

 Continuation Criteria for
SpeedyWriter

Source: Black (2009, pg. 59)

 Exit Criteria
for Test Activities
• Exit criteria define when test activities can be considered as
finished and when the component under test can be delivered to
the next tasks.

• Activity exit criteria are often based on metrics or on coverage

ratio, such as a successful execution of all the previously defined
tests cases, coverage of all instructions or all branches, of all
functionalities, or of a selected subset of functionalities.

• Too strict requirements may become blocking factors during test

execution, and it is recommended that mechanisms that prevent
testing from being considered as an activity preventing software
delivery are included.

• Exit criteria are defined per test level, per activity or for the whole
software. If the exit criteria are sufficiently detailed, they will help
in designing a strategy for unit testing and integration testing.
 Examples of Exit Criteria

• No modification (design, code or characteristics) during the last 3

weeks, except to deal with the defects identified during system
tests;

• No stopping, crash, or inexplicable end of processes on any

software servers or systems during the last 3 weeks;

• No customer systems have become unstable or unusable

following an installation failure during system testing;

• The testing team has run all tests planned on the delivery
candidate version of the software;

• The development team has solved all the “to be fixed” defects, as
planned by sales, marketing or customer services;
 Examples of Exit Criteria
(cont.)
• The testing team has checked that all issues identified in the
defect management tool have been either closed or postponed to
a subsequent version and – where applicable – have been
verified by adequate regression and confirmation tests;

• Test metrics indicate a stable and reliable product, the end of all
the planned tests, and adequate coverage of all the critical quality
risks identified;

• The product management team accepts that the product, as

defined in the last cycle of system testing, will meet reasonable
quality objectives for the customers;

• The project management team held a meeting for the end of the
system testing phase and accepts that system testing can be
considered as finished.
Exit Criteria for SpeedyWriter

Source: Black (2009, pg. 60)

Evaluating Exit Criteria
 Exit Criteria
should be Avoided
• Stopping testing when the planned test termination date is
reached. This criterion can be reached without any test being
designed or executed. It does not allow measuring any level of
quality;

• Stopping testing when the planned test effort has been reached.
This criterion does not allow measuring the final level of quality of
the software or system, and is dependent on the initial level of
quality of the software. If the planned workload is too small, the
criterion will be reached quickly, without significantly impacting
the quality level of the software;

• Stopping when all the test cases have been executed without
finding new defects. This criterion is simultaneously
counterproductive (pushes for the design of test cases that have
a small possibility of identifying defects) and depends on the
quality of the tests.
 Considerations
for Exit Criteria
• If the exit criteria specified in the test planning and organizational
phase is ignored, the following phases – including production –
provide a system that is not sufficiently mature and will potentially
have significant defects. These subsequent phases will be less
efficient (increase in the number of defects identified that have to
be corrected) and more expensive. If the exit criteria apply to
system tests or to acceptance tests, poor-quality, defective
software will be delivered to the customers and end users,
thereby leading to unhappy users, loss of perceived quality of
products, and potential loss of sales for the company.

• Exit criteria must be considered simultaneously on the basis of

customer test objectives and selected techniques. For example,
from experience-based tests, we can determine whether the
distribution of the faults corresponds to industry statistics or
shows an improvement in comparison to the previous versions.
Thank You