Testing is famous for not quite going to plan, starting

Golden Rules too late and being squeezed by fixed end dates.

My Golden Rules are not an exhaustive list and I’m

sure all readers could provide others from their own

for experiences.

Here is my top 20 (not in order of importance) that

may help you navigate your Test Strategy.
Quality Testing by Andy Redwood

Golden Rule Number 1
Testing is a Risk Management Exercise.
One of the first principles I learned as a
tester – if there is no risk nobody will be
worried. If there’s no worry, particularly
from senior management or the Business,
why expend valuable resources searching
for defects that are not important or viable
to find.
Ensure all the Requirements are risk-
assessed to the best method that works for
you or your organisation. Prioritise all the
testing to target the most risk with the least
test effort. When there are only minor risks
outstanding, an informed decision can be
made on when to stop testing.
Golden Rule Number 2
Training in Testing Reduces Long-
Term Costs
It is not un-common for less than 1% of an
organisation’s training budget to be dedi-
cated to training in testing, although with
the introduction of ISEB Foundation
Certificates this percentage is on the
increase.
Formal training in Test Management, Test
Design, Static and Dynamic techniques, to
name but a few of the many testing activi-
ties, will put systematic and structured
method into the preparation, design and
execution of tests.
If testing is an average 40% of project
budget, why is training investment so
low?
Golden Rule Number 3
The Testing Culture in an Organisation
can Only Mature at the rate of its
Capability
If you wish to improve the effectiveness of
the testing, don’t try too much too soon.
The scale of what can be achieved will
vary from one organisation to another and
will be dependent on many variables.
The first objective must be to establish all
the perceived problems. Usually integra-
tion and de-duplication within and across
different areas of responsibility will create
a balanced economy of scale. I have
worked on a number of process improve-
ment programmes, where the benefits
have been measured in the multi-million
pound mark for testing improvements
alone.
Golden Rule Number 4
The Greater the Number of Changes to
Requirements, the More Rework is
Required.
How many times have you been in a situ-
ation where the requirements are not just
changing but new ones are being created,
even when you’re in test execution?
Try and influence business and senior
managers to work to the “Bus to Bus
Stop” rule.
All the requirements get on the bus and the
doors close. Nothing gets on and nothing
gets off as the bus travel from A to B (ring-
fenced scope). Requirements may change
seats on the bus (change control). If any
new requirements are identified, they get
on the next bus coming along behind
(good release management).
If this rule is broken (and it sometimes is
for very good reason), there will be rework
associated with test preparation. The asso-
ciated cost, which can impact the original
Business Case, increases the later new
requirements are added, or when major
changes in scope occur.
Golden Rule Number 5
Testing Estimates Should be Based on
Business Risk.
It’s very tempting to give your best guess
when estimating the testing effort. If
you’re lucky, or you’ve been collecting
testing measures and metrics, you may
have previous project information to guide
you.
If you adopt Golden Rule Number 1, you
should be able to weight the risk with a
“time taken to test.” This is more difficult
on the first pass but each proceeding proj-
ect will allow you to predict with ever
increasing accuracy.
A word of warning, this does not mean
that the same tasks always take the same

22 Journal of Software Testing Professionals http://www.testinginstitute.com September 2002

time. You will need to evaluate any new
“hotspots.” However it is quite common to
hear -
Manager: “How long does it take to test?”
Acceptance Testers: “We think about six
weeks”.
Manager: “How do you know?”
Acceptance Tester: “Because it does”.
Please don’t estimate by habit!
Golden Rule Number 6
Drafting a Test Strategy from a
Facilitated Workshop is a Quick Win.
Workshops firm up known testing require-
ments very early in the life cycle.
They also allow unknown activities to sur-
face early, creating contingency within the
programme.
My measures show that on average, of the
last 100 test strategy workshops I have
facilitated an average thirty-six previously
unknown issues have been raised. Some
of these would not otherwise have been
spotted until system test execution.
Golden Rule Number 7
Linking Development Documents to
Test Documents Improves Under-
standing..
Figure 1 is an old picture (thanks to Paul
Smith), but it demonstrates the relation-
ship between development and testing
documents. It is the documentation that
we as testers must test back to. In many
cases this relationship is implied and not
explicit.
Golden Rule Number 8
All Test Conditions Should have a
Parent Requirement.
Traceability between requirements and
test conditions within a test repository (or
inventory) will ensure all requirements are
Figure 1.
tested and aid test progress reporting.
September 2002 http://www.testinginstitute.com
As part of the test design the test analyst
should continually scan back and forth
looking for requirements that have no con-
ditions and of equal importance, look for
test conditions that do not map to a
requirement.
In this situation you may for example
come across missing business processes or
have an item that requires a business deci-
sion.
Golden Rule Number 9
Always Test the Test Environment.
“The log-ons were all wrong.” “The inter-
face link was down”. “We were pointing at
the wrong files.” Just a few of the remarks
you may experience on day one of the test-
ing. These are usually due to a lack of
“pipe-cleaning” of the test environment
prior to starting the real testing. Most of
these tests (as with most testing activity)
can be check-listed.
Golden Rule Number 10
Tests Should be Executed to Prove that
the Item under Test Doesn’t Work,
Rather than that it does Work.
Journal of Software Testing Professionals
This can be a difficult mindset for some
people. There is a conflict with reporting a
high number of errors and making good
progress to plan. However, a high number
of found errors in test must remain an
important test objective, in order to ensure
a high quality and robust system in
Production.
Tests should be designed to challenge
high-risk attributes at the earliest opportu-
nity.
Golden Rule Number 11
It is Better to Test the Errors out in
Early Project Phases as Opposed to
Having a Failure in Production.
It has been suggested that 50% of all errors
found in system testing originate before
coding.
It can cost hundreds of times more to fix a
problem in production than in the require-
ments phase.
A focus on static testing techniques would
be advantages, for example, walk-
throughs, reviews, workshops and formal
inspections will raise early incidents.
23
Unfortunately in some organisations it is
still radical for testing to be perceived as
an across the life-cycle group of activities.
In general, the more senior the manage-
ment level, the harder it gets to persuade
people that testing can begin as soon as the
requirements are defined.
Golden Rule Number 12
Each Test Phase Should Attempt to Test
Something Different with as Little
Duplication as Possible.
Duplication costs. Some duplication is
unavoidable.
Testing is more efficient and effective if
the testing undertaken is defined for each
test phase, with minimal duplication, with-
in the test strategy. Details for each test
phase will appear in phase specific test
plan documents, each linked to correspon-
ding development deliverables.
I think it needs to be questioned if a par-
ticular test phase is actually required. The
basis for this decision could be risk and
viability. In my travels in and out of organ-
isations I see test phases executed only
because they appear in the life cycle or the
programme management method as an
activity. The objective for undertaking the
phase, in some cases, has already been
proven in previous phases.
Golden Rule Number 13
An Incident is not a Error or a Fault,
but is Anything that was not Expected.
Incidents can occur in any programme
phase. Incidents need not be testing spe-
cific. An incident that results from human
error or a component fault will be logged
as a problem.
It is important to evaluate all incidents,
categorise them under strategically
defined incident types and allow sufficient
time to conduct root cause analysis. In this
way you will gather meaningful measures,
establish trends, calculate metrics and
24 Journal of Software Testing Professionals
embark on a path of continuous process
improvement with reduced costs.
Golden Rule Number 14
All Incidents Should be Traceable to a
Test Condition and a Requirement.
This is fundamental if you are to know
that:
(a) you have mitigated all the risks
(b) you know you can stop testing.
This activity has been made easier over the
years by test management tools.
It is also possible to evaluate data on types
of incidents raised against requirements
(or test conditions).
Where you have raised few incidents
against certain requirements or conditions,
you may conclude that these are likely to
be stable parts of the system. In my expe-
rience areas of least testing, point at the
high-risk areas in the production system to
monitor in the first few weeks post imple-
mentation.
Golden Rule Number 15
Test Assets and Project Histories Must
be Considered for Re-use.
Project histories give factual information
and documentation of the previous
release. Having access to all documenta-
tion sets, test logs, measures and metrics
from the previous project will:
• reduce analysis, design and build costs
• allow better understanding of the exist-
ing functionality
• allow more accurate project estimates
• enable earlier test preparation
Testware is worth an average 40% of
every project budget; why not re-use this
on iterative projects to reduce the test
preparation time?
http://www.testinginstitute.com
Test packs run as regression tests:
• provide the status of changes since the
previous release
• can be constructed at all test levels to
enable maximum flexibility for testing
large or small projects
• form a baseline from which to com-
mence preparation of new tests.
Golden Rule Number 16
If you can’t Measure, you will not
Know if what you Achieve is Good or
Bad.
Measures lead to metrics lead to trends.
Metrics form management information
and allow informed decisions. Measures
make perceptions factual.
Golden Rule Number 17
Measures Point to Root Causes. Fix the
Root Cause and You Improve the
Quality.
Fixing the root cause in processes and pro-
cedures so that they never recur, will
reduce costs and improve quality of deliv-
ery.
Care is required, as it is easy to mistake
analysis for the root cause for a witch-
hunt.
The root cause could be anywhere and is
likely not to be in the phase the incident
was identified. The important thing is to
fix it and learn from it, not apportion any
blame.
Golden Rule Number 18
Tools Can Only Automate Existing
Process. Tools Require Support.
Existing process needs to be in a state that
allows it to be automated. This can be
done as the programme evolves, but in my
experience automation should run as a
parallel project to the main testing activity
on a project.
September 2002
Build the tests, run them manually and
then follow up behind with a small team
who will automate the testing for the
future. Less risk, more control.
My old friend Terry Stevens use to have a
catch phrase, “the cost of the tool is not the
cost of the tool.” This is still true. Tools are
licensed software and require some
expertise to obtain full benefit.
Those of you thinking of embarking on
automation projects can now make use of
the many user groups, limited (sometimes
free) on-site vendor support or help from
the many consultancies with experience in
this field.
Golden Rule Number 19
If You Don’t Ask a Third Party
Supplier the Question, You May Never
be Told the Answer.
I’ll try not to get on a soapbox about this
even though it is one of the most frustrat-
ing things that a Test Manager may have to
deal with.
Here are some of the most common ques-
tions you might wish to ask. I’d be happy
to supply a more detailed set (email
andy.redwood@cresta.net).
• How did you test this package before
you handed it over to us?
• Do you have any test documentation I
could look at?
• What errors do you know to be out-
standing that are in this release?
• Does this release include changes for
any other client and if so, how does it
affect us?
• What’s the turnaround time for fixing
our priority 1 problems?
September 2002 http://www.testinginstitute.com
Golden Rule Number 20
Sign off Criteria Should Never be a
Surprise to Management
When criteria are agreed in advance via
the testing strategy or lower level test
plans and reviewed at set points in the life
cycle, sign off into production is the norm,
not the exception.
Nothing escapes into production with
high-risk caveats or with insufficient test-
ing.
Build yourself a list of all interfacing man-
agers that need to always be informed as to
the status and progress of testing across all
projects. Recommend via the staged
update reports if sign off is ‘ok to go’ or
what the deviation is from the standard
approach detailed in the test strategy.
Managers will be able to make proactive
decisions as the project progresses rather
than have to wait until it has finished to
react.
About the Author
Andy Redwood is a Principal Consultant
with Cresta Group, working with clients to
help them improve the quality, efficiency
and effectiveness of their testing
resources. Andy is a frequent public
speaker at European Conferences and is
an active member within the British
Computer Society and ISEB (Information
Systems Examinations Board).
Journal of Software Testing Professionals
CALL FOR SUBMISSIONS
JSTP is looking for articles reflecting real
life experiences with any of the following
areas:
• Testing (test process, test management,
testing techniques, performance test
ing, security testing, test automation, etc.)
• Requirements
• Bug tracking and reporting
• Incident management
• Configuration management
• Release management
• Risk assessment
• Test process measurement
and improvement
All submissions for publication in JSTP
must adhere to the following guidelines:
1. Author’s Bio. A 25 word or less author’s
biography shall accompany each submis-
sion.
2. All graphics must be in high resolution
format and submitted in a separate graphic
file such as a TIF.
3. Original Content. Unless otherwise
agreed, all article submissions shall contain
original content not previously published.
Exceptions may apply only if written per-
mission is granted by the holder of any pre-
vious copyright.
4. Grant of Copyright. Unless otherwise
agreed, the copyright for articles submitted
shall be granted to JSTP for publication and
reprints.
5. Format. Unless otherwise agreed, sub-
missions shall be provided in MS Wordtm
format with page numbers (x of y) in the
footer.
6. Size. The length shall not exceed 15 pages
of MS Wordtm single-spaced, including all
charts and graphical illustrations.
7. Multiple authors. Submission of multi-
author papers is taken as an indication that
all authors agree to its publication.
8. Style. The text of the paper must be in
clear, concise and grammatically correct
English. Abbreviations and acronyms must
be clearly defined.
9. Attribution. Any quotes or excerpts from
any other person or publication shall be duly
attributed either in the text or in a footnote.
10. Introduction. The first paragraph(s) of
the article should provide a concise summa-
ry of the article’s topic, including the bene-
fit(s) to the reader.
11. Sub-Headings. Where appropriate, sub-
headings should be added to indicated main
ideas and to assist the reader in locating spe-
cific information.
12. Summary. The final paragraph(s) of the
article should provide a concise summary of
the article.
13. References. References in the text
should include the authors’ names together
with the year.
14. Bibliography. In the bibliography, list
references in alphabetical order with the
titles of journals cited in full.
25