The Social

May 2009
TESTER £1.00
Fruit tower topples over injuring tech lead

TESTERS CAR FAILS MOT

Stubborn cat refuses to move.
http://thesocialtester.posterous.com/stubborn-cat

TEST. TESTER NOT AMUSED

A software tester in Hampshire is not impressed by his car
failing MOT test. More on page 10002

EXHAUSTIVE
TESTING IS NOT
POSSIBLE
Software Tester decides
Do a search through any of the
software testing forums and I can
guarantee you will find posts from

best tactic to explain

people asking you to solve their
testing problems. More often than
not, these testing problems will

exhaustive and
consist of the worlds „test‟
„everything‟ „exhaustively‟ „how?‟.

exploratory testing is
Unless the application under test is
incredibly simple or you have an
army of testers and a fistful of

down the pub. Anyone

dollars then the answer is
inevitably „Not Possible‟.

fancy a pint? And we all know why it‟s not

possible. Because there are too
many permutations to test and not
enough time or money.

1
Continued from Front Page
Many people do respond with „Not Possible‟
and others simply don‟t even respond. They‟ve
seen the question all too many times. But for
me, there‟s another way of looking at this and
it involves exploratory testing.
I‟m not going to write an article about how it is
possible to test everything using exploratory
testing because it‟s not, the same number of
permutations exists, but I am going to argue
that understanding how the software is to be
used by the end user and for what purpose
could lead you down an exploratory road that
hits the software where it‟s needed.
The questions being asked are always from a
scripted testing point of view. i.e. How can I
write my tests in advance to hit all options.
The problem I see with scripted in advance
testing is that tests are written for everything
the spec/design says will be implemented.
This makes sense but it also means that
unexpected and unintentional side effects of
meeting the requirements often go over
looked. After all, you can‟t write a test case
explaining how to generate a bug and what
the bug will look like. You can only write a test
case to question the system to search for a
bug.
2
So we can check the software meets expected
outcomes which is valid but often doesn‟t tell
us the whole story. We often struggle to write
scripts that explore the application with
information that is relevant, up-to-date and
accurate. This is best left to exploration. And a
good combination of both scripted and
exploratory is a fantastic place to be.
Being able understand the system, the end
users and the contexts in which the system will
be used will undoubtedly help us explore the
system better. Knowing how people interact
will always give us a natural advantage over
testers who don‟t know their audience,
purpose and context. Sure, they may raise
defects but are they as important as the defect
that realistically affects the end user?
In the past I‟ve worked on products that are
shiny and spanky and work as expected yet on
the first day of release each customer reports
defects. It didn‟t work under this circumstance,
it fell over when I did this whilst Bob was doing
that, it refuses to do this action that wasn‟t in
the requirements but should have been etc.
We‟ve all seen this and read about this. It‟s
part of software development. But we can do
our part to counteract this and as testers, I feel
it is our obligation to try to understand how the
end users operate. We should never replace
the end user in testing. A perfect place to be
though would be to schedule time with the end
user at various times in the development,
letting them use it in anger…and in context.
Agile methodologies are going someway to
countering this with regular customer feedback
but often the customer on the project is not the
end user of the software.
One of the ways I try to teach this to my testers is to ask them to imagine being asked to test a pub. Yes. A
pub. What a job.

“Go to your average pub and test each and every single service being offered there”

Wow. Some people hear this and say simple – and off they go. Others think it‟s impossible.

The fact is, it is impossible. The main reason it is impossible is because there is no such thing as an
average pub. But also, what defines a service? What constitutes a pass/fail?

Yet all the time we hear of testers facing these same dilemmas. They have a vast system that does X, Y, Z
but interfaces with A, B and C and is used by D, E and F. How can I test it. I must write thousands of tests
for each permutation. Must have every test in advance.

So I ask my learners to write down a list of no more than 20 of the most important test cases they would
write to test a pub. And here‟s the pretty average list I get back.

1. We could write tests for each beer, cider and lager on offer.
2. We could write tests for each wine on offer.
3. We could write tests for each soft drink on offer.
4. We could write tests for each spirit on offer.
5. We could write tests for each alco-pop on offer.
6. We could write tests for each snack on offer.
7. We could write tests for each food on offer.
8. We could write tests for each cocktail on offer.
9. We could write tests for each spirit and mixer on offer.
10. We could write tests for each hot drink on offer.
11. We could write tests for each pub machine/quiz on offer.
12. We could write tests for each toilet available.
13. We could write tests for each potential interaction on offer.
14. We could even go further and work out the typical person who drinks there and write tests for each
of them too.

Ok, so we‟ve got lots of scripted tests and they are pretty much spot on. They are the basics and I would be
surprised if any tester (who drinks) had not got that list. But the problem for me is that no matter how many
times we ran each of those scripts we have missed one absolutely crucial aspect. Context. We have still not
fully understood the pub. The only real way to fully understand the context is to observe the people who will
use the service. In this example – the pub goers.

3
So I asked the class to list a typical group of drinkers and I got these:

 The locals/regulars
 The binge drinkers at the weekend
 The diners
 The underage
 Students

It‟s actually a pretty standard list and I know they had pre-conceived ideas based on *their* experience. But
here‟s what I actually observed in my local.

 The locals/regulars
 The binge drinkers
 The diners
 The underage
 Students
 The incredibly drunk and aggressive
 The Hen and Stag Do
 The birthday party (and other private party)
 The social meeting (a local development group meet there)
 The one off drinker – killing time waiting for a train etc.
 The tourist
 The business man
 The emotional meeting (engagement, first date etc)
 Business Meeting

We now have a list of end users which will no doubt help us test the pub. We could get blinding drunk and
see whether the beer still tastes good. But we are still missing some more elements of context. These are
distractions.

 Attractive pub staff and customers

 Quiz and games machines
 Toilet trips
 Aggressive customers
 Dropped glasses and spilt drinks
 The drunk

4
  Pub staff
 Partners and friends
 Mobile Phones
 Laptops
 The TV

But that‟s not all context has to offer us. Context is also about the time at which something happens.

 Week night out

 Weekend
 Weekend away
 On holiday
 During the day
 Early morning
 Afternoon only
 Late at night

But hold on, there‟s some more contexts to consider in the form of the type of pub/pub.

 Trendy wine pub

 Traditional Pub
 Restaurant with licensed pub
 Social club
 Hotel or other establishment.

And there‟s some more in terms of the social/economical make up of the customers:

 Rich
 Poor
 Average
 Celebrating
 Stag or Hen Do
 Business Meeting

And yet more when I looked at the actual people in the pub

 Their mood
 Their drink threshold or capacity

5
  Mental state
 Tiredness
 Alcoholic state
 Hungry?
 Irritable?

Wait – there‟s more. What about the actual pub?

 Village?
 City?
 Busy street?
 Quiet Street?
 Big?
 Small?
 Fast Service?
 Slow Service?
 Modern?
 Old?
 Reputable and reliable suppliers?

I feel a matrix coming on here.

So you get the picture. One pub. Thousands of contexts. The pub is a container with processes, services,
options and features. It‟s the people and the way they interact that truly makes a pub a pub. And this is of-
ten so overlooked by testers.

Sure, we shouldn‟t assume we are the end user but we certainly should be trying to understand them and
the context in which they use the software.

Most pubs probably don‟t cater for all circumstances and all people but even so, there are still literally thou-
sands of permutations. What happens when the stag party orders a drink made of beer, Tia Maria and lime
juice? Does the till recognize that combination? Do the pub staff know how to make it? Does it make the
Stag Party too rowdy and if so, what happens then? What if the drink the stag wants has run out? What if
the drunk passes out on the floor? What if someone orders food 1 minute past the deadline? What if the
pub is heaving?

Even looking at the drinks an average pub serves offers up a whole range of combinations.

6
And yet the testing community seem hung up on writing tests from specs and designs, which in turn are
written from the customer‟s ideas and requirements which may or may not have been written by observing
the real end users.

My guess is that they haven‟t been written from observations otherwise they‟d be too complicated to com-
municate. So when a tester asks me where to start when trying to test a mammoth system. I tell them to
observe the end user, chat to the end user, find out who they are, find out why they want the software and
find out under what context the software will be used.

There‟s a Purpose, an Audience and a Context to every single piece of software out there. And if these are
confused, unknown or not catered for the software will fail. If the team building and testing the software
know the Purpose, the Audience and the Context then the software stands a much better chance of being
fit for purpose.

To not understand the purpose, audience and context is to not understand the software under test. To write
test cases for the combinations identified and not spend time exploring under contexts is to miss the real
use of the system.

Pub. Computer System? It‟s all the same really. So when one of your team asks how they can test every-
thing or how every test should have a test case; take them down the pub for a pint.

Article brought to you by The Social Tester

www.thesocialtester.co.uk
http://thesocialtester.posterous.com/