E-Book : Software Testing By Ron Patton Publisher: Sams Publishing (Chapter 1 Chapter 1.

Software Testing Background IN THIS CHAPTER !nfamous Software Error Case Stu"ies #hat !s a Bug$ #hy %o Bugs &''ur$ The Cost of Bugs #hat E(a'tly %oes a Software Tester %o$ #hat )akes a *oo" Software Tester$

In 1947, computers were big, room-sized machines operating on mechanical relays and glowing vacuum tubes. The state o the art at the time was the !ar" II, a behemoth being built at #arvard $niversity. Technicians were running the new computer through its paces when it suddenly stopped wor"ing. They scrambled to igure out why and discovered, stuc" between a set o relay contacts deep in the bowels o the computer, a moth. It had apparently lown into the system, attracted by the light and heat, and was zapped by the high voltage when it landed on the relay. The computer bug was born. %ell, o"ay, it died, but you get the point. %elcome to the irst chapter o &o tware Testing. In this chapter, you'll learn about the history o so tware bugs and so tware testing. #ighlights o this chapter include #ow so tware bugs impact our lives %hat bugs are and why they occur %ho so tware testers are and what they do

Infamous Software Error Case Studies It's easy to ta"e so tware or granted and not really appreciate how much it has in iltrated our daily lives. (ac" in 1947, the !ar" II computer re)uired legions o programmers to constantly maintain it. The average person never conceived o someday having his own computer in his home. *ow there's ree so tware +,--.!s attached to cereal bo/es and more so tware in our "ids' video games than on the space shuttle. %hat once were techie gadgets, such as pagers and cell phones, have become commonplace. !ost o us now can't go a day without logging on to the Internet and chec"ing our email. %e rely on overnight pac"ages, long-distance phone service, and cutting-edge medical treatments.

&o tware is everywhere. #owever, it's written by peopleso it's not per ect, as the ollowing e/amples show. %isney+s ,ion -ing. 1//01//1 In the all o 1994, the ,isney company released its irst multimedia +,--.! game or children, The 0ion 1ing 2nimated &toryboo". 2lthough many other companies had been mar"eting children's programs or years, this was ,isney's irst venture into the mar"et and it was highly promoted and advertised. &ales were huge. It was 3the game to buy3 or children that holiday season. %hat happened, however, was a huge debacle. .n ,ecember 45, the day a ter +hristmas, ,isney's customer support phones began to ring, and ring, and ring. &oon the phone support technicians were swamped with calls rom angry parents with crying children who couldn't get the so tware to wor". *umerous stories appeared in newspapers and on T6 news. It turns out that ,isney ailed to test the so tware on a broad representation o the many di erent 7+ models available on the mar"et. The so tware wor"ed on a ew systemsli"ely the ones that the ,isney programmers used to create the gamebut not on the most common systems that the general public had. !ntel Pentium 2loating-Point %i3ision Bug. 1//0 8nter the ollowing e)uation into your 7+'s calculator9 (4195835 / 3145727) * 3145727 - 4195835 I the answer is zero, your computer is :ust ine. I you get anything else, you have an old Intel 7entium +7$ with a loating-point division buga so tware bug burned into a computer chip and reproduced over and over in the manu acturing process. .n .ctober ;<, 1994, ,r. Thomas -. *icely o 0ynchburg =6irginia> +ollege traced an une/pected result rom one o his e/periments to an incorrect answer by a division problem solved on his 7entium 7+. #e posted his ind on the Internet and soon a terward a irestorm erupted as numerous other people duplicated his problem and ound additional situations that resulted in wrong answers. ?ortunately, these cases were rare and resulted in wrong answers only or e/tremely math-intensive, scienti ic, and engineering calculations. !ost people would never encounter them doing their ta/es or running their businesses. %hat ma"es this story notable isn't the bug, but the way Intel handled the situation9 Their so tware test engineers had ound the problem while per orming their own tests be ore the chip was released. Intel's management decided that the problem wasn't severe enough or li"ely enough to warrant i/ing it or even publicizing it. .nce the bug was ound, Intel attempted to diminish its perceived severity through press releases and public statements. %hen pressured, Intel o ered to replace the aulty chips, but only i a user could prove that he was a ected by the bug.

There was a public outcry. Internet newsgroups were :ammed with irate customers demanding that Intel i/ the problem. *ews stories painted the company as uncaring and incredulous. In the end, Intel apologized or the way it handled the bug and too" a charge o more than @4<< million to cover the costs o replacing bad chips. Intel now reports "nown problems on its website and care ully monitors customer eedbac" on Internet newsgroups. *.T8

.n 2ugust 4Ath, 4<<<, shortly be ore the irst edition o this boo" went to press, Intel announced a recall o all the 1.1;!#z 7entium III processors it had shipped a ter the chip had been in production or a month. 2 problem was discovered with the e/ecution o certain instructions that could cause running applications to reeze. +omputer manu acturers were creating plans or recalling the 7+s already in customers' hands and calculating the costs o replacing the de ective chips. 2s the baseball legend Bogi (erra once said, 3This is li"e dC:D vu all over again.3 45S5 )ars Polar ,an"er. 1/// .n ,ecember ;, 1999, *2&2's !ars 7olar 0ander disappeared during its landing attempt on the !artian sur ace. 2 ?ailure -eview (oard investigated the ailure and determined that the most li"ely reason or the mal unction was the une/pected setting o a single data bit. !ost alarming was why the problem wasn't caught by internal tests. In theory, the plan or landing was this9 2s the lander ell to the sur ace, it was to deploy a parachute to slow its descent. 2 ew seconds a ter the chute deployed, the probe's three legs were to snap open and latch into position or landing. %hen the probe was about 1,A<< meters rom the sur ace, it was to release the parachute and ignite its landing thrusters to gently lower it the remaining distance to the ground. To save money, *2&2 simpli ied the mechanism or determining when to shut o the thrusters. In lieu o costly radar used on other spacecra t, they put an ine/pensive contact switch on the leg's oot that set a bit in the computer commanding it to shut o the uel. &imply, the engines would burn until the legs 3touched down.3 $n ortunately, the ?ailure -eview (oard discovered in their tests that in most cases when the legs snapped open or landing, a mechanical vibration also tripped the touch-down switch, setting the atal bit. It's very probable that, thin"ing it had landed, the computer turned o the thrusters and the !ars 7olar 0ander smashed to pieces a ter alling 1,A<< meters to the sur ace. The result was catastrophic, but the reason behind it was simple. The lander was tested by multiple teams. .ne team tested the leg old-down procedure and another the landing process rom that point on. The irst team never loo"ed to see i the touch-down bit was setit wasn't their areaE the second team always reset the computer, clearing the bit, be ore it started its testing. (oth pieces wor"ed per ectly individually, but not when put together.

Patriot )issile %efense System. 1//1 The $.&. 7atriot missile de ense system is a scaled-bac" version o the &trategic ,e ense Initiative =3&tar %ars3> program proposed by 7resident -onald -eagan. It was irst put to use in the Ful %ar as a de ense or Ira)i &cud missiles. 2lthough there were many news stories touting the success o the system, it did ail to de end against several missiles, including one that "illed 4A $.&. soldiers in ,hahran, &audi 2rabia. 2nalysis ound that a so tware bug was the problem. 2 small timing error in the system's cloc" accumulated to the point that a ter 14 hours, the trac"ing system was no longer accurate. In the ,hahran attac", the system had been operating or more than 1<< hours. The 67- (6ear 7888 Bug. 'ir'a 1/90 &ometime in the early 197<s a computer programmerlet's suppose his name was ,avewas wor"ing on a payroll system or his company. The computer he was using had very little

memory or storage, orcing him to conserve every last byte he could. ,ave was proud that he could pac" his programs more tightly than any o his peers. .ne method he used was to shorten dates rom their 4-digit ormat, such as 197;, to a 4-digit ormat, such as 7;. (ecause his payroll program relied heavily on date processing, ,ave could save lots o e/pensive memory space. #e brie ly considered the problems that might occur when the current year hit 4<<< and his program began doing computations on years such as << and <1. #e "new there would be problems but decided that his program would surely be replaced or updated in 4G years and his immediate tas"s were more important than planning or something that ar out in time. 2 ter all, he had a deadline to meet. In 199G, ,ave's program was still being used, ,ave was retired, and no one "new how to get into the program to chec" i it was B41 compliant, let alone how to i/ it. It's estimated that several hundred billion dollars were spent, worldwide, to replace or update computer programs such as ,ave's, to i/ potential Bear 4<<< ailures. %angerous :iewing 5hea". 7880 .n 2pril 1, 1994, a message was posted to several Internet user groups and then )uic"ly circulated as an email that a virus was discovered embedded in several H78F ormat pictures available on the Internet. The warning stated that simply opening and viewing the in ected pictures would install the virus on your 7+. 6ariations o the warning stated that the virus could damage your monitor and that &ony Trinitron monitors were 3particularly susceptible.3 !any heeded the warning, purging their systems o H78F iles. &ome system administrators even went so ar as to bloc" H78F images rom being received via email on the systems. 8ventually people realized that the original message was sent on 32pril ?ools ,ay3 and that the whole event was nothing but a :o"e ta"en too ar. 8/perts chimed in that there was no possible way viewing a H78F image could in ect your 7+ with a virus. 2 ter all, a picture is :ust data, it's not e/ecutable program code. Ten years later, in the all o 4<<4, a proo -o -concept virus was created, proving that a H78F picture could be loaded with a virus that would in ect the system used to view it. &o tware patches were )uic"ly made and updates distributed to prevent such a virus rom spreading. #owever, it may only be a matter o time until a means o transmission, such as an innocuous picture, succeeds in wrec"ing havoc over the Internet.

#hat !s a Bug$ Bou've :ust read e/amples o what happens when so tware ails. It can be inconvenient, as when a computer game doesn't wor" properly, or it can be catastrophic, resulting in the loss o li e. It can cost only pennies to i/ but millions o dollars to distribute a solution. In the e/amples, above, it was obvious that the so tware didn't operate as intended. 2s a so tware tester you'll discover that most ailures are hardly ever this obvious. !ost are simple, subtle ailures, with many being so small that it's not always clear which ones are true ailures, and which ones aren't. Terms for Software 2ailures ,epending on where you're employed as a so tware tester, you will use di erent terms to describe what happens when so tware ails. #ere are a ew9 ,e ect ?ault 6ariance ?ailure

,e ect

6ariance

7roblem Inconsistency 8rror Incident 2nomaly =There's also a list o unmentionable terms, but they're most o ten used privately among programmers.> Bou might be amazed that so many names could be used to describe a so tware ailure. %hy so manyI It's all really based on the company's culture and the process the company uses to develop its so tware. I you loo" up these words in the dictionary, you'll ind that they all have slightly di erent meanings. They also have in erred meanings by how they're used in day-today conversation. ?or e/ample, ault, ailure, and de ect tend to imply a condition that's really severe, maybe even dangerous. It doesn't sound right to call an incorrectly colored icon a ault. These words also tend to imply blame9 3It's his ault that the so tware ailed.3 2nomaly, incident, and variance don't sound )uite so negative and are o ten used to in er unintended operation rather than all-out ailure. 3The president stated that it was a so tware anomaly that caused the missile to go o course.3 7roblem, error, and bug are probably the most generic terms used. ;<ST C5,, !T #=5T !T !S 54% *ET &4 #!T= !T It's interesting that some companies and product teams will spend hours and hours o precious development time arguing and debating which term to use. 2 well-"nown computer company spent wee"s in discussion with its engineers be ore deciding to rename 7roduct 2nomaly -eports =72-s> to 7roduct Incident -eports =7I-s>. +ountless dollars were spent in the process o deciding which term was better. .nce the decision was made, all the paperwor", so tware, orms, and so on had to be updated to re lect the new term. It's un"nown i it made any di erence to the programmer's or tester's productivity. &o, why bring this topic upI It's important as a so tware tester to understand the personality behind the product development team you're wor"ing with. #ow they re er to their so tware problems is a tell-tale sign o how they approach their overall development process. 2re they cautious, care ul, direct, or :ust plain bluntI 2lthough your team may choose a di erent name, in this boo", all so tware problems will be called bugs. It doesn't matter i it's big, small, intended, unintended, or someone's eelings will be hurt because they create one. There's no reason to dice words. 2 bug's a bug's a bug. Software Bug: 5 2ormal %efinition ?eature (ug

+alling any and all so tware problems bugs may sound simple enough, but doing so hasn't really addressed the issue. *ow the word problem needs to be de ined. To "eep rom running in circular de initions, there needs to be a de initive description o what a bug is. ?irst, you need a supporting term9 product speci ication. 2 product speci ication, sometimes re erred to as simply a spec or product spec, is an agreement among the so tware development team. It de ines the product they are creating, detailing what it will be, how it will act, what it will do, and what it won't do. This agreement can range in orm rom a simple verbal understanding, an email, or a scribble on a nap"in, to a highly detailed, ormalized written document. In +hapter 4, 3The &o tware ,evelopment 7rocess,3 you will learn more about so tware speci ications and the development process, but or now, this de inition is su icient. ?or the purposes o this boo" and much o the so tware industry, a so tware bug occurs when one or more o the ollowing ive rules is true9

1. The so tware doesn't do something that the product speci ication says it should do. 2. The so tware does something that the product speci ication says it shouldn't do. 3. The so tware does something that the product speci ication doesn't mention. 4. The so tware doesn't do something that the product speci ication doesn't mention but
should.

5. The so tware is di icult to understand, hard to use, slow, or in the so tware tester's
eyes will be viewed by the end user as :ust plain not right. To better understand each rule, try the ollowing e/ample o applying them to a calculator. The speci ication or a calculator probably states that it will per orm correct addition, subtraction, multiplication, and division. I you, as the tester, receive the calculator, press the J "ey, and nothing happens, that's a bug because o -ule K1. I you get the wrong answer, that's also a bug because o -ule K1. The product spec might state that the calculator should never crash, loc" up, or reeze. I you pound on the "eys and get the calculator to stop responding to your input, that's a bug because o -ule K4. &uppose that you receive the calculator or testing and ind that besides addition, subtraction, multiplication, and division, it also per orms s)uare roots. *owhere was this ever speci ied. 2n ambitious programmer :ust threw it in because he elt it would be a great eature. This isn't a eatureit's really a bug because o -ule K;. The so tware is doing something that the product speci ication didn't mention. This unintended operation, although maybe nice to have, will add to the test e ort and will li"ely introduce even more bugs. The ourth rule may read a bit strange with its double negatives, but its purpose is to catch things that were orgotten in the speci ication. Bou start testing the calculator and discover when the battery gets wea" that you no longer receive correct answers to your calculations. *o one ever considered how the calculator should react in this mode. 2 bad assumption was made that the batteries would always be ully charged. Bou e/pected it to "eep wor"ing until the batteries were completely dead, or at least noti y you in some way that they were wea". +orrect calculations didn't happen with wea" batteries, and it wasn't speci ied what should happen. -ule K4 ma"es this a bug. -ule KG is the catch-all. 2s a tester you are the irst person to really use the so tware. I you weren't there, it would be the customer using the product or the irst time. I you ind something that you don't eel is right, or whatever reason, it's a bug. In the case o the calculator, maybe you ound that the buttons were too small. !aybe the placement o the L "ey made it hard to use. !aybe the display was di icult to read under bright lights. 2ll o these are bugs because o -ule KG. *.T8

8very person who uses a piece o so tware will have di erent e/pectations and opinions as to how it should wor". It would be impossible to write so tware that every user thought was per ect. 2s a so tware tester, you should "eep this in mind when you apply -ule KG to your testing. (e thorough, use your best :udgment, and, most importantly, be reasonable. Bour opinion counts, but, as you'll learn in later chapters, not all the bugs you ind can or will be i/ed. These are greatly simpli ied e/amples, so thin" about how the rules apply to so tware that you use every day. %hat is e/pected, what is une/pectedI %hat do you thin" was speci ied and what was orgottenI 2nd, what do you :ust plain disli"e about the so twareI This de inition o a bug covers a lot o ground but using all ive o its rules will help you identi y the di erent types o problems in the so tware you're testing.

#hy %o Bugs &''ur$ *ow that you "now what bugs are, you might be wondering why they occur. %hat you'll be surprised to ind out is that most o them aren't caused by programming errors. *umerous studies have been per ormed on very small to e/tremely large pro:ects and the results are always the same. The number one cause o so tware bugs is the speci ication =see ?igure 1.1>.

2igure 1>1> Bugs are 'ause" for numerous reasons. but. in this sample pro?e't analysis. the main 'ause 'an be tra'e" to the spe'ifi'ation>

There are several reasons speci ications are the largest bug producer. In many instances a spec simply isn't written. .ther reasons may be that the spec isn't thorough enough, it's constantly changing, or it's not communicated well to the entire development team. 7lanning so tware is vitally important. I it's not done correctly, bugs will be created. The ne/t largest source o bugs is the design. This is where the programmers lay out their plan or the so tware. +ompare it to an architect creating the blueprints or a building. (ugs occur

here or the same reason they occur in the speci ication. It's rushed, changed, or not well communicated. *.T8 There's an old saying, 3I you can't say it, you can't do it.3 This applies per ectly to so tware development and testing.

+oding errors may be more amiliar to you i you're a programmer. Typically, these can be traced to the so tware's comple/ity, poor documentation =especially in code that's being updated or revised>, schedule pressure, or :ust plain dumb mista"es. It's important to note that many bugs that appear on the sur ace to be programming errors can really be traced to speci ication and design errors. It's )uite common to hear a programmer say, 3.h, so that's what it's supposed to do. I somebody had :ust told me that I wouldn't have written the code that way.3 The other category is the catch-all or what's le t. &ome bugs can be blamed on alse positives, conditions that were thought to be bugs but really weren't. There may be duplicate bugs, multiple ones that resulted rom the same root cause. &ome bugs can also be traced to testing errors. In the end, these bugs =or what once were thought o as bugs> turn out to not be bugs at all and ma"e up a very small percentage o all the bugs reported.

The Cost of Bugs 2s you will learn in +hapter 4, so tware doesn't :ust magically appear there's usually a planned, methodical development process used to create it. ?rom its inception, through the planning, programming, and testing, to its use by the public, there's the potential or bugs to be ound. ?igure 1.4 shows an e/ample o how the cost o i/ing these bugs can grow over time.

The costs are logarithmic that is, they increase ten old as time increases. 2 bug ound and i/ed during the early stages when the speci ication is being written might cost ne/t to nothing, or @1 in our e/ample. The same bug, i not ound until the so tware is coded and tested, might cost @1< to @1<<. I a customer inds it, the cost could easily be thousands or even millions o dollars. 2s an e/ample o how this wor"s, consider the ,isney 0ion 1ing case discussed earlier. The root cause o the problem was that the so tware wouldn't wor" on a very popular 7+ plat orm. I , in the early speci ication stage, someone had researched what 7+s were popular and speci ied that the so tware needed to be designed and tested to wor" on those con igurations, the cost o that e ort would have been minimal. I that didn't occur, a bac"up would have been or the so tware testers to collect samples o the popular 7+s and veri y the so tware on them. They would have ound the bug, but it would have been more e/pensive to i/ because the so tware would have to be debugged, i/ed, and retested. The development team could have also sent out a preliminary version o the so tware to a small group o customers in what's called a beta test. Those customers, chosen to represent the larger mar"et, would have li"ely discovered the problem. 2s it turned out, however, the bug was completely missed until many thousands o +,--.!s were created and purchased. ,isney ended up paying or telephone customer support, product returns, replacement +,--.!s, as well as another debug, i/, and test cycle. It's very easy to burn up your entire product's pro it i serious bugs ma"e it to the customer. #hat E(a'tly %oes a Software Tester %o$ Bou've now seen e/amples o really nasty bugs, you "now what the de inition o a bug is, and you "now how costly they can be. (y now it should be pretty evident what a tester's goal is9 The goal o a so tware tester is to ind bugs. Bou may run across product teams who want their testers to simply con irm that the so tware wor"s, not to ind bugs. -eread the case study about the !ars 7olar 0ander, and you'll see why this is the wrong approach. I you're only testing things that should wor" and setting up your tests so they'll pass, you will miss the things that don't wor". Bou will miss the bugs. I you're missing bugs, you're costing your pro:ect and your company money. 2s a so tware tester you shouldn't be content at :ust inding bugsyou should thin" about how to ind them sooner in the development process, thus ma"ing them cheaper to i/. The goal o a so tware tester is to ind bugs and ind them as early as possible. (ut, inding bugs, even inding them early, isn't enough. -emember the de inition o a bug. Bou, the so tware tester, are the customer's eyes, the irst one to see the so tware. Bou spea" or the customer and must see" per ection. The goal o a so tware tester is to ind bugs, ind them as early as possible, and ma"e sure they get i/ed. This inal de inition is very important. +ommit it to memory and re er bac" to it as you learn the testing techni)ues discussed throughout the rest o this boo". *.T8 It's important to note that 3 i/ing3 a bug does not necessarily imply correcting the so tware. It could mean adding a comment in the user manual or providing special training to the customers. It could re)uire changing the statistics that the mar"eting group advertises or even postponing the release o the buggy eature. Bou'll learn throughout this boo" that although you're see"ing per ection and ma"ing sure that the bugs get i/ed, that there are practical

realities to so tware testing. ,on't get caught in the dangerous spiral o unattainable per ection.

#hat )akes a *oo" Software Tester$ In the movie &tar Tre" II9 The %rath o 1han, &poc" says, 32s a matter o cosmic history, it has always been easier to destroy than to create.3 2t irst glance, it may appear that a so tware tester's :ob would be easier than a programmer's. (rea"ing code and inding bugs must surely be easier than writing the code in the irst place. &urprisingly, it's not. The methodical and disciplined approach to so tware testing that you'll learn in this boo" re)uires the same hard wor" and dedication that programming does. It involves very similar s"ills, and although a so tware tester doesn't necessarily need to be a ull- ledged programmer, having that "nowledge is a great bene it. Today, most mature companies treat so tware testing as a technical engineering pro ession. They recognize that having trained so tware testers on their pro:ect teams and allowing them to apply their trade early in the development process allows them to build better )uality so tware. $n ortunately, there are still a ew companies that don't appreciate the challenge o so tware testing and the value o well-done testing e ort. In a ree mar"et society, these companies usually aren't around or long because the customers spea" with their wallets and choose not to buy their buggy products. 2 good test organization =or the lac" o one> can ma"e or brea" a company. #ere's a list o traits that most so tware testers should have9 They are e/plorers. &o tware testers aren't a raid to venture into un"nown situations. They love to get a new piece o so tware, install it on their 7+, and see what happens. They are troubleshooters. &o tware testers are good at iguring out why something doesn't wor". They love puzzles. They are relentless. &o tware testers "eep trying. They may see a bug that )uic"ly vanishes or is di icult to re-create. -ather than dismiss it as a lu"e, they will try every way possible to ind it. They are creative. Testing the obvious isn't su icient or so tware testers. Their :ob is to thin" up creative and even o -the-wall approaches to ind bugs. They are =mellowed> per ectionists. They strive or per ection, but they "now when it becomes unattainable and they're o"ay with getting as close as they can. They e/ercise good :udgment. &o tware testers need to ma"e decisions about what they will test, how long it will ta"e, and i the problem they're loo"ing at is really a bug. They are tact ul and diplomatic. &o tware testers are always the bearers o bad news. They have to tell the programmers that their baby is ugly. Food so tware testers "now how to do so tact ully and pro essionally and "now how to wor" with programmers who aren't always tact ul and diplomatic. They are persuasive. (ugs that testers ind won't always be viewed as severe enough to be i/ed. Testers need to be good at ma"ing their points clear, demonstrating why the bug does indeed need to be i/ed, and ollowing through on ma"ing it happen.

10

S&2T#5RE TEST!4* !S 2<4@ 2 undamental trait o so tware testers is that they simply li"e to brea" things. They live to ind those elusive system crashes. They ta"e great satis action in laying to waste the most comple/ programs. They're o ten seen :umping up and down in glee, giving each other highives, and doing a little dance when they bring a system to its "nees. It's the simple :oys o li e that matter the most.

In addition to these traits, having some education in so tware programming is a big plus. 2s you'll see in +hapter 5, 38/amining the +ode,3 "nowing how so tware is written can give you a di erent view o where bugs are ound, thus ma"ing you a more e icient and e ective tester. It can also help you develop the testing tools discussed in +hapter 1G, 32utomated Testing and Test Tools.3 0astly, i you're an e/pert in some non-computer ield, your "nowledge can be invaluable to a so tware team creating a new product. &o tware is being written to do :ust about everything today. Bour "nowledge o teaching, coo"ing, airplanes, carpentry, medicine, or whatever would be a tremendous help inding bugs in so tware or those areas. Summary &o tware testing is a critical :ob. %ith the size and comple/ity o today's so tware, it's imperative that so tware testing be per ormed pro essionally and e ectively. Too much is at ris". %e don't need more de ective computer chips, crashed systems, or stolen credit card numbers. In the ollowing chapters o 7art I, 3The Big Pi'ture,3 you'll learn more about the big picture o so tware development and how so tware testing its in. This "nowledge is critical to helping you apply the speci ic test techni)ues covered in the remainder o this boo". AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. In the Bear 4<<< bug e/ample, did ,ave do anything wrongI 2. True or ?alse9 It's important what term your company or team calls a problem in its so tware. ;. %hat's wrong with :ust testing that a program wor"s as e/pectedI 4. #ow much more does it cost to i/ a bug ound a ter the product is released than it does rom the very start o the pro:ectI G. %hat's the goal o a so tware testerI

6. True or ?alse9 2 good tester relentlessly strives or per ection.

7. Five several reasons why the product speci ication is usually the largest source o bugs in a so tware product.

11

Chapter 2. The Software Development Process

I* T#I& +#27T8 7roduct +omponents &o tware 7ro:ect &ta &o tware ,evelopment 0i ecycle !odels

To be an e ective so tware tester, it's important to have at least a high-level understanding o the overall process used to develop so tware. I you write small programs as a student or hobbyist, you'll ind that the methods you use are much di erent rom what big companies use to develop so tware. The creation o a new so tware product may involve dozens, hundreds, even thousands o team members all playing di erent roles and wor"ing together under tight schedules. The speci ics o what these people do, how they interact, and how they ma"e decisions are all part o the so tware development process. The goal o this chapter isn't to teach you everything about the so tware development process that would ta"e an entire boo"N The goal is to give you an overview o the all the pieces that go into a so tware product and a loo" at a ew o the common approaches in use today. %ith this "nowledge you'll have a better understanding o how best to apply the so tware testing s"ills you learn in the later chapters o this boo". The highlights o this chapter include %hat ma:or components go into a so tware product %hat di erent people and s"ills contribute to a so tware product #ow so tware progresses rom an idea to a inal product

Pro"u't Components %hat e/actly is a so tware productI !any o us thin" o it as simply a program that we download rom the Internet or install rom a ,6, that runs on our computer. That's a pretty good description, but in reality, many hidden pieces go into ma"ing that so tware. There are also many pieces that 3come in the bo/3 that are o ten ta"en or granted or might even be ignored. 2lthough it may be easy to orget about all those parts, as a so tware tester, you need to be aware o them, because they're all testable pieces and can all have bugs. #hat Effort *oes !nto a Software Pro"u't$ ?irst, loo" at what e ort goes into a so tware product. ?igure 4.1 identi ies a ew o the abstract pieces that you may not have considered. 2igure 7>1> 5 lot of hi""en effort goes into a software pro"u't>

12

&o what are all these things, besides the actual code, that get unneled into the so twareI 2t irst glance they probably seem much less tangible than the program listing a programmer creates. 2nd they de initely aren't something that can be viewed directly rom the product's +,--.!. (ut, to paraphrase a line rom an old spaghetti sauce commercial, 3they're in there.3 2t least, they should be. The term used in the so tware industry to describe a so tware product component that's created and passed on to someone else is deliverable. The easiest way to e/plain what all these deliverables are is to organize them into ma:or categories.

Customer ReCuirements &o tware is written to ul ill some need that a person or a group o people has. 0et's call them the customer. To properly ill that need, the product development team must ind out what the customer wants. &ome teams simply guess, but most collect detailed in ormation in the orm o surveys, eedbac" rom previous versions o the so tware, competitive product in ormation, magazine reviews, ocus groups, and numerous other methods, some ormal, some not. 2ll this in ormation is then studied, condensed, and interpreted to decide e/actly what eatures the so tware product should have.

P<T 6&<R 2E5T<RES !4 PERSPECT!:E #!T= 2&C<S *R&<PS 2 popular means to get direct eedbac" rom potential customers o a so tware product is to use ocus groups. ?ocus groups are o ten organized by independent survey companies who set up o ices in shopping malls. The surveyors typically wal" around the mall with a clipboard and as" passers-by i they want to ta"e part in a study. They'll as" a ew )uestions to )uali y you such as 3,o you have a 7+ at homeI ,o you use so tware OI #ow much time do you spend onlineI3 2nd so on. I you it their demographic, they'll invite you to return or a ew hours to participate with several other people in a ocus group. There, you'll be as"ed more detailed )uestions about computer so tware. Bou may be shown various so tware bo/es and be as"ed to

13

choose your avorite. .r, you may discuss as a group eatures you'd li"e to see in a new product. (est o all, you get paid or your time. !ost ocus groups are conducted in such a way that the so tware company re)uesting the in ormation is "ept anonymous. (ut, it's usually easy to igure out who they are. Spe'ifi'ations The result o the customer re)uirements studies is really :ust raw data. It doesn't describe the proposed product, it :ust con irms whether it should =or shouldn't> be created and what eatures the customers want. The speci ications ta"e all this in ormation plus any unstated but mandatory re)uirements and truly de ine what the product will be, what it will do, and how it will loo". The ormat o speci ications varies greatly. &ome companies specially those developing products or the government, aerospace, inancial and medical industries use a very rigorous process with many chec"s and balances. The result is an e/tremely detailed and thorough speci ication that's loc"ed down, meaning that it can't change e/cept under very e/treme conditions. 8veryone on the development team "nows e/actly what they are creating. There are development teams, usually ones creating so tware or less-critical applications, who produce speci ications on coc"tail nap"ins, i they create them at all. This has the distinct advantage o being very le/ible, but there's lots o ris" that not everyone is 3on the same page.3 2nd, what the product inally becomes isn't "nown until it's released. S'he"ules 2 "ey part o a so tware product is its schedule. 2s a pro:ect grows in size and comple/ity, with many pieces and many people contributing to the product, it becomes necessary to have some mechanism to trac" its progress. This could range rom simple tas" lists to Fantt charts =see ?igure 4.4> to detailed trac"ing o every minute tas" with pro:ect management so tware. 2igure 7>7> 5 *antt 'hart is a bar 'hart that shows a pro?e't+s tasks against a horiBontal timeline>

The goals o scheduling are to "now which wor" has been completed, how much wor" is still le t to do, and when it will all be inished. Software %esign %o'uments .ne common misconception is that when a programmer creates a program, he simply sits down and starts writing code. That may happen in some small, in ormal so tware shops, but or anything other than the smallest programs, there must be a design process to plan out how the so tware will be written. Thin" about this boo", which re)uired an outline be ore the irst

14

words were typed, or a building, which has blueprints drawn be ore the irst concrete is poured. The same planning should happen with so tware. The documents that programmers create vary greatly depending on the company, the pro:ect, and the team, but their purpose is to plan and organize the code that is to be written. #ere is a list o a ew common so tware design documents9 2rchitecture. 2 document that describes the overall design o the so tware, including descriptions o all the ma:or pieces and how they interact with each other. ,ata ?low ,iagram. 2 ormalized diagram that shows how data moves through a program. It's sometimes re erred to as a bubble chart because it's drawn by using circles and lines. &tate Transition ,iagram. 2nother ormalized diagram that brea"s the so tware into basic states, or conditions, and shows the means or moving rom one state to the ne/t. ?lowchart. The traditional means or pictorially describing a program's logic. ?lowcharting isn't very popular today, but when it's used, writing the program code rom a detailed lowchart is a very simple process. +ommented +ode. There's an old saying that you may write code once, but it will be read by someone at least 1< times. 7roperly embedding use ul comments in the so tware code itsel is e/tremely important, so that programmers assigned to maintain the code can more easily igure out what it does and how.

Test %o'uments Test documentation is discussed in detail in +hapters 174< but is mentioned here because it's integral to what ma"es up a so tware product. ?or the same reasons that programmers must plan and document their wor", so tware testers must as well. It's not unheard o or a so tware test team to create more deliverables than the programmers. #ere's a list o the more important test deliverables9 The test plan "es'ribes the o3erall metho" to be used to veri y that the so tware meets the product speci ication and the customer's needs. It includes the )uality ob:ectives, resource needs, schedules, assignments, methods, and so orth. Test cases list the speci ic items that will be tested and describe the detailed steps that will be ollowed to veri y the so tware. (ug reports describe the problems ound as the test cases are ollowed. These could be done on paper but are o ten trac"ed in a database. Test tools and automation are described in detail in +hapter 1G, 32utomated Testing and Test Tools.3 I your team is using automated methods to test your so tware, the tools you use, either purchased or written in-house, must be documented. !etrics, statistics, and summaries convey the progress being made as the test wor" progresses. They ta"e the orm o graphs, charts, and written reports.

15

#hat Parts )ake <p a Software Pro"u't$ &o ar in this chapter you've learned about the e ort that goes into creating a so tware product. It's also important to realize that when the product is ready to be bo/ed up and shipped out the door, it's not :ust the code that gets delivered. *umerous supporting parts go along with it =see ?igure 4.;>. &ince all these parts are seen or used by the customer, they need to be tested too. 2igure 7>D> The software C%-R&) is ?ust one of the many pie'es that make up a software pro"u't>

It's un ortunate, but these components are o ten overloo"ed in the testing process. Bou've surely attempted to use a product's built-in help ile and ound it to be not so help ul or worse :ust plain wrong. .r, maybe you've chec"ed the system re)uirements on a stic"er on the side o a so tware bo/ only to ind out a ter you bought it that the so tware didn't wor" on your 7+. These seem li"e simple things to test, but no one probably even gave them a second loo" be ore the product was ."ayed or release. Bou will. 0ater in this boo" you'll learn about these non-so tware pieces and how to properly test them. $ntil then, "eep this list in mind as :ust a sampling o what more there is to a so tware product than :ust the code9

#elp iles

$ser's manual

&amples and e/amples 0abels and stic"ers

16

7roduct support in o 8rror messages &etup and installation

Icons and art 2ds and mar"eting material -eadme ile

%&4+T 2&R*ET T& TEST ERR&R )ESS5*ES 8rror messages are one o the most overloo"ed parts o a so tware product. 7rogrammers, not trained writers, typically write them. They're seldom planned or and are usually hac"ed in while i/ing bugs. It's also very di icult or testers to ind and display all o them. ,on't let error messages such as these creep into your so tware9 [View full width] Error !e"#o$rd %ot fou%d. &re'' (1 to )o%ti%ue. *$%+t i%'t$%ti$te the ,ideo thi%-. .i%dow' h$' fou%d $% u%/%ow% de,i)e $%d i' i%'t$lli%- $ dri,er for it. 0 ($t$l E1)e2tio% 336 h$' o))urred $t 3333 3333337.

Software Pro?e't Staff *ow that you "now what goes into a so tware product and what ships with one, it's time to learn about all the people who create so tware. . course, this varies a great deal based on the company and the pro:ect, but or the most part the roles are the same, it's :ust the titles that are di erent. The ollowing lists, in no particular order, the ma:or players and what they do. The most common names are given, but e/pect variations and additions9 7ro:ect managers, program managers, or producers drive the pro:ect rom beginning to end. They're usually responsible or writing the product spec, managing the schedule, and ma"ing the critical decisions and trade-o s. 2rchitects or system engineers are the technical e/perts on the product team. They're usually very e/perienced and there ore are )uali ied to design the overall systems architecture or design or the so tware. They wor" very closely with the programmers. 7rogrammers, developers, or coders design and write so tware and i/ the bugs that are ound. They wor" closely with the architects and pro:ect managers to create the so tware. Then, they wor" closely with the pro:ect managers and testers to get the bugs i/ed. Testers or M2 =Muality 2ssurance> &ta is responsible or inding and reporting problems in the so tware product. They wor" very closely with all members o the team as they develop and run their tests, and report the problems they ind. +hapter 41, 3&o tware Muality 2ssurance,3 thoroughly covers the di erences between so tware testing and so tware )uality assurance tas"s. Technical writers, user assistance, user education, manual writers, or illustrators create the paper and online documentation that comes with a so tware product.

17

+on iguration management or builder handles the process o pulling together all the so tware written by the programmers and all the documentation created by the writers and putting it together into a single pac"age.

2s you can see, several groups o people contribute to a so tware product. .n large teams there may be dozens or hundreds wor"ing together. To success ully communicate and organize their approach, they need a plan, a method or getting rom point 2 to point (. That's what the ne/t section is about

Software %e3elopment ,ife'y'le )o"els 2 running :o"e in the computer industry is that three things should never be seen in the process o being created9 laws, sausage, and so tware. Their creation process is so messy and disgusting that it's best to :ust wait and see the inal result. That may or may not be totally true, but with most old sayings, there is a grain o truth behind the words. &ome so tware is developed with the rigor and discipline o a ine cra tsman, some so tware with tightly controlled chaos, and other so tware is stuc" together with duct tape and chewing gum. $sually, in the end, it's apparent to the customer what process was used. The process used to create a so tware product rom its initial conception to its public release is "nown as the so tware development li ecycle model. 2s discussed previously, there are many di erent methods that can be used or developing so tware, and no model is necessarily the best or a particular pro:ect. There are our re)uently used models, with most others :ust variations o these9 (ig-(ang +ode-and-?i/ %ater all &piral

8ach model has its advantages and disadvantages. 2s a tester, you will li"ely encounter them all and will need to tailor your test approach to it the model being used or your current pro:ect. -e er to these model descriptions as you read the rest o this boo" and thin" about how you would apply the various testing techni)ues you learn under each o them. Big-Bang )o"el .ne theory o the creation o the universe is the big-bang theory. It states that billions o years ago, the universe was created in a single huge e/plosion o nearly in inite energy. 8verything that e/ists is the result o energy and matter lining up to produce this boo", ,6,s, and (ill Fates. I the atoms didn't line up :ust right, these things might all be :ust )uivering masses o goop. The big-bang model or so tware development shown in ?igure 4.4 ollows much the same principle. 2 huge amount o matter =people and money> is put together, a lot o energy is e/pended o ten violently and out comes the per ect so tware productPor it doesn't.

2igure 7>0> The big-bang mo"el is by far the simplest metho" of software "e3elopment>

18

The beauty o the big-bang method is that it's simple. There is little i any planning, scheduling, or ormal development process. 2ll the e ort is spent developing the so tware and writing the code. It's a process that is used i the product re)uirements aren't well understood and the inal release date is completely le/ible. It's also important to have very le/ible customers, too, because they won't "now what they're getting until the very end. *otice that testing isn't shown in ?igure 4.4. In most cases, there is little to no ormal testing done under the big-bang model. I testing does occur, it's s)ueezed in :ust be ore the product is released. It's a mystery why testing is sometimes inserted into this model, but it's probably to ma"e everyone eel good that some testing was per ormed. I you are called in to test a product under the big-bang model, you have both an easy and a di icult tas". (ecause the so tware is already complete, you have the per ect speci ication the product itsel . 2nd, because it's impossible to go bac" and i/ things that are bro"en, your :ob is really :ust to report what you ind so the customers can be told about the problems. The downside is that, in the eyes o pro:ect management, the product is ready to go, so your wor" is holding up delivery to the customer. The longer you ta"e to do your :ob and the more bugs you ind, the more contentious the situation will become. Try to stay away rom testing in this model. Co"e-an"-2i( )o"el The code-and- i/ model shown in ?igure 4.G is usually the one that pro:ect teams all into by de ault i they don't consciously attempt to use something else. It's a step up, procedurally, rom the big-bang model, in that it at least re)uires some idea o what the product re)uirements are.

2igure 7>1> The 'o"e-an"-fi( mo"el repeats until someone gi3es up>

19

2 wise man once said, 3There's never time to do it right, but there's always time to do it over.3 That pretty much sums up this model. 2 team using this approach usually starts with a rough idea o what they want, does some simple design, and then proceeds into a long repeating cycle o coding, testing, and i/ing bugs. 2t some point they decide that enough is enough and release the product. 2s there's very little overhead or planning and documenting, a pro:ect team can show results immediately. ?or this reason, the code-and- i/ model wor"s very well or small pro:ects intended to be created )uic"ly and then thrown out shortly a ter they're done, such as prototypes and demos. 8ven so, code-and- i/ has been used on many large and well-"nown so tware products. I your word processor or spreadsheet so tware has lots o little bugs or it :ust doesn't seem )uite inished, it was li"ely created with the code-and- i/ model. 0i"e the big-bang model, testing isn't speci ically called out in the code-and- i/ model but does play a signi icant role between the coding and the i/ing. 2s a tester on a code-and- i/ pro:ect, you need to be aware that you, along with the programmers, will be in a constant state o cycling. 2s o ten as every day you'll be given new or updated releases o the so tware and will set o to test it. Bou'll run your tests, report the bugs, and then get a new so tware release. Bou may not have inished testing the previous release when the new one arrives, and the new one may have new or changed eatures. 8ventually, you'll get a chance to test most o the eatures, ind ewer and ewer bugs, and then someone =or the schedule> will decide that it's time to release the product. Bou will most li"ely encounter the code-and- i/ model during your wor" as a so tware tester. It's a good introduction to so tware development and will help you appreciate the more ormal methods. #aterfall )o"el The water all method is usually the irst one taught in programming school. It's been around orever. It's simple, elegant, and ma"es sense. 2nd, it can wor" well on the right pro:ect. ?igure 4.5 shows the steps involved in this model.

2igure 7>E> The software "e3elopment pro'ess flows from one step to the ne(t in the waterfall mo"el>

20

2 pro:ect using the water all model moves down a series o steps starting rom an initial idea to a inal product. 2t the end o each step, the pro:ect team holds a review to determine i they're ready to move to the ne/t step. I the pro:ect isn't ready to progress, it stays at that level until it's ready. *otice three important things about the water all method9 There's a large emphasis on speci ying what the product will be. *ote that the development or coding phase is only a single bloc"N The steps are discreteE there's no overlap. There's no way to bac" up. 2s soon as you're on a step, you need to complete the tas"s or that step and then move on you can't go bac".Q1R
Q1R

6ariations o the water all model loosen the rules a bit, allowing some overlap o the steps and the ability to bac" up one step i necessary. This may sound very limiting, and it is, but it wor"s well or pro:ects with a well-understood product de inition and a disciplined development sta . The goal is to wor" out all the un"nowns and nail down all the details be ore the irst line o code is written. The drawbac" is that in today's ast moving culture, with products being developed on Internet time, by the time a so tware product is so care ully thought out and de ined, the original reason or its being may have changed. ?rom a testing perspective, the water all model o ers one huge advantage over the other models presented so ar. 8verything is care ully and thoroughly speci ied. (y the time the so tware is delivered to the test group, every detail has been decided on, written down, and turned into so tware. ?rom that, the test group can create an accurate plan and schedule. They "now e/actly what they're testing, and there's no )uestion about whether something is a eature or a bug. (ut, with this advantage, comes a large disadvantage. (ecause testing occurs only at the end, a undamental problem could creep in early on and not be detected until days be ore the scheduled product release. -emember rom +hapter 1, 3&o tware Testing (ac"ground,3 how the

21

cost o bugs increases over timeI %hat's needed is a model that olds the testing tas"s in earlier to ind problems be ore they become too costly.

Spiral )o"el It's not )uite utopia, but the spiral model =see ?igure 4.7> goes a long way in addressing many o the problems inherent with the other models while adding a ew o its own nice touches. 2igure7>9> The spiral mo"el starts small an" gra"ually e(pan"s as the pro?e't be'omes better "efine" an" gains stability>

The spiral model was introduced by (arry (oehm in 19A5 in his 2ssociation or +omputing !achinery =2+!> paper, 32 &piral !odel o &o tware ,evelopment and 8nhancement.3 It's used airly o ten and has proven to be an e ective approach to developing so tware. The general idea behind the spiral model is that you don't de ine everything in detail at the very beginning. Bou start small, de ine your important eatures, try them out, get eedbac" rom your customers, and then move on to the ne/t level. Bou repeat this until you have your inal product. 8ach time around the spiral involves si/ steps9 1. 4. ;. 4. G. 5. ,etermine ob:ectives, alternatives, and constraints. Identi y and resolve ris"s. 8valuate alternatives. ,evelop and test the current level. 7lan the ne/t level. ,ecide on the approach or the ne/t level.

22

(uilt into the spiral model is a bit o water all =the steps o analysis, design, develop, test>, a bit o code-and- i/ =each time around the spiral>, and a bit o big-bang =loo" at it rom the outside>. +ouple this with the lower costs o inding problems early, and you have a pretty good development model. I you're a tester, you'll li"e this model. Bou'll get a chance to in luence the product early by being involved in the preliminary design phases. Bou'll see where the pro:ect has come rom and where it's going. 2nd, at the very end o the pro:ect, you won't eel as rushed to per orm all your testing at the last minute. Bou've been testing all along, so the last push should only be a validation that everything is o"ay.

5*!,E S&2T#5RE %E:E,&P)E4T 2 type o development process that has gained in popularity among a number o so tware companies is "nown as 2gile &o tware ,evelopment. Bou might hear other names or it such as -apid 7rototyping, 8/treme 7rogramming, or 8volutionary ,evelopment, among others. The goal o 2gile &o tware ,evelopment, as described in its mani esto at www.agilemani esto.org, is9

3Individuals and interactions over processes and tools %or"ing so tware over comprehensive documentation +ustomer collaboration over contract negotiations -esponding to change over ollowing a planP That is, while there is value in the items on the right, we value the items on the le t more.3 2gile &o tware ,evelopment has developed a bit o a ollowing and has had some success ul pro:ects but is ar rom main stream. It sounds li"e a great philosophy i you and your pro:ect seem to be bogged down in process and documentation but, in my opinion, it can too easily digress into chaos. It's a method to watch and learn about and your team may consider using it. ?or now, however, I won't be riding in a :et whose so tware was rapidly prototyped by an e/treme programmer. !aybe someday. ?or more in ormation, chec" out9 2gile !odeling, Teach Boursel 8/treme 7rogramming in 44 #ours, and 8/treme 7rogramming 7ractices in 2ction. 2ll are published by &ams 7ublishing.

Summary Bou now have an understanding o how so tware products are created both what goes into them and the processes used to put them together. 2s you can see, there's no de initive approach. The our models presented here are :ust e/amples. There are many others and lots o variations o these. 8ach company, each pro:ect, and each team will choose what wor"s or them. &ometimes they will choose right, sometimes they will choose wrong. Bour :ob as a so tware tester is to wor" the best you can in the development model you're in, applying the testing s"ills you learn in the rest o this boo" to create the best so tware possible. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answers but don't pee"N 1. *ame several tas"s that should be per ormed be ore a programmer starts writing the irst line o code. 4. %hat disadvantage is there to having a ormal, loc"ed-down speci icationI ;. %hat is the best eature o the big-bang model o so tware developmentI

23

4. %hen using the code-and- i/ model, how do you "now when the so tware is ready to releaseI G. %hy can the water all method be di icult to useI 5. %hy would a so tware tester li"e the spiral model better than the othersI

Chapter 3. The I* T#I& +#27T8

ealities of Software Testing

Testing 2/ioms &o tware Testing Terms and ,e initions

In +hapter 1, 3&o tware Testing (ac"ground,3 and +hapter 4, 3The &o tware ,evelopment 7rocess,3 you learned about the basics o so tware testing and the so tware development process. The in ormation presented in these chapters o ered a very high-level and arguably idealistic view o how so tware pro:ects might be run. $n ortunately, in the real world you will never see a pro:ect per ectly ollow any o the development models. Bou will never be given a thoroughly detailed speci ication that per ectly meets the customer's needs and you will never have enough time to do all the testing you need to do. It :ust doesn't happen. (ut, to be an e ective so tware tester, you need to understand what the ideal process is so that you have something to aim or. The goal o this chapter is to temper that idealism with a reality chec" rom a so tware tester's perspective. It will help you see that, in practice, trade-o s and concessions must be made throughout the development cycle. !any o those trade-o s are directly related to the so tware test e ort. The bugs you ind and the problems you prevent all signi icantly a ect the pro:ect. 2 ter reading this chapter, you'll have a much clearer picture o the roles, the impact, and the responsibilities that so tware testing has and you'll hope ully appreciate the behindthe-scenes decisions that must be made to create a so tware product. The highlights o this chapter include %hy so tware can never be per ect %hy so tware testing isn't :ust a technical problem The terms commonly used by so tware testers

Testing 5(ioms This irst section o this chapter is a list o a/ioms, or truisms. Thin" o them as the 3rules o the road3 or the 3 acts o li e3 or so tware testing and so tware development. 8ach o them is a little tidbit o "nowledge that helps put some aspect o the overall process into perspective. !t+s !mpossible to Test a Program Completely 2s a new tester, you might believe that you can approach a piece o so tware, ully test it, ind all the bugs, and assure that the so tware is per ect. $n ortunately, this isn't possible, even with the simplest programs, due to our "ey reasons9 The number o possible inputs is very large.

24

The number o possible outputs is very large. The number o paths through the so tware is very large. The so tware speci ication is sub:ective. Bou might say that a bug is in the eye o the beholder.

!ultiply all these 3very large3 possibilities together and you get a set o test conditions that's too large to attempt. I you don't believe it, consider the e/ample shown in ?igure ;.1, the !icroso t %indows +alculator. 2igure D>1> E3en a simple program su'h as the #in"ows Cal'ulator is too 'omple( to 'ompletely test>

2ssume that you are assigned to test the %indows +alculator. Bou decide to start with addition. Bou try 1435. Bou get an answer o 1. That's correct. Then you try 1415. Bou get 2. #ow ar do you goI The calculator accepts a ;4-digit number, so you must try all the possibilities up to 14999999999999999999999999999999995 .nce you complete that series, you can move on to 2435, 2415, 2425, and so on. 8ventually you'll get to 999999999999999999999999999999994999999999999999999999999999999995 *e/t you should try all the decimal values9 1.343.1, 1.343.2, and so on. .nce you veri y that regular numbers sum properly, you need to attempt illegal inputs to assure that they're properly handled. -emember, you're not limited to clic"ing the numbers on screen you can press "eys on your computer "eyboard, too. Food values to try might be 14$, 641, 1$142#2,P. There are literally billions upon billions o these. 8dited inputs must also be tested. The %indows +alculator allows the 7$)/'2$)e and 8elete "eys, so you should try them. 1242 should e)ual 4. 8verything you've tested so ar must be retested by pressing the (ac"space "ey or each entry, or each two entries, and so on. I you or your heirs manage to complete all these cases, you can then move on to adding three numbers, then our numbers,P. There are so many possible entries that you could never complete them, even i you used a supercomputer to eed in the numbers. 2nd that's only or addition. Bou still have subtraction, multiplication, division, s)uare root, percentage, and inverse to cover.

25

The point o this e/ample is to demonstrate that it's impossible to completely test a program, even so tware as simple as a calculator. I you decide to eliminate any o the test conditions because you eel they're redundant or unnecessary, or :ust to save time, you've decided not to test the program completely. Software Testing !s a Risk-Base" E(er'ise I you decide not to test every possible test scenario, you've chosen to ta"e on ris". In the calculator e/ample, what i you choose not to test that 13244132452348I It's possible the programmer accidentally le t in a bug or that situation. I you don't test it, a customer will eventually enter it, and he or she will discover the bug. It'll be a costly bug, too, since it wasn't ound until the so tware was in the customer's hands. This may all sound pretty scary. Bou can't test everything, and i you don't, you will li"ely miss bugs. The product has to be released, so you will need to stop testing, but i you stop too soon, there will still be areas untested. %hat do you doI .ne "ey concept that so tware testers need to learn is how to reduce the huge domain o possible tests into a manageable set, and how to ma"e wise ris"-based decisions on what's important to test and what's not. ?igure ;.4 shows the relationship between the amount o testing per ormed and the number o bugs ound. I you attempt to test everything, the costs go up dramatically and the number o missed bugs declines to the point that it's no longer cost e ective to continue. I you cut the testing short or ma"e poor decisions o what to test, the costs are low but you'll miss a lot o bugs. The goal is to hit that optimal amount o testing so that you don't test too much or too little. 2igure D>7> E3ery software pro?e't has an optimal test effort>

Bou will learn how to design and select test scenarios that minimize ris" and optimize your testing in +hapters 4 through 7. Testing Can+t Show That Bugs %on+t E(ist

26

Thin" about this or a moment. Bou're an e/terminator charged with e/amining a house or bugs. Bou inspect the house and ind evidence o bugs maybe live bugs, dead bugs, or nests. Bou can sa ely say that the house has bugs. Bou visit another house. This time you ind no evidence o bugs. Bou loo" in all the obvious places and see no signs o an in estation. !aybe you ind a ew dead bugs or old nests but you see nothing that tells you that live bugs e/ist. +an you absolutely, positively state that the house is bug reeI *ope. 2ll you can conclude is that in your search you didn't ind any live bugs. $nless you completely dismantled the house down to the oundation, you can't be sure that you didn't simply :ust miss them. &o tware testing wor"s e/actly as the e/terminator does. It can show that bugs e/ist, but it can't show that bugs don't e/ist. Bou can per orm your tests, ind and report bugs, but at no point can you guarantee that there are no longer any bugs to ind. Bou can only continue your testing and possibly ind more. The )ore Bugs 6ou 2in". the )ore Bugs There 5re There are even more similarities between real bugs and so tware bugs. (oth types tend to come in groups. I you see one, odds are there will be more nearby. ?re)uently, a tester will go or long spells without inding a bug. #e'll then ind one bug, then )uic"ly another and another. There are several reasons or this9 7rogrammers have bad days. 0i"e all o us, programmers can have o days. +ode written one day may be per ectE code written another may be sloppy. .ne bug can be a tell-tale sign that there are more nearby. 7rogrammers o ten ma"e the same mista"e. 8veryone has habits. 2 programmer who is prone to a certain error will o ten repeat it. &ome bugs are really :ust the tip o the iceberg. 6ery o ten the so tware's design or architecture has a undamental problem. 2 tester will ind several bugs that at irst may seem unrelated but eventually are discovered to have one primary serious cause.

It's important to note that the inverse o this 3bugs ollow bugs3 idea is true, as well. I you ail to ind bugs no matter how hard you try, it may very well be that the eature you're testing was cleanly written and that there are indeed ew i any bugs to be ound.

The Pesti'i"e Para"o( In 199<, (oris (eizer, in his boo" &o tware Testing Techni)ues, &econd 8dition, coined the term pesticide parado/ to describe the phenomenon that the more you test so tware, the more immune it becomes to your tests. The same thing happens to insects with pesticides =see ?igure ;.;>. I you "eep applying the same pesticide, the insects eventually build up resistance and the pesticide no longer wor"s. 2igure D>D> Software un"ergoing the same repetiti3e tests e3entually buil"s up resistan'e to them>

27

-emember the spiral model o so tware development described in +hapter 4I The test process repeats each time around the loop. %ith each iteration, the so tware testers receive the so tware or testing and run their tests. 8ventually, a ter several passes, all the bugs that those tests would ind are e/posed. +ontinuing to run them won't reveal anything new. To overcome the pesticide parado/, so tware testers must continually write new and di erent tests to e/ercise di erent parts o the program and ind more bugs. 4ot 5ll the Bugs 6ou 2in" #ill Be 2i(e" .ne o the sad realities o so tware testing is that even a ter all your hard wor", not every bug you ind will be i/ed. *ow, don't be disappointed this doesn't mean that you've ailed in achieving your goal as a so tware tester, nor does it mean that you or your team will release a poor )uality product. It does mean, however, that you'll need to rely on a couple o those traits o a so tware tester listed in +hapter 1e/ercising good :udgment and "nowing when per ection isn't reasonably attainable. Bou and your team will need to ma"e trade-o s, ris"-based decisions or each and every bug, deciding which ones will be i/ed and which ones won't. There are several reasons why you might choose not to i/ a bug9 There's not enough time. In every pro:ect there are always too many so tware eatures, too ew people to code and test them, and not enough room le t in the schedule to inish. I you're wor"ing on a ta/ preparation program, 2pril 1G isn't going to moveyou must have your so tware ready in time. It's really not a bug. !aybe you've heard the phrase, 3It's not a bug, it's a eatureN3 It's not uncommon or misunderstandings, test errors, or spec changes to result in wouldbe bugs being dismissed as eatures. It's too ris"y to i/. $n ortunately, this is all too o ten true. &o tware can be ragile, intertwined, and sometimes li"e spaghetti. Bou might ma"e a bug i/ that causes other bugs to appear. $nder the pressure to release a product under a tight schedule, it might be too ris"y to change the so tware. It may be better to leave in the "nown bug to avoid the ris" o creating new, un"nown ones. It's :ust not worth it. This may sound harsh, but it's reality. (ugs that would occur in re)uently or bugs that appear in little-used eatures may be dismissed. (ugs that have wor"-arounds, ways that a user can prevent or avoid the bug, are o ten not i/ed. It all comes down to a business decision based on ris".

28

The decision-ma"ing process usually involves the so tware testers, the pro:ect managers, and the programmers. 8ach carries a uni)ue perspective on the bugs and has his own in ormation and opinions as to why they should or shouldn't be i/ed. In +hapter 19, 3-eporting %hat Bou ?ind,3 you'll learn more about reporting bugs and getting your voice heard. #=5T =5PPE4S #=E4 6&< )5-E T=E #R&4* %EC!S!&4$ -emember the Intel 7entium bug described in +hapter 1I The Intel test engineers ound this bug be ore the chip was released, but the product team decided that it was such a small, rare bug that it wasn't worth i/ing. They were under a tight schedule and decided to meet their current deadline and i/ the bug in later releases o the chip. $n ortunately, the bug was discovered and the rest, they say, is history. In any piece o so tware, or every 37entium type3 bug that ma"es the headlines, there could be perhaps hundreds o bugs that are le t un i/ed because they were perceived to not have a ma:or negative e ect. .nly time can tell i those decisions were right or wrong.

#hen a Bug+s a Bug !s %iffi'ult to Say I there's a problem in the so tware but no one ever discovers itnot programmers, not testers, and not even a single customeris it a bugI Fet a group o so tware testers in a room and as" them this )uestion. Bou'll be in or a lively discussion. 8veryone has their own opinion and can be pretty vocal about it. The problem is that there's no de initive answer. The answer is based on what you and your development team decide wor"s best or you. ?or the purposes o this boo", re er bac" to the rules to de ine a bug rom +hapter 19 1. The so tware doesn't do something that the product speci ication says it should do. 4. The so tware does something that the product speci ication says it shouldn't do. ;. The so tware does something that the product speci ication doesn't mention. 4. The so tware doesn't do something that the product speci ication doesn't mention but should. G. The so tware is di icult to understand, hard to use, slow, orin the so tware tester's eyeswill be viewed by the end user as :ust plain not right. ?ollowing these rules helps clari y the dilemma by ma"ing a bug a bug only i it's observed. To claim that the so tware does or doesn't do 3something3 implies that the so tware was run and that 3something3 or the lac" o 3something3 was witnessed. &ince you can't report on what you didn't see, you can't claim that a bug e/ists i you didn't see it. #ere's another way to thin" o it. It's not uncommon or two people to have completely di erent opinions on the )uality o a so tware product. .ne may say that the program is incredibly buggy and the other may say that it's per ect. #ow can both be rightI The answer is that one has used the product in a way that reveals lots o bugs. The other hasn't. *.T8 (ugs that are undiscovered or haven't yet been observed are o ten re erred to as latent bugs. I this is as clear as mud, don't worry. ,iscuss it with your peers in so tware testing and ind out what they thin". 0isten to others' opinions, test their ideas, and orm your own de inition.

29

-emember the old )uestion, 3I a tree alls in the orest and there's no one there to hear it, does it ma"e a soundI3

Pro"u't Spe'ifi'ations 5re 4e3er 2inal &o tware developers have a problem. The industry is moving so ast that last year's cuttingedge products are obsolete this year. 2t the same time, so tware is getting larger and gaining more eatures and comple/ity, resulting in longer and longer development schedules. These two opposing orces result in con lict, and the result is a constantly changing product speci ication. There's no other way to respond to the rapid changes. 2ssume that your product had a loc"eddown, inal, absolutely-can't-change-it product spec. Bou're hal way through the planned twoyear development cycle, and your main competitor releases a product very similar to yours but with several desirable eatures that your product doesn't have. ,o you continue with your spec as is and release an in erior product in another yearI .r, does your team regroup, rethin" the product's eatures, rewrite the product spec, and wor" on a revised productI In most cases, wise business dictates the latter. 2s a so tware tester, you must assume that the spec will change. ?eatures will be added that you didn't plan to test. ?eatures will be changed or even deleted that you had already tested and reported bugs on. It will happen. Bou'll learn techni)ues or being le/ible in your test planning and test e/ecution in the remainder o this boo". Software Testers 5ren+t the )ost Popular )embers of a Pro?e't Team -emember the goal o a so tware testerI The goal o a so tware tester is to ind bugs, ind them as early as possible, and ma"e sure they get i/ed. Bour :ob is to inspect and criti)ue your peer's wor", ind problems with it, and publicize what you've ound. .uchN Bou won't win a popularity contest doing this :ob. #ere are a couple o tips to "eep the peace with your ellow teammates9 ?ind bugs early. That's your :ob, o course, but wor" hard at doing this. It's much less o an impact and much more appreciated i you ind a serious bug three months be ore, rather than one day be ore, a product's scheduled release. Temper your enthusiasm. ."ay, you really love your :ob. Bou get really e/cited when you ind a terrible bug. (ut, i you bounce into a programmer's cubicle with a huge grin on your ace and tell her that you :ust ound the nastiest bug o your career and it's in her code, she won't be happy. ,on't :ust report bad news. I you ind a piece o code surprisingly bug ree, tell the world. 7op into a programmer's cubicle occasionally :ust to chat. I all you ever do is report bad news, people will see you coming and will run and hide.

Software Testing !s a %is'ipline" Te'hni'al Profession It used to be that so tware testing was an a terthought. &o tware products were small and not very complicated. The number o people with computers using so tware was limited. 2nd, the ew programmers on a pro:ect team could ta"e turns debugging each others' code. (ugs weren't that much o a problem. The ones that did occur were easily i/ed without much cost or disruption. I so tware testers were used, they were re)uently untrained and brought into the

30

pro:ect late to do some 3ad-hoc banging on the code to see what they might ind.3 Times have changed. 0oo" at the so tware help-wanted ads and you'll see numerous listings or so tware testers. The so tware industry has progressed to the point where pro essional so tware testers are mandatory. It's now too costly to build bad so tware. To be air, not every company is on board yet. !any computer game and small-time so tware companies still use a airly loose development modelusually big-bang or code-and- i/. (ut much more so tware is now developed with a disciplined approach that has so tware testers as core, vital members o their sta . This is great news i you're interested in so tware testing. It can now be a career choicea :ob that re)uires training and discipline, and allows or advancement. Software Testing Terms an" %efinitions This chapter wraps up the irst section o this boo" with a list o so tware testing terms and their de initions. These terms describe undamental concepts regarding the so tware development process and so tware testing. (ecause they're o ten con used or used inappropriately, they're de ined here as pairs to help you understand their true meanings and the di erences between them. (e aware that there is little agreement in the so tware industry over the de inition o many, seemingly common, terms. 2s a tester, you should re)uently clari y the meaning o the terms your team is using. It's o ten best to agree to a de inition rather than ight or a 3correct3 one. Pre'ision an" 5''ura'y 2s a so tware tester, it's important to "now the di erence between precision and accuracy. &uppose that you're testing a calculator. &hould you test that the answers it returns are precise or accurateI (othI I the pro:ect schedule orced you to ma"e a ris"-based decision to ocus on only one o these, which one would you chooseI %hat i the so tware you're testing is a simulation game such as baseball or a light simulatorI &hould you primarily test its precision or its accuracyI ?igure ;.4 helps to graphically describe these two terms. The goal o this dart game is to hit the bull's-eye in the center o the board. The darts on the board in the upper le t are neither precise nor accurate. They aren't closely grouped and not even close to the center o the target. 2igure D>0> %arts on a "artboar" "emonstrate the "ifferen'e between pre'ision an" a''ura'y>

31

The board on the upper right shows darts that are precise but not accurate. They are closely grouped, so the thrower has precision, but he's not very accurate because the darts didn't even hit the board. The board on the lower le t is an e/ample o accuracy but poor precision. The darts are very close to the center, so the thrower is getting close to what he's aiming at, but because they aren't closely positioned, the precision is o . The board in the lower right is a per ect match o precision and accuracy. The darts are closely grouped and on target. %hether the so tware you test needs to be precise or accurate depends much on what the product is and ultimately what the development team is aiming at =e/cuse the pun>. 2 so tware calculator li"ely demands that both are achieveda right answer is a right answer. (ut, it may be decided that calculations will only be accurate and precise to the i th decimal place. 2 ter that, the precision can vary. 2s long as the testers are aware o that speci ication, they can tailor their testing to con irm it. :erifi'ation an" :ali"ation 6eri ication and validation are o ten used interchangeably but have di erent de initions. These di erences are important to so tware testing. 6eri ication is the process con irming that somethingso twaremeets its speci ication. 6alidation is the process con irming that it meets the user's re)uirements. These may sound very similar, but an e/planation o the #ubble space telescope problems will help show the di erence. In 2pril 199<, the #ubble space telescope was launched into orbit around the 8arth. 2s a re lective telescope, #ubble uses a large mirror as its primary means to magni y the ob:ects it's aiming at. The construction o the mirror was a huge underta"ing re)uiring e/treme precision and accuracy. Testing o the mirror was di icult since the telescope was designed or use in space and couldn't be positioned or even viewed through while it was still on 8arth. ?or this reason, the only means to test it was to care ully measure all its attributes and compare the measurements with what was speci ied. This testing was per ormed and #ubble was declared it or launch. $n ortunately, soon a ter it was put into operation, the images it returned were ound to be out o ocus. 2n investigation discovered that the mirror was improperly manu actured. The mirror was ground according to the speci ication, but the speci ication was wrong. The mirror was e/tremely precise, but it wasn't accurate. Testing had con irmed that the mirror met the specveri icationbut it didn't con irm that it met the original re)uirementvalidation.

32

In 199;, a space shuttle mission repaired the #ubble telescope by installing a 3corrective lens3 to re ocus the image generated by the improperly manu actured mirror. 2lthough this is a not a so tware e/ample, veri ication and validation apply e)ually well to so tware testing. *ever assume that the speci ication is correct. I you veri y the spec and validate the inal product, you help avoid problems such as the one that hit the #ubble telescope.

Auality an" Reliability !erriam-%ebster's +ollegiate ,ictionary de ines )uality as 3a degree o e/cellence3 or 3superiority in "ind.3 I a so tware product is o high )uality, it will meet the customer's needs. The customer will eel that the product is e/cellent and superior to his other choices. &o tware testers o ten all into the trap o believing that )uality and reliability are the same thing. They eel that i they can test a program until it's stable, dependable, and reliable, they are assuring a high-)uality product. $n ortunately, that isn't necessarily true. -eliability is :ust one aspect o )uality. 2 so tware user's idea o )uality may include the breadth o eatures, the ability o the product to run on his old 7+, the so tware company's phone support availability, and, o ten, the price o the product. -eliability, or how o ten the product crashes or trashes his data, may be important, but not always. To ensure that a program is o high )uality and is reliable, a so tware tester must both veri y and validate throughout the product development process. Testing an" Auality 5ssuran'e (A5 The last pair o de initions is testing and )uality assurance =sometimes shortened to M2>. These two terms are the ones most o ten used to describe either the group or the process that's veri ying and validating the so tware. In +hapter 41, 3&o tware Muality 2ssurance,3 you'll learn more about so tware )uality assurance, but or now, consider these de initions9 The goal o a so tware tester is to ind bugs, ind them as early as possible, and ma"e sure they get i/ed. 2 so tware )uality assurance person's main responsibility is to create and en orce standards and methods to improve the development process and to prevent bugs rom ever occurring.

. course, there is overlap. &ome testers will do a ew M2 tas"s and some M2-ers will per orm a bit o testing. The two :obs and their tas"s are intertwined. %hat's important is that you "now what your primary :ob responsibilities are and communicate that in ormation to the rest o the development team. +on usion among the team members about who's testing and who's not has caused lots o process pain in many pro:ects. Summary &ausages, laws, and so twarewatching them being made can be pretty messy. #ope ully the previous three chapters haven't scared you o . !any so tware testers have come into a pro:ect not "nowing what was happening around them, how decisions were being made, or what procedure they should be ollowing. It's impossible to be e ective that way. %ith the in ormation you've learned so ar about so tware testing and the so tware development process, you'll have a head start when you begin testing or the irst time. Bou'll "now what your role should be, or at least "now what )uestions to as" to ind your place in the big picture. ?or now, all the process stu is out o the way, and the ne/t chapter o this boo" begins a new section that will introduce you to the basic techni)ues o so tware testing.

33

AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. Fiven that it's impossible to test a program completely, what in ormation do you thin" should be considered when deciding whether it's time to stop testingI 2. &tart the %indows +alculator. Type 59333-55 =the comma is important>. 0oo" at the result. Is this a bugI %hy or why notI ;. I you were testing a simulation game such as a light simulator or a city simulator, what do you thin" would be more important to testits accuracy or its precisionI 4. Is it possible to have a high-)uality and low-reliability productI %hat might an e/ample beI G. %hy is it impossible to test a program completelyI 5. I you were testing a eature o your so tware on !onday and inding a new bug every hour, at what rate would you e/pect to ind bugs on TuesdayI

Chapter !. "#amining the Specification I* T#I& +#27T8 Fetting &tarted 7er orming a #igh-0evel -eview o the &peci ication 0ow-0evel &peci ication Test Techni)ues

This chapter will introduce you to your irst real hands-on testingbut it may not be what you e/pect. Bou won't be installing or running so tware and you won't be pounding on the "eyboard hoping or a crash. In this chapter, you'll learn how to test the product's speci ication to ind bugs be ore they ma"e it into the so tware. Testing the product spec isn't something that all so tware testers have the lu/ury o doing. &ometimes you might come into a pro:ect midway through the development cycle a ter the speci ication is written and the coding started. I that's the case, don't worryyou can still use the techni)ues presented here to test the inal speci ication. I you're ortunate enough to be involved on the pro:ect early and have access to a preliminary speci ication, this chapter is or you. ?inding bugs at this stage can potentially save your pro:ect huge amounts o time and money. #ighlights o this chapter include %hat is blac"-bo/ and white-bo/ testing #ow static and dynamic testing di er %hat high-level techni)ues can be used or reviewing a product speci ication %hat speci ic problems you should loo" or when reviewing a product speci ication in detail

34

*etting Starte" Thin" bac" to the our development models presented in +hapter 4, 3The &o tware ,evelopment 7rocess39 big-bang, code-and- i/, water all, and spiral. In each model, e/cept big-bang, the development team creates a product speci ication rom the re)uirements document to de ine what the so tware will become. Typically, the product speci ication is a written document using words and pictures to describe the intended product. 2n e/cerpt rom the %indows +alculator =see ?igure 4.1> product spec might read something li"e this9 The 8dit menu will have two selections9 +opy and 7aste. These can be chosen by one o three methods9 pointing and clic"ing to the menu items with the mouse, using access-"eys =2ltJ8 and then + or +opy and 7 or 7aste>, or using the standard %indows shortcut "eys o +trlJ+ or +opy and +trlJ6 or 7aste. The +opy unction will copy the current entry displayed in the number te/t bo/ into the %indows +lipboard. The 7aste unction will paste the value stored in the %indows +lipboard into the number te/t bo/.

2igure 0>1> The stan"ar" #in"ows Cal'ulator "isplaying the "rop-"own E"it menu>

2s you can see, it too" )uite a ew words :ust to describe the operation o two menu items in a simple calculator program. 2 thoroughly detailed spec or the entire application could be a hundred pages long. It may seem li"e over"ill to create a meticulous document or such simple so tware. %hy not :ust let a programmer write a calculator program on his ownI The problem is that you would have no idea what you'd eventually get. The programmer's idea o what it should loo" li"e, what unctionality it should have, and how the user would use it could be completely di erent rom yours. The only way to assure that the end product is what the customer re)uiredand to properly plan the test e ortis to thoroughly describe the product in a speci ication. The other advantage o having a detailed spec, and the basis o this chapter, is that as a tester you'll also have a document as a testable item. Bou can use it to ind bugs be ore the irst line o code is written. Bla'k-Bo( an" #hite-Bo( Testing Two terms that so tware testers use to describe how they approach their testing are blac"-bo/ testing and white-bo/ testing. ?igure 4.4 shows the di erence between the two approaches. In blac"-bo/ testing, the tester only "nows what the so tware is supposed to dohe can't loo" in the bo/ to see how it operates. I he types in a certain input, he gets a certain output. #e doesn't "now how or why it happens, :ust that it does.

35

2igure 0>7> #ith bla'k-bo( testing. the software tester "oesn+t know the "etails of how the software works>

TI7 (lac"-bo/ testing is sometimes re erred to as unctional testing or behavioral testing. ,on't get caught up in what the actual terms are, as your team may use something di erent. It's your :ob to understand their meaning and how they're used by your team. Thin" about the %indows +alculator shown in ?igure 4.1. I you type 3.14159 and press the s)rt button, you get 1.772453132341. %ith blac"-bo/ testing, it doesn't matter what gyrations the so tware goes through to compute the s)uare root o pi. It :ust does it. 2s a so tware tester, you can veri y the result on another 3certi ied3 calculator and determine i the %indows +alculator is unctioning correctly. In white-bo/ testing =sometimes called clear-bo/ testing>, the so tware tester has access to the program's code and can e/amine it or clues to help him with his testinghe can see inside the bo/. (ased on what he sees, the tester may determine that certain numbers are more or less li"ely to ail and can tailor his testing based on that in ormation. *.T8 There is a ris" to white-bo/ testing. It's very easy to become biased and ail to ob:ectively test the so tware because you might tailor the tests to match the code's operation. Stati' an" %ynami' Testing Two other terms used to describe how so tware is tested are static testing and dynamic testing. &tatic testing re ers to testing something that's not runninge/amining and reviewing it. ,ynamic testing is what you would normally thin" o as testingrunning and using the so tware. The best analogy or these terms is the process you go through when chec"ing out a used car. 1ic"ing the tires, chec"ing the paint, and loo"ing under the hood are static testing techni)ues. &tarting it up, listening to the engine, and driving down the road are dynamic testing techni)ues. Stati' Bla'k-Bo( Testing: Testing the Spe'ifi'ation Testing the speci ication is static blac"-bo/ testing. The speci ication is a document, not an e/ecuting program, so it's considered static. It's also something that was created using data rom many sourcesusability studies, ocus groups, mar"eting input, and so on. Bou don't necessarily need to "now how or why that in ormation was obtained or the details o the

36

process used to obtain it, :ust that it's been boiled down into a product speci ication. Bou can then ta"e that document, per orm static blac"-bo/ testing, and care ully e/amine it or bugs. 8arlier you saw an e/ample o a product speci ication or the %indows +alculator. This e/ample used a standard written document with a picture to describe the so tware's operation. 2lthough this is the most common method or writing a spec, there are lots o variations. Bour development team may emphasize diagrams over words or it may use a sel -documenting computer language such as 2da. %hatever their choice, you can still apply all the techni)ues presented in this chapter. Bou will have to tailor them to the spec ormat you have, but the ideas are still the same. %hat do you do i your pro:ect doesn't have a specI !aybe your team is using the big-bang model or a loose code-and- i/ model. 2s a tester, this is a di icult position. Bour goal is to ind bugs earlyideally inding them be ore the so tware is codedbut i your product doesn't have a spec, this may seem impossible to do. 2lthough the spec may not be written down, someone, or several people, "now what they're trying to build. It may be the developer, a pro:ect manager, or a mar"eter. $se them as the wal"ing, tal"ing, product spec and apply the same techni)ues or evaluating this 3mental3 speci ication as though it was written on paper. Bou can even ta"e this a step urther by recording the in ormation you gather and circulating it or review. Tell your pro:ect team, 3This is what I plan to test and submit bugs against.3 Bou'll be amazed at how many details they'll immediately ill in. TI7 Bou can test a speci ication with static blac"-bo/ techni)ues no matter what the ormat o the speci ication. It can be a written or graphical document or a combination o both. Bou can even test an unwritten speci ication by )uestioning the people who are designing and writing the so tware. Performing a =igh-,e3el Re3iew of the Spe'ifi'ation ,e ining a so tware product is a di icult process. The spec must deal with many un"nowns, ta"e a multitude o changing inputs, and attempt to pull them all together into a document that describes a new product. The process is an ine/act science and is prone to having problems. The irst step in testing the speci ication isn't to :ump in and loo" or speci ic bugs. The irst step is to stand bac" and view it rom a high level. 8/amine the spec or large undamental problems, oversights, and omissions. Bou might consider this more research than testing, but ultimately the research is a means to better understand what the so tware should do. I you have a better understanding o the whys and hows behind the spec, you'll be much better at e/amining it in detail. Preten" to Be the Customer The easiest thing or a tester to do when he irst receives a speci ication or review is to pretend to be the customer. ,o some research about who the customers will be. Tal" to your mar"eting or sales people to get hints on what they "now about the end user. I the product is an internal so tware pro:ect, ind out who will be using it and tal" to them. It's important to understand the customer's e/pectations. -emember that the de inition o )uality means 3meeting the customer's needs.3 2s a tester, you must understand those needs to test that the so tware meets them. To do this e ectively doesn't mean that you must be an e/pert in the ield o nuclear physics i you're testing so tware or a power plant, or that you must be a pro essional pilot i you're testing a light simulator. (ut, gaining some amiliarity with the ield the so tware applies to would be a great help. 2bove all else, assume nothing. I you review a portion o the spec and don't understand it, don't assume that it's correct and go on. 8ventually, you'll have to use this speci ication to design your so tware tests, so you'll eventually have to understand it. There's no better time to learn than now. I you ind bugs along the way =and you will>, all the better. TI7

37

,on't orget about so tware security when pretending to be the customer. Bour users will assume that the so tware is secure, but you can't assume that the programmers will handle security issues properly. It must be speci iedin detail. +hapter 1;, 3Testing or &o tware &ecurity,3 discusses what you should loo" or when reviewing the product design and speci ication or security issues. Resear'h E(isting Stan"ar"s an" *ui"elines (ac" in the days be ore !icroso t %indows and the 2pple !acintosh, nearly every so tware product had a di erent user inter ace. There were di erent colors, di erent menu structures, unlimited ways to open a ile, and myriad cryptic commands to get the same tas"s done. !oving rom one so tware product to another re)uired complete retraining. Than" ully, there has been an e ort to standardize the hardware and the so tware. There has also been e/tensive research done on how people use computers. The result is that we now have products reasonably similar in their loo" and eel that have been designed with ergonomics in mind. Bou may argue that the adopted standards and guidelines aren't per ect, that there may be better ways to get certain tas"s done, but e iciency has greatly improved because o this commonality. +hapter 11, 3$sability Testing,3 will cover this topic in more detail, but or now you should thin" about what standards and guidelines might apply to your product. *.T8 The di erence between standards and guidelines is a matter o degree. 2 standard is much more irm than a guideline. &tandards should be strictly adhered to i your team has decided that it's important to comply with them completely. Fuidelines are optional but should be ollowed. It's not uncommon or a team to decide to use a standard as a guidelineas long as everyone "nows that is the plan. #ere are several e/amples o standards and guidelines to consider. This list isn't de initive. Bou should research what might apply to your so tware9 +orporate Terminology and +onventions. I this so tware is tailored or a speci ic company, it should adopt the common terms and conventions used by the employees o that company. Industry -e)uirements. The medical, pharmaceutical, industrial, and inancial industries have very strict standards that their so tware must ollow. Fovernment &tandards. The government, especially the military, has strict standards. Fraphical $ser Inter ace =F$I>. I your so tware runs under !icroso t %indows or 2pple !acintosh operating systems, there are published standards and guidelines or how the so tware should loo" and eel to a user. &ecurity &tandards. Bour so tware and its inter aces and protocols may need to meet certain security standards or levels. It may also need to be independently certi ied that it does, indeed, meet the necessary criteria.

2s a tester, your :ob isn't to de ine what guidelines and standards should be applied to your so tware. That :ob lies with the pro:ect manager or whoever is writing the speci ication. Bou do, however, need to per orm your own investigation to 3test3 that the correct standards are being used and that none are overloo"ed. Bou also have to be aware o these standards and test against them when you veri y and validate the so tware. +onsider them as part o the speci ication.

38

Re3iew an" Test Similar Software .ne o the best methods or understanding what your product will become is to research similar so tware. This could be a competitor's product or something similar to what your team is creating. It's li"ely that the pro:ect manager or others who are speci ying your product have already done this, so it should be relatively easy to get access to what products they used in their research. The so tware li"ely won't be an e/act match =that's why you're creating new so tware, rightI>, but it should help you thin" about test situations and test approaches. It should also lag potential problems that may not have been considered. &ome things to loo" or when reviewing competitive products include &cale. %ill there be ewer or greater eaturesI %ill there be less or more codeI %ill that size di erence matter in your testingI +omple/ity. %ill your so tware be more or less comple/I %ill this impact your testingI Testability. %ill you have the resources, time, and e/pertise to test so tware such as thisI MualityS-eliability. Is this so tware representative o the overall )uality planned or your so twareI %ill your so tware be more or less reliableI &ecurity. #ow does the competitor's so tware security, both advertised and actual, compare to what you'll be o eringI

There's no substitute or hands-on e/perience, so do whatever you can to get a hold o similar so tware, use it, bang on it, and put it through its paces. Bou'll gain a lot o e/perience that will help you when you review your speci ication in detail. TI7 ,on't orget about reading online and printed so tware reviews and articles about the competition. This can be especially help ul or security issues as you may not li"ely see the security laws as you casually use the application. They will, though, be well "nown in the press. ,ow-,e3el Spe'ifi'ation Test Te'hniCues 2 ter you complete the high-level review o the product speci ication, you'll have a better understanding o what your product is and what e/ternal in luences a ect its design. 2rmed with this in ormation, you can move on to testing the speci ication at a lower level. The remainder o this chapter e/plains the speci ics or doing this. Q1R
Q1R

The chec"lists are adapted rom pp.494-49G and ;<;-;<A o the #andboo" o %al"throughs, Inspections, and Technical -eviews, ;rd 8dition +opyright 199<, 19A4 by ,.7. ?reedman and F.!. %einberg. $sed by permission o ,orset #ouse 7ublishing =www.dorsethouse.com>. 2ll rights reserved. Spe'ifi'ation 5ttributes Che'klist 2 good, well-thought-out product speci ication, with 3all its t's crossed and its i's dotted,3 has eight important attributes9 +omplete. Is anything missing or orgottenI Is it thoroughI ,oes it include everything necessary to ma"e it stand aloneI

39

2ccurate. Is the proposed solution correctI ,oes it properly de ine the goalI 2re there any errorsI 7recise, $nambiguous, and +lear. Is the description e/act and not vagueI Is there a single interpretationI Is it easy to read and understandI +onsistent. Is the description o the eature written so that it doesn't con lict with itsel or other items in the speci icationI -elevant. Is the statement necessary to speci y the eatureI Is it e/tra in ormation that should be le t outI Is the eature traceable to an original customer needI ?easible. +an the eature be implemented with the available personnel, tools, and resources within the speci ied budget and scheduleI +ode- ree. ,oes the speci ication stic" with de ining the product and not the underlying so tware design, architecture, and codeI Testable. +an the eature be testedI Is enough in ormation provided that a tester could create tests to veri y its operationI

%hen you're testing a product spec, reading its te/t, or e/amining its igures, care ully consider each o these traits. 2s" yoursel i the words and pictures you're reviewing have these attributes. I they don't, you've ound a bug that needs to be addressed. Spe'ifi'ation Terminology Che'klist 2 complement to the previous attributes list is a list o problem words to loo" or while reviewing a speci ication. The appearance o these words o ten signi ies that a eature isn't yet completely thought outit li"ely alls under one o the preceding attributes. 0oo" or these words in the speci ication and care ully review how they're used in conte/t. The spec may go on to clari y or elaborate on them, or it may leave them ambiguousin which case, you've ound a bug. 2lways, 8very, 2ll, *one, *ever. I you see words such as these that denote something as certain or absolute, ma"e sure that it is, indeed, certain. 7ut on your tester's hat and thin" o cases that violate them. +ertainly, There ore, +learly, .bviously, 8vidently. These words tend to persuade you into accepting something as a given. ,on't all into the trap. &ome, &ometimes, . ten, $sually, .rdinarily, +ustomarily, !ost, !ostly. These words are too vague. It's impossible to test a eature that operates 3sometimes.3 8tc., 2nd &o ?orth, 2nd &o .n, &uch 2s. 0ists that inish with words such as these aren't testable. 0ists need to be absolute or e/plained so that there's no con usion as to how the series is generated and what appears ne/t in the list. Food, ?ast, +heap, 8 icient, &mall, &table. These are un)uanti iable terms. They aren't testable. I they appear in a speci ication, they must be urther de ined to e/plain e/actly what they mean.

40

#andled, 7rocessed, -e:ected, &"ipped, 8liminated. These terms can hide large amounts o unctionality that need to be speci ied. I PThenP=but missing 8lse>. 0oo" or statements that have 3I PThen3 clauses but don't have a matching 38lse.3 2s" yoursel what will happen i the 3i 3 doesn't happen.

Summary 2 ter completing this chapter, you may have decided that testing a speci ication is a very sub:ective process. #igh-level review techni)ues will lush out oversights and omissions, and low-level techni)ues help assure that all the details are de ined. (ut, these techni)ues aren't really step-by-step processes to ollow, or two reasons9 This is an introductory boo" whose aim is to get you rapidly up the testing curve. The material presented here will do :ust that. 2rmed with the in ormation presented in this chapter, you will ma"e a big dent in any so tware spec you're given to test. The ormat o speci ications can vary widely. Bou'll be able to apply the techni)ues rom this chapter whether you're pulling the spec out o someone's brain, loo"ing at a high-level diagram, or parsing through sentences. Bou will ind bugs.

I you're interested in pursuing more advanced techni)ues or reviewing speci ications, do some research on the wor" o !ichael ?agan. %hile at I(!, !r. ?agan pioneered a detailed and methodical approach called so tware inspections that many companies use, especially companies creating mission-critical so tware, to ormally review their so tware speci ications and code. Bou can ind more in ormation on his website9 www.m agan.com. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. +an a so tware tester per orm white-bo/ testing on a speci icationI 4. +ite a ew e/amples o !ac or %indows standards or guidelines. ;. 8/plain what's wrong with this speci ication statement9 %hen the user selects the +ompact !emory option, the program will compress the mailing list data as small as possible using a #u man-sparse-matri/ approach. 4. 8/plain what a tester should worry about with this line rom a spec9 The so tware will allow up to 1<< million simultaneous connections, although no more than 1 million will normally be used.

Chapter $. Testing the Software with Blinders %n I* T#I& +#27T8 ,ynamic (lac"-(o/ Testing9 Testing the &o tware %hile (lind olded Test-to-7ass and Test-to-?ail 8)uivalence 7artitioning

41

,ata Testing &tate Testing .ther (lac"-(o/ Test Techni)ues

."ay, now or the good stu N This chapter covers what most people imagine when they thin" o so tware testing. It's time to crac" your "nuc"les, sit in ront o your computer, and start loo"ing or bugs. 2s a new so tware tester, this may be the irst :ob you're assigned to do. I you're interviewing or a so tware test position, you will no doubt be as"ed how you'd approach testing a new so tware program or a new program's eatures. It's very easy to :ump right in, start pounding on "eys, and hope that something brea"s. &uch an approach might wor" or a little while. I the so tware is still under development, it's very easy to get luc"y and ind a ew bugs right away. $n ortunately, those easy pic"ings will )uic"ly disappear and you'll need a more structured and targeted approach to continue inding bugs and to be a success ul so tware tester. This chapter describes the most common and e ective techni)ues or testing so tware. It doesn't matter what "ind o program you're testingthe same techni)ues will wor" whether it's a custom accounting pac"age or your company, an industrial automation program, or a massmar"et shoot-'em-up computer game. Bou also don't need to be a programmer to use these techni)ues. 2lthough they're all based on undamental programming concepts, they don't re)uire you to write code. 2 ew techni)ues have some bac"ground in ormation that e/plains why they're e ective, but any code samples are short and written in a simple macro language to easily demonstrate the point. I you're into programming and want to learn more low-level test techni)ues, a ter you inish reading this chapter, move on to +hapter 5, 38/amining the +ode,3 and +hapter 7, 3Testing the &o tware with O--ay Flasses,3 the white-bo/ testing chapters. Topics covered in this chapter include %hat is dynamic blac"-bo/ testingI #ow to reduce the number o test cases by e)uivalence partitioning #ow to identi y troublesome boundary conditions Food data values to use to induce bugs #ow to test so tware states and state transitions #ow to use repetition, stress, and high loads to locate bugs 2 ew secret places where bugs hide

%ynami' Bla'k-Bo( Testing: Testing the Software #hile Blin"fol"e"

42

Testing so tware without having an insight into the details o underlying code is dynamic blac"bo/ testing. It's dynamic because the program is runningyou're using it as a customer would. 2nd, it's blac"-bo/ because you're testing it without "nowing e/actly how it wor"swith blinders on. Bou're entering inputs, receiving outputs, and chec"ing the results. 2nother name commonly used or dynamic blac"-bo/ testing is behavioral testing because you're testing how the so tware actually behaves when it's used. To do this e ectively re)uires some de inition o what the so tware doesnamely, a re)uirements document or product speci ication. Bou don't need to be told what happens inside the so tware 3bo/3you :ust need to "now that inputting 2 outputs ( or that per orming operation + results in ,. 2 good product spec will provide you with these details. .nce you "now the ins and outs o the so tware you're about to test, your ne/t step is to start de ining the test cases. Test cases are the speci ic inputs that you'll try and the procedures that you'll ollow when you test the so tware. ?igure G.1 shows an e/ample o several cases that you might use or testing the addition unction o the %indows +alculator. 2igure 1>1> Test 'ases show the "ifferent inputs an" the steps to test a program>

*.T8 &electing test cases is the single most important tas" that so tware testers do. Improper selection can result in testing too much, testing too little, or testing the wrong things. Intelligently weighing the ris"s and reducing the in inite possibilities to a manageable e ective set is where the magic is. The rest o this chapter and much o the rest o the boo" will teach you how to strategically select good test cases. +hapter 1A, 3%riting and Trac"ing Test +ases,3 discusses the speci ic techni)ues or writing and managing test cases. <SE EFP,&R5T&R6 TEST!4* !2 6&< %&4+T =5:E 5 SPEC 2 pro essional, mature so tware development process will have a detailed speci ication or the so tware. I you're stuc" in a big-bang model or a sloppy codeand- i/ model, you may not have a so tware spec to base your tests on. That's not an ideal situation or a so tware tester, but you can use a wor"able solution "nown as e/ploratory testingsimultaneously learning the so tware, designing tests, and e/ecuting those tests. Bou need to treat the so tware as the speci ication. !ethodically e/plore the so tware eature by eature. Ta"e notes on what the so tware does, map out the eatures, and apply some o the static blac"-bo/ techni)ues you learned in +hapter 4, 38/amining the &peci ication.3 2nalyze the so tware as though it is the speci ication. Then apply the dynamic blac"-bo/ techni)ues rom this chapter. Bou won't be able to test the so tware as thoroughly as you would i you had a specyou won't necessarily "now i a eature is missing, or e/ample. (ut, you will be

43

able to systematically test it. In this situation, inding any bugs would be a positive thing. Test-to-Pass an" Test-to-2ail There are two undamental approaches to testing so tware9 test-to-pass and test-to- ail. %hen you test-to-pass, you really assure only that the so tware minimally wor"s. Bou don't push its capabilities. Bou don't see what you can do to brea" it. Bou treat it with "id gloves, applying the simplest and most straight orward test cases. Bou may be thin"ing that i your goal is to ind bugs, why would you test-to-passI %ouldn't you want to ind bugs by any means possibleI The answer is no, not initially. Thin" about an analogy with a newly designed car =see ?igure G.4>. Bou're assigned to test the very irst prototype that has :ust rolled o the assembly line and has never been driven. Bou probably wouldn't get in, start it up, head or the test trac", and run it wide open at ull speed as hard as you could. Bou'd probably crash and die. %ith a new car, there'd be all "inds o bugs that would reveal themselves at low speed under normal driving conditions. !aybe the tires aren't the right size, or the bra"es are inade)uate, or the engine is too loud. Bou could discover these problems and have them i/ed be ore getting on the trac" and pushing the limits. 2igure 1>7> <se test-to-pass to re3eal bugs before you test-to-fail>

*.T8 %hen designing and running your test cases, always run the test-to-pass cases irst. It's important to see i the so tware undamentally wor"s be ore you throw the "itchen sin" at it. Bou might be surprised how many bugs you ind :ust using the so tware normally. 2 ter you assure yoursel that the so tware does what it's speci ied to do in ordinary circumstances, it's time to put on your snea"y, conniving, devious hat and attempt to ind bugs by trying things that should orce them out. ,esigning and running test cases with the sole purpose o brea"ing the so tware is called testing-to- ail or error- orcing. Bou'll learn later in this chapter that test-to- ail cases o ten don't appear intimidating. They o ten loo" li"e test-topass cases, but they're strategically chosen to probe or common wea"nesses in the so tware. ERR&R )ESS5*ES: TEST-T&-P5SS &R TEST-T&-25!, 2 common class o test cases is one that attempts to orce error messages. Bou "now the onesli"e saving a ile to a loppy dis" but not having one inserted in the drive. These cases actually straddle the line between test-to-pass and test-to- ail. The speci ication probably states that certain input conditions should result in an error message. That seems pretty clear as a test-to-pass case. (ut, you're also orcing an error, so it could be viewed as test-to- ail. In the end, it's probably both. ,on't worry about the distinction. %hat's important is to try to orce the error

44

messages that are speci ied and to invent test cases to orce errors that were never considered. Bou'll li"ely end up inding both test-to-pass and test-to- ail bugs. ECui3alen'e Partitioning &electing test cases is the single most important tas" that so tware testers do and e)uivalence partitioning, sometimes called e)uivalence classing, is the means by which they do it. 8)uivalence partitioning is the process o methodically reducing the huge =in inite> set o possible test cases into a much smaller, but still e)ually e ective, set. -emember the %indows +alculator e/ample rom +hapter ;I It's impossible to test all the cases o adding two numbers together. 8)uivalence partitioning provides a systematic means or selecting the values that matter and ignoring the ones that don't. ?or e/ample, without "nowing anything more about e)uivalence partitioning, would you thin" that i you tested 141, 142, 143, and 144 that you'd need to test 145 and 146I ,o you thin" you could sa ely assume that they'd wor"I #ow about 1499999999999999999999999999999999 =the ma/imum number you can type in>I Is this test case maybe a little di erent than the others, maybe in a di erent class, a di erent e)uivalence partitionI I you had the choice, would you include it or 1413I &ee, you're already starting to thin" li"e a so tware testerN *.T8 2n e)uivalence class or e)uivalence partition is a set o test cases that tests the same thing or reveals the same bug. %hat is the di erence between 1499999999999999999999999999999999 and 1413I In the case o 1413, it loo"s li"e a standard simple addition, a lot li"e 145 or 14392. #owever, 14999... is way out there, on the edge. I you enter the largest possible number and then add 1 to it, something bad might happenpossibly a bug. This e/treme case is in a uni)ue partition, a di erent one rom the normal partition o regular numbers. *.T8 %hen loo"ing or e)uivalence partitions, thin" about ways to group similar inputs, similar outputs, and similar operation o the so tware. These groups are your e)uivalence partitions. 0oo" at a ew e/amples9 In the case o adding two numbers together, there seemed to be a distinct di erence between testing 1413 and 1499999999999999999999999999999999. +all it a gut eeling, but one seemed to be normal addition and the other seemed to be ris"y. That gut eeling is right. 2 program would have to handle the addition o 1 to a ma/ed-out number di erently than the addition o two small numbers. It would need to handle an over low condition. These two cases, because the so tware most li"ely operates on them di erently, are in di erent e)uivalence partitions. I you have some programming e/perience, you might be thin"ing o several more 3special3 numbers that could cause the so tware to operate di erently. I you're not a programmer, don't worryyou'll learn the techni)ues very shortly and be able to apply them without having to understand the code in detail. ?igure G.; shows the +alculator's 8dit menu selected to display the copy and paste commands. There are ive ways to per orm each unction. ?or copy, you clic" the +opy menu item, type ) or * when the menu is displayed, or press +trlJc or +trlJ&hi tJc. 8ach input path copies the current number into the +lipboardthey per orm the same output and produce the same result. 2igure 1>D> The multiple ways to in3oke the 'opy fun'tion all ha3e the same result>

45

I your :ob is to test the copy command, you could partition these ive input paths down to three9 +lic"ing the command on the menu, typing a c, or pressing +trlJc. 2s you grow more con ident with the so tware's )uality and "now that the copy unction, no matter how it's enabled, is wor"ing properly, you might even partition these down into a single partition, maybe +trlJc. 2s a third e/ample, consider the possibilities or entering a ilename in the standard &ave 2s dialog bo/ =see ?igure G.4>. 2igure 1>0> The 2ile 4ame te(t bo( in the Sa3e 5s "ialog bo( illustrates se3eral eCui3alen'e partition possibilities>

2 %indows ilename can contain any characters e/cept :/ * ; < = > and ?. ?ilenames can have rom 1 to 4GG characters. I you're creating test cases or ilenames, you will have e)uivalence partitions or valid characters, invalid characters, valid length names, names that are too short, and names that are too long. -emember, the goal o e)uivalence partitioning is to reduce the set o possible test cases into a smaller, manageable set that still ade)uately tests the so tware. Bou're ta"ing on ris" because you're choosing not to test everything, so you need to be care ul how you choose your classes. *.T8

46

I you e)uivalence partition too ar in your e ort to reduce the number o test cases, you ris" eliminating tests that could reveal bugs. I you're new to testing, always get someone with more e/perience to review your proposed classes. 2 inal point about e)uivalence partitioning is that it can be sub:ective. It's science but it's also art. Two testers who test a comple/ program may arrive at two di erent sets o partitions. That's o"ay as long as the partitions are reviewed and everyone agrees that they acceptably cover the so tware being tested. %ata Testing The simplest view o so tware is to divide its world into two parts9 the data =or its domain> and the program. The data is the "eyboard input, mouse clic"s, dis" iles, printouts, and so on. The program is the e/ecutable low, transitions, logic, and computations. 2 common approach to so tware testing is to divide up the test wor" along the same lines. %hen you per orm so tware testing on the data, you're chec"ing that in ormation the user inputs, results that he receives, and any interim results internal to the so tware are handled correctly. 8/amples o data would be The words you type into a word processor The numbers entered into a spreadsheet The number o shots you have remaining in your space game The picture printed by your photo so tware The bac"up iles stored on your loppy dis" The data being sent by your modem over the phone lines

The amount o data handled by even the simplest programs can be overwhelming. -emember all the possibilities o input data or per orming simple addition on a calculatorI +onsider a word processor, a missile guidance system, or a stoc" trading program. The tric" =i you can call it that> to ma"ing any o these testable is to intelligently reduce the test cases by e)uivalence partitioning based on a ew "ey concepts9 boundary conditions, sub-boundary conditions, nulls, and bad data. Boun"ary Con"itions The best way to describe boundary condition testing is shown in ?igure G.G. I you can sa ely and con idently wal" along the edge o a cli without alling o , you can almost certainly wal" in the middle o a ield. I so tware can operate on the edge o its capabilities, it will almost certainly operate well under normal conditions. 2igure 1>1> 5 software boun"ary is mu'h like the e"ge of a 'liff>

47

(oundary conditions are special because programming, by its nature, is susceptible to problems at its edges. &o tware is very binarysomething is either true or it isn't. I an operation is per ormed on a range o numbers, odds are the programmer got it right or the vast ma:ority o the numbers in the middle, but maybe made a mista"e at the edges. 0isting G.1 shows how a boundary condition problem can ma"e its way into a very simple program. ,isting 1>1> 5 Simple B5S!C Program %emonstrating a Boun"ary Con"ition Bug 1 2 3 4 5 6 7 8 @eA *re$te $ 13 eleAe%t i%te-er $rr$" @eA B%iti$li6e e$)h eleAe%t to -1 8iA d$t$(13) 0' B%te-er 8iA i 0' B%te-er (or i 5 1 Co 13 d$t$(i) 5 -1 De1t i E%d

The purpose o this code is to create a 1<-element array and initialize each element o the array to 1. It loo"s airly simple. 2n array =data> o 1< integers and a counter = i> are created. 2 (or loop runs rom 1 to 1<, and each element o the array rom 1 to 1< is assigned a value o 1. %here's the boundary problemI In most (2&I+ scripts, when an array is dimensioned with a stated rangein this case, 8iA d$t$(13) $' B%te-erthe irst element created is <, not 1. This program actually creates a data array o 11 elements rom d$t$(3) to d$t$(13). The program loops rom 1 to 1< and initializes those values o the array to 1, but since the irst element o our array is d$t$(3), it doesn't get initialized. %hen the program completes, the array values loo" li"e this9

48

d$t$(3) L < d$t$(6) L 1 d$t$(1) L 1 d$t$(7) L 1 d$t$(2) L 1 d$t$(8) L 1 d$t$(3) L 1 d$t$(9) L 1 d$t$(4) L 1 d$t$(13) L 1 d$t$(5) L 1 *otice that d$t$(3)'s value is <, not 1. I the same programmer later orgot about, or a di erent programmer wasn't aware o how this data array was initialized, he might use the irst element o the array, d$t$(3), thin"ing it was set to 1. 7roblems such as this are very common and, in large comple/ so tware, can result in very nasty bugs. Types of Boun"ary Con"itions *ow it's time to open your mind and really thin" about what constitutes a boundary. (eginning testers o ten don't realize how many boundaries a given set o data can have. $sually there are a ew obvious ones, but i you dig deeper you'll ind the more obscure, interesting, and o ten bug-prone boundaries. *.T8 (oundary conditions are those situations at the edge o the planned operational limits o the so tware. %hen you're presented with a so tware test problem that involves identi ying boundaries, loo" or the ollowing types9 *umeric &peed

+haracter 0ocation 7osition Muantity 2nd, thin" about the ollowing characteristics o those types9 ?irstS0ast &tartS?inish !inS!a/ .verS$nder &ize

49

?irstS0ast 8mptyS?ull &lowestS?astest 0argestS&mallest *e/t-ToS?arthest-?rom

!inS!a/ &hortestS0ongest &oonestS0atest #ighestS0owest

These are not by any means de initive lists. They cover many o the possible boundary conditions, but each so tware testing problem is di erent and may involve very di erent data with very uni)ue boundaries. TI7 I you have a choice o what data you're going to include in your e)uivalence partition, choose data that lies on the boundary. Testing the Boun"ary E"ges %hat you've learned so ar is that you need to create e)uivalence partitions o the di erent data sets that your so tware operates on. &ince so tware is susceptible to bugs at the boundaries, i you're choosing what data to include in your e)uivalence partition, you'll ind more bugs i you choose data rom the boundaries. (ut testing the data points :ust at the edge o the boundary line isn't usually su icient. 2s the words to the 3#o"ey 7o"ey3 imply =37ut your right hand in, put your right hand out, put your right hand in, and you sha"e it all aboutP3>, it's a good idea to test on both sides o the boundaryto sha"e things up a bit. Bou'll ind the most bugs i you create two e)uivalence partitions. The irst should contain data that you would e/pect to wor" properlyvalues that are the last one or two valid points inside the boundary. The second partition should contain data that you would e/pect to cause an errorthe one or two invalid points outside the boundary. TI7 %hen presented with a boundary condition, always test the valid data :ust inside the boundary, test the last possible valid data, and test the invalid data :ust outside the boundary. Testing outside the boundary is usually as simple as adding one, or a bit more, to the ma/imum value and subtracting one, or a bit more, rom the minimum value. ?or e/ample9 ?irst1S0astJ1 &tart1S?inishJ1 0ess than 8mptyS!ore than ?ull 8ven &lowerS8ven ?aster 0argestJ1S&mallest1 !in1S!a/J1

50

Hust .verSHust $nder 8ven &horterS0onger 8ven &oonerS0ater #ighestJ1S0owest1

0oo" at a ew e/amples so you can start thin"ing about all the boundary possibilities9 I a te/t entry ield allows 1 to 4GG characters, try entering 1 character and 4GG characters as the valid partition. Bou might also try 4G4 characters as a valid choice. 8nter < and 4G5 characters as the invalid partitions. I a program reads and writes to a +,--, try saving a ile that's very small, maybe with one entry. &ave a ile that's very large:ust at the limit or what the disc holds. 2lso try saving an empty ile and a ile that's too large to it on the disc. I a program allows you to print multiple pages onto a single page, try printing :ust one =the standard case> and try printing the most pages that it allows. I you can, try printing zero pages and one more than it allows. !aybe the so tware has a data-entry ield or a 9-digit TI7 code. Try <<<<<-<<<<, the simplest and smallest. Try entering 99999-9999 as the largest. Try entering one more or one less digit than what's allowed. I you're testing a light simulator, try lying right at ground level and at the ma/imum allowed height or your plane. Try lying below ground level and below sea level as well as into outer space.

&ince you can't test everything, per orming e)uivalence partitioning around boundary conditions, such as in these e/amples, to create your test cases is critical. It's the most e ective way to reduce the amount o testing you need to per orm. *.T8 It's vitally important that you continually loo" or boundaries in every piece o so tware you wor" with. The more you loo", the more boundaries you'll discover, and the more bugs you'll ind. *.T8 (u er .verruns are caused by boundary condition bugs. They are the number one cause o so tware security issues. +hapter 1;, 3Testing or &o tware &ecurity,3 discusses the speci ic situations that cause bu er overruns and how you can test or them.

Sub-Boun"ary Con"itions The normal boundary conditions :ust discussed are the most obvious to ind. They're the ones de ined in the speci ication or evident when using the so tware. &ome boundaries, though, that are internal to the so tware aren't necessarily apparent to an end user but still need to be chec"ed by the so tware tester. These are "nown as sub-boundary conditions or internal boundary conditions. These boundaries don't re)uire that you be a programmer or that you be able to read the raw code that you're testing, but they do re)uire a bit o general "nowledge about how so tware

51

wor"s. Two e/amples are powers-o -two and the 2&+II table. The so tware that you're testing can have many others, so you should tal" with your team's programmers to see i they can o er suggestions or other sub-boundary conditions that you should chec". Powers-of-Two +omputers and so tware are based on binary numbersbits representing <s and 1s, bytes made up o A bits, words =on ;4-bit systems> made up o 4 bytes, and so on. Table G.1 shows the common powers-o -two units and their e)uivalent values. Table 5.1. Software Powers-ofTwo Term (it *ibble (yte %ord 1ilo !ega Figa Tera Range or Value < or 1 <1G <4GG <4,494,957,49G 1,<44 1,<4A,G75 1,<7;,741,A44 1,<99,G11,547,775

The ranges and values shown in Table G.1 are critical values to treat as boundary conditions. Bou li"ely won't see them speci ied in a re)uirements document unless the so tware presents the same range to the user. . ten, though, they're used internally by the so tware and are invisible, unless o course they create a situation or a bug. 54 EF5)P,E &2 P&#ERS-&2-T#& 2n e/ample o how powers-o -two come into play is with communications so tware. (andwidth, or the trans er capacity o your in ormation, is always limited. There's always a need to send and receive in ormation aster than what's possible. ?or this reason, so tware engineers try to pac" as much data into communications strings as they can. .ne way they do this is to compress the in ormation into the smallest units possible, send the most common in ormation in these small units, and then e/pand to the ne/t size units as necessary. &uppose that a communications protocol supports 4G5 commands. The so tware could send the most common 1G commands encoded into a small nibble o data. ?or the 15th through 4G5th commands, the so tware could then switch over to send the commands encoded into the longer bytes. The so tware user "nows only that he can issue 4G5 commandsE he doesn't "now that the so tware is per orming special calculations and di erent operations on the nibbleSbyte boundary.

52

%hen you create your e)uivalence partitions, consider whether powers-o -two boundary conditions need to be included in your partition. ?or e/ample, i your so tware accepts a range o numbers rom 1 to 1<<<, you've learned to include in your valid partition 1 and 1<<<, maybe 4 and 999. To cover any possible powers-o -two sub-boundaries, also include the nibble boundaries o 14, 1G, and 15, and the byte boundaries o 4G4, 4GG, and 4G5. 5SC!! Table 2nother common sub-boundary condition is the 2&+II character table. Table G.4 is a partial listing o the 2&+II table. Table 5.2. A Partial ASCII Table of Values Characte ASCII r Value *ull &pace S < 1 4 9 9 U 2 < ;4 47 4A 49 G< G7 GA 54 5G Characte ASCII r Value ( B T Q ' a b y z V 55 A9 9< 91 95 97 9A 141 144 14;

*otice that Table G.4 is not a nice, contiguous list. < through 9 are assigned to 2&+II values 4A through G7. The slash character, S, alls be ore <. The colon, 9, comes a ter 9. The uppercase letters 2 through T go rom 5G to 9<. The lowercase letters span 97 to 144. 2ll these cases represent sub-boundary conditions. I you're testing so tware that per orms te/t entry or te/t conversion, you'd be very wise to re erence a copy o the 2&+II table and consider its boundary conditions when you de ine what values to include in your data partitions. ?or e/ample, i you are testing a te/t bo/ that accepts only the characters 2T and az, you should include in your invalid partition the values :ust below and above those in the 2&+II tableU, Q, ', and V. 5SC!! 54% <4!C&%E 2lthough 2&+II is still very popular as the common means or so tware to represent character data, it's being replaced by a new standard called $nicode. $nicode was developed by the $nicode +onsortium in 1991 to solve 2&+II's problem o not being able to represent all characters in all written languages. 2&+II, using only A bits, can represent only 4G5 di erent characters. $nicode, which uses 15 bits, can represent 5G,G;5 characters. To date, more than ;9,<<< characters have been assigned, with more than 41,<<< being used or +hinese ideographs. %efault. Empty. Blank. 4ull. Gero. an" 4one

53

2nother source o bugs that may seem obvious is when the so tware re)uests an entrysay, in a te/t bo/but rather than type the correct in ormation, the user types nothing. #e may :ust press 8nter. This situation is o ten overloo"ed in the speci ication or orgotten by the programmer but is a case that typically happens in real li e. %ell-behaved so tware will handle this situation. It will usually de ault to the lowest valid boundary limit or to some reasonable value in the middle o the valid partition, or return an error. The %indows 7aint 2ttributes dialog bo/ =see ?igure G.5> normally places de ault values in the %idth and #eight te/t ields. I the user accidentally or purposely deletes them so that the ields are blan" and then clic"s .1, what happensI 2igure 1>E> The #in"ows Paint 5ttributes "ialog bo( with the #i"th an" =eight te(t fiel"s blanke" out>

Ideally, the so tware would handle this by de aulting to some valid width and height. I it didn't do that, some error should be returned, which is e/actly what you get =see ?igure G.7>. The error 3(itmaps must be greater than one pi/el on a side3 isn't the most descriptive one ever written, but that's another topic. 2igure 1>9> The error message returne" if Enter is presse" with the #i"th an" =eight te(t fiel"s blanke" out>

TI7 2lways consider creating an e)uivalence partition that handles the de ault, empty, blan", null, zero, or none conditions. Bou should create a separate e)uivalence partition or these values rather than lump them into the valid cases or the invalid cases because the so tware usually handles them di erently. It's li"ely that in this de ault case, a di erent so tware path is ollowed than i the user typed < or 1 as invalid values. &ince you e/pect di erent operation o the so tware, they should be in their own partition.

54

!n3ali". #rong. !n'orre't. an" *arbage %ata The inal type o data testing is garbage data. This is where you test-to- ail. Bou've already proven that the so tware wor"s as it should by testing-to-pass with boundary testing, subboundary testing, and de ault testing. *ow it's time to throw the trash at it. &o tware testing purists might argue that this isn't necessary, that i you've tested everything discussed so ar you've proven the so tware will wor". In the real world, however, there's nothing wrong with seeing i the so tware will handle whatever a user can do to it. I you consider that so tware today can sell hundreds o millions o copies, it's conceivable that some percentage o the users will use the so tware incorrectly. I that results in a crash or data loss, users won't blame themselvesthey will blame the so tware. I the so tware doesn't do what they e/pect, it has a bug. 7eriod. &o, with invalid, wrong, incorrect, and garbage data testing, have some un. I the so tware wants numbers, give it letters. I it accepts only positive numbers, enter negative numbers. I it's date sensitive, see i it'll wor" correctly on the year ;<<<. 7retend to have 3 at ingers3 and press multiple "eys at a time. There are no real rules or this testing other than to try to brea" the so tware. (e creative. (e devious. #ave un. State Testing &o ar what you've been testing is the datathe numbers, words, inputs, and outputs o the so tware. The other side o so tware testing is to veri y the program's logic low through its various states. 2 so tware state is a condition or mode that the so tware is currently in. +onsider ?igures G.A and G.9. 2igure 1>H> The #in"ows Paint program in the pen'il "rawing state>

2igure 1>/> The #in"ows Paint program in the airbrushing state>

55

?igure G.A shows the %indows 7aint program in the pencil drawing state. This is the initial state in which the so tware starts. *otice that the pencil tool is selected, the cursor loo"s li"e a pencil, and a ine line is used to draw onscreen. ?igure G.9 shows the same program in the airbrush state. In this state, the airbrush tool is selected, airbrush sizes are provided, the cursor loo"s li"e a spray-paint can, and drawing results in a spray-paint loo". Ta"e a closer loo" at all the available options that 7aint providesall the tools, menu items, colors, and so on. %henever you select one o these and ma"e the so tware change its loo", its menus, or its operation, you're changing its state. The so tware ollows a path through the code, toggles some bits, sets some variables, loads some data, and arrives at a new state o being. *.T8 2 so tware tester must test a program's states and the transitions between them. Testing the Software+s ,ogi' 2low -emember the e/ample in +hapter ; that showed the in inite data possibilities or testing the %indows +alculatorI Bou learned earlier in this chapter that to ma"e the testing manageable, you must reduce the data possibilities by creating e)uivalence partitions o only the most vital numbers. Testing the so tware's states and logic low has the same problems. It's usually possible to visit all the states =a ter all, i you can't get to them, why have themI>. The di iculty is that e/cept or the simplest programs, it's o ten impossible to traverse all paths to all states. The comple/ity o the so tware, especially due to the richness o today's user inter aces, provides so many choices and options that the number o paths grows e/ponentially.

56

The problem is similar to the well-"nown traveling salesman problem9 Fiven a i/ed number o cities and the distance between each pair o them, ind the shortest route to visit all o them once, returning to your starting point. I there were only ive cities, you could do some )uic" math and discover that there are 14< di erent routes. Traversing each o them and inding the shortest route to all wouldn't be that di icult or ta"e that much time. I you increase that to hundreds or thousands o citiesor, in our case, hundreds or thousands o so tware statesyou soon have a di icult-to-solve problem. The solution or so tware testing is to apply e)uivalence partition techni)ues to the selection o the states and paths, assuming some ris" because you will choose not to test all o them, but reducing that ris" by ma"ing intelligent choices. Creating a State Transition )ap The irst step is to create your own state transition map o the so tware. &uch a map may be provided as part o the product speci ication. I it is, you should statically test it as described in +hapter 4, 38/amining the &peci ication.3 I you don't have a state map, you'll need to create one. There are several di erent diagramming techni)ues or state transition diagrams. ?igure G.1< shows two e/amples. .ne uses bo/es and arrows and the other uses circles =bubbles> and arrows. The techni)ue you use to draw your map isn't important as long as you and the other members o your pro:ect team can read and understand it. 2igure 1>18> State transition "iagrams 'an be "rawn by using "ifferent te'hniCues>

*.T8 &tate transition diagrams can become )uite large. !any development teams cover their o ice walls with the printouts. I you e/pect that your diagrams will become that comple/, loo" or commercial so tware that helps you draw and manage them. 2 state transition map should show the ollowing items9 8ach uni)ue state that the so tware can be in. 2 good rule o thumb is that i you're unsure whether something is a separate state, it probably is. Bou can always collapse it into another state i you ind out later that it isn't. The input or condition that ta"es it rom one state to the ne/t. This might be a "ey press, a menu selection, a sensor input, a telephone ring, and so on. 2 state can't be e/ited without some reason. The speci ic reason is what you're loo"ing or here. &et conditions and produced output when a state is entered or e/ited. This would include a menu and buttons being displayed, a lag being set, a printout occurring, a calculation being per ormed, and so on. It's anything and everything that happens on the transition rom one state to the ne/t.

57

-8!I*,8(ecause you are per orming blac"-bo/ testing, you don't need to "now what low-level variables are being set in the code. +reate your map rom the user's view o the so tware.

Re"u'ing the 4umber of States an" Transitions to Test +reating a map or a large so tware product is a huge underta"ing. #ope ully, you'll be testing only a portion o the overall so tware so that ma"ing the map is a more reasonable tas". .nce you complete the map, you'll be able to stand bac" and see all the states and all the ways to and rom those states. I you've done your :ob right, it'll be a scary pictureN I you had in inite time, you would want to test every path through the so twarenot :ust each line connecting two states, but each set o lines, bac" to ront, round and round. 2s in the traveling salesman problem, it would be impossible to hit them all. Hust as you learned with e)uivalence partitioning or data, you need to reduce the huge set o possibilities to a set o test cases o wor"able size. There are ive ways to do this9 6isit each state at least once. It doesn't matter how you get there, but each state needs to be tested. Test the state-to-state transitions that loo" li"e the most common or popular. This sounds sub:ective, and it is, but it should be based on the "nowledge you gained when you per ormed static blac"-bo/ analysis =in +hapter ;> o the product speci ication. &ome user scenarios will be more re)uently used than others. Bou want those to wor"N Test the least common paths between states. It's li"ely that these paths were overloo"ed by the product designers and the programmers. Bou may be the irst one to try them. Test all the error states and returning rom the error states. !any times error conditions are di icult to create. 6ery o ten programmers write the code to handle speci ic errors but can't test the code themselves. There are o ten cases when errors aren't properly handled, when the error messages are incorrect, or when the so tware doesn't recover properly when the error is i/ed. Test random state transitions. I you have a printed state map, throw darts at it and try to move rom dart to dart. I you have time to do more, read +hapter 1G, 32utomated Testing and Test Tools,3 or in ormation on how to automate your random state transition testing.

#hat to Spe'ifi'ally Test 2 ter you identi y the speci ic states and state transitions that you want to test, you can begin de ining your test cases. Testing states and state transitions involves chec"ing all the state variablesthe static conditions, in ormation, values, unctionality, and so on that are associated with being in that state or moving to and rom that state. ?igure G.11 shows an e/ample o %indows 7aint in the startup state. 2igure 1>11> The #in"ows Paint opening s'reen in the startup state>

58

#ere's a partial list o the state variables that de ine 7aint's startup state9 The window loo"s as shown in ?igure G.11. The window size is set to what it was the last time 7aint was used. The drawing area is blan". The tool bo/, color bo/, and status bar are displayed. The pencil tool is selected. 2ll the others are not. The de ault colors are blac" oreground on a white bac"ground. The document name is u%titled.

There are many, many more state variables to consider or 7aint, but these should give you an idea o what's involved in de ining a state. 1eep in mind that the same process o identi ying state conditions is used whether the state is something visible such as a window or a dialog bo/, or invisible such as one that's part o a communications program or a inancial pac"age. It's a good idea to discuss your assumptions about the states and state transitions with your team's spec writers and programmers. They can o er insights into states that happen behind the scenes that you may not have considered. T=E %!RT6 %&C<)E4T 2,5* &tate variables can be invisible but very important. 2 common e/ample is the dirty document lag. %hen a document is loaded into an editor, such as a word processor or painting program, an internal state variable called the dirty document lag is cleared and the so tware is in the 3clean3 state. The so tware stays in this state as long as no changes

59

are made to the document. It can be viewed and scrolled and the state stays the same. 2s soon as something is typed or the document is modi ied in some way, the so tware changes state to the 3dirty3 state. I an attempt is made to close or e/it the so tware in the clean state, it shuts down normally. I the document is dirty, users will get a message as"ing i they want to save their wor" be ore )uitting. &ome so tware is so sophisticated that i an edit is made that dirties the document and then the edit is undone to restore the document to its original condition, the so tware is returned to the clean state. 8/iting the program will occur without a prompt to save the document.

Testing States to 2ail 8verything discussed so ar regarding state testing has been about testing-to-pass. Bou're reviewing the so tware, s"etching out the states, trying many valid possibilities, and ma"ing sure the states and state transitions wor". The lip side to this, :ust as in data testing, is to ind test cases that test the so tware to ail. 8/amples o such cases are race conditions, repetition, stress, and load. Ra'e Con"itions an" Ba" Timing !ost operating systems today, whether or personal computers or or specialized e)uipment, can do multitas"ing. !ultitas"ing means that an operating system is designed to run separate processes concurrently. These processes can be separate programs such as a spreadsheet and email. .r they can be part o the same program such as printing in the bac"ground while allowing new words to be typed into a word processor. ,esigning a multitas"ing operating system isn't a trivial e/ercise, and designing applications so tware to ta"e advantage o multitas"ing is a di icult tas". In a truly multitas"ing environment, the so tware can't ta"e anything or granted. It must handle being interrupted at any moment, be able to run concurrently with everything else on the system, and share resources such as memory, dis", communications, and other hardware. The results o all this are race condition problems. These are when two or more events line up :ust right and con use so tware that didn't e/pect to be interrupted in the middle o its operation. In other words, it's bad timing. The term race condition comes rom :ust what you'd thin"multiple processes racing to a inish line, not "nowing which will get there irst. *.T8 -ace condition testing is di icult to plan or, but you can get a good start by loo"ing at each state in your state transition map and thin"ing about what outside in luences might interrupt that state. +onsider what the state might do i the data it uses isn't ready or is changing when it's needed. %hat i two or more o the connecting arcs or lines occur at e/actly the same timeI #ere are a ew e/amples o situations that might e/pose race conditions9 &aving and loading the same document at the same time with two di erent programs &haring the same printer, communications port, or other peripheral 7ressing "eys or sending mouse clic"s while the so tware is loading or changing states &hutting down or starting up two or more instances o the so tware at the same time $sing di erent programs to simultaneously access a common database

60

These may sound li"e harsh tests, but they aren't, and the user o ten causes them by accident. &o tware must be robust enough to handle these situations. Bears ago they may have been out o the ordinary but today, users e/pect their so tware to wor" properly under these conditions. Repetition. Stress. an" ,oa" Three other test-to- ail state tests are repetition, stress, and load. These tests target state handling problems where the programmer didn't consider what might happen in the worst-case scenarios. -epetition testing involves doing the same operation over and over. This could be as simple as starting up and shutting down the program over and over. It could also mean repeatedly saving and loading data or repeatedly selecting the same operation. Bou might ind a bug a ter only a couple repetitions or it might ta"e thousands o attempts to reveal a problem. The main reason or doing repetition testing is to loo" or memory lea"s. 2 common so tware problem happens when computer memory is allocated to per orm a certain operation but isn't completely reed when the operation completes. The result is that eventually the program uses up memory that it depends on to wor" reliably. I you've ever used a program that wor"s ine when you irst start it up, but then becomes slower and slower or starts to behave erratically over time, it's li"ely due to a memory lea" bug. -epetition testing will lush these problems out. &tress testing is running the so tware under less-than-ideal conditionslow memory, low dis" space, slow +7$s, slow modems, and so on. 0oo" at your so tware and determine what e/ternal resources and dependencies it has. &tress testing is simply limiting them to their bare minimum. Bour goal is to starve the so tware. ,oes this sound li"e boundary condition testingI It is. 0oad testing is the opposite o stress testing. %ith stress testing, you starve the so twareE with load testing, you eed it all that it can handle. .perate the so tware with the largest possible data iles. I the so tware operates on peripherals such as printers or communications ports, connect as many as you can. I you're testing an Internet server that can handle thousands o simultaneous connections, do it. !a/ out the so tware's capabilities. 0oad it down. ,on't orget about time as a load testing variable. %ith most so tware, it's important or it to run over long periods. &ome so tware should be able to run orever without being restarted. *.T8 There's no reason that you can't combine repetition, stress, and load, running all the tests at the same time. This is a sure way to e/pose severe bugs that might otherwise be di icult to ind. There are two important considerations with repetition, stress, and load testing9 Bour team's programmers and pro:ect managers may not be completely receptive to your e orts to brea" the so tware this way. Bou'll probably hear them complain that no customer will use the system this way or stress it to the point that you are. The short answer is that yes, they will. Bour :ob is to ma"e sure that the so tware does wor" in these situations and to report bugs i it doesn't. +hapter 19, 3-eporting %hat Bou ?ind,3 discusses how to best report your bugs to ma"e sure that they're ta"en seriously and are i/ed. .pening and closing your program a million times is probably not possible i you're doing it by hand. 0i"ewise, inding a ew thousand people to connect to your Internet server might be di icult to organize. +hapter 1G covers test automation and will give you ideas on how to per orm testing such as this without re)uiring people to do the dirty wor".

&ther Bla'k-Bo( Test Te'hniCues

61

The remaining categories o blac"-bo/ test techni)ues aren't standalone methods as much as they are variations o the data testing and state testing that has already been described. I you've done thorough e)uivalence partitioning o your program's data, created a detailed state map, and developed test cases rom these, you'll ind most so tware bugs that a user would ind. %hat's le t are techni)ues or inding the stragglers, the ones that, i they were real living bugs, might appear to have a mind o their own, going their own way. ?inding them might appear a bit sub:ective and not necessarily based on reason, but i you want to lush out every last bug, you'll have to be a bit creative. Beha3e ,ike a %umb <ser The politically correct term might be ine/perienced user or new user, but in reality, they're all the same thing. 7ut a person who's un amiliar with the so tware in ront o your program and they'll do and try things that you never imagined. They'll enter data that you never thought o . They'll change their mind in mid-stream, bac" up, and do something di erent. They'll sur through your website, clic"ing things that shouldn't be clic"ed. They'll discover bugs that you completely missed. It can be rustrating, as a tester, to watch someone who has no e/perience in testing spend ive minutes using a piece o so tware and crash it. #ow do they do itI They weren't operating on any rules or ma"ing any assumptions. %hen you're designing your test cases or loo"ing at the so tware or the irst time, try to thin" li"e a dumb user. Throw out any preconceived ideas you had about how the so tware should wor". I you can, bring in a riend who isn't wor"ing on the pro:ect to brainstorm ideas with you. 2ssume nothing. 2dding these test cases to your designed library o test cases will create a very comprehensive set. ,ook for Bugs #here 6ou+3e 5lrea"y 2oun" Them There are two reasons to loo" or bugs in the areas where you've already ound them9 2s you learned in +hapter ;, the more bugs you ind, the more bugs there are. I you discover that you're inding lots o bugs at the upper boundary conditions across various eatures, it would be wise to emphasize testing these upper boundaries on all eatures. . course you're going to test these anyway, but you might want to throw in a ew special cases to ma"e sure the problem isn't pervasive. !any programmers tend to i/ only the speci ic bug you report. *o more, no less. I you report a bug that starting, stopping, and restarting a program 4GG times results in a crash, that's what the programmer will i/. There may have been a memory lea" that caused the problem and the programmer ound and i/ed it. %hen you get the so tware bac" to retest, ma"e sure you rerun the same test or 4G5 times and beyond. There could very well be yet another memory lea" somewhere out there.

Think like a =a'ker 2s you'll learn in +hapter 1;, no so tware is 1<<W secure. The hac"ers o the world "now this and will see" to ind vulnerabilities in your so tware and e/ploit them. 2s a tester, you'll need to be devious and conniving. Try to thin" o what things o value your so tware contains, why someone might want to gain access to them, and what the methods o entry might be or them to get in. ,on't be gentle. They won't be. 2ollow E(perien'e. !ntuition. an" =un'hes

62

There's no better way to improve as a so tware tester than to gain e/perience. There's no better learning tool than :ust doing it, and there's no better lesson than getting that irst phone call rom a customer who ound a bug in the so tware you :ust inished testing. 8/perience and intuition can't be taught. They must be gained over time. Bou can apply all the techni)ues you've learned so ar and still miss important bugs. It's the nature o the business. 2s you progress through your career, learning to test di erent types and sizes o products, you'll pic" up little tips and tric"s that steer you toward those tough-to- ind bugs. Bou'll be able to start testing a new piece o so tware and )uic"ly ind bugs that your peers would have missed. Ta"e notes o what wor"s and what doesn't. Try di erent approaches. I you thin" something loo"s suspicious, ta"e a closer loo". Fo with your hunches until you prove them alse. 8/perience is the name everyone gives to their mista"es. .scar %ilde Summary It's been a long chapter. ,ynamic blac"-bo/ testing covers a lot o ground. ?or new testers, this may be the single most important chapter in the boo". It's li"ely that at your interviews or your irst day on the :ob you'll be given so tware and as"ed to test it. 2pplying this chapter's techni)ues is a sure way to immediately ind bugs. ,on't assume, though, that this is all there is to so tware testing. I it was, you could stop reading right now and ignore the remaining chapters. ,ynamic blac"-bo/ testing will :ust get you in the door. There's so much more to so tware testing, and you're :ust getting started. The ne/t two chapters introduce you to so tware testing when you have access to the code and can see how it wor"s and what it does on the lowest levels. The same blac"-bo/ techni)ues are still valid, but you'll be able to complement them with new techni)ues that will help you become an even more e ective so tware tester AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N

1. True or ?alse9 Bou can per orm dynamic blac"-bo/ testing without a product

speci ication or re)uirements document. 4. I you're testing a program's ability to print to a printer, what generic test-to- ail test cases might be appropriateI ;. &tart up %indows %ord7ad and select 7rint rom the ?ile menu. Bou'll get the dialog shown in ?igure G.14. %hat boundary conditions e/ist or the 7rint -ange eature shown in the lower-le t cornerI 2igure 1>17> The #in"ows Print "ialog bo( showing the Print Range feature>

63

4. 2ssume that you have a 1<-character-wide TI7 code te/t bo/, such as the one shown in ?igure G.1;. %hat e)uivalence partitions would you create or this te/t bo/I 2igure 1>1D> 5 sample G!P 'o"e te(t bo( that hol"s up to 18 'hara'ters>

G. True or ?alse9 6isiting all the states that a program has assures that you've also traversed all the transitions among them. 5. There are many di erent ways to draw state transition diagrams, but there are three things that they all show. %hat are theyI 7. %hat are some o the initial state variables or the %indows +alculatorI A. %hat actions do you per orm on so tware when attempting to e/pose race condition bugsI 9.True or ?alse9 It's an un air test to per orm stress testing at the same time you per orm load testing.

Chapter &. "#amining the Code I* T#I& +#27T8 &tatic %hite-(o/ Testing9 8/amining the ,esign and +ode ?ormal -eviews +oding &tandards and Fuidelines

64

Feneric +ode -eview +hec"list

&o tware testing isn't limited to treating the speci ication or the program li"e a blac" bo/ as described in +hapters 4, 38/amining the &peci ication,3 and G, 3Testing the &o tware with (linders .n.3 I you have some programming e/perience, even i it's :ust a little, you can also per orm testing on the so tware's architecture and code. In some industries, such veri ication isn't as common as blac"-bo/ testing. #owever, i you're testing military, inancial, actory automation, or medical so tware, or i you're luc"y enough to be wor"ing in a highly disciplined development model, it may be routine to veri y the product at this level. I you're testing so tware or security issues, it's imperative. This chapter introduces you to the basics o per orming veri ication on the design and code. 2s a new so tware tester, it may not be your irst tas", but it's one that you can eventually move into i your interests lie in programming. #ighlights rom this chapter include The bene its o static white-bo/ testing The di erent types o static white-bo/ reviews +oding guidelines and standards #ow to generically review code or errors

Stati' #hite-Bo( Testing: E(amining the %esign an" Co"e -emember the de initions o static testing and white-bo/ testing rom +hapter 4I &tatic testing re ers to testing something that isn't runninge/amining and reviewing it. %hite-bo/ =or clear-bo/> testing implies having access to the code, being able to see it and review it. &tatic white-bo/ testing is the process o care ully and methodically reviewing the so tware design, architecture, or code or bugs without e/ecuting it. It's sometimes re erred to as structural analysis. The obvious reason to per orm static white-bo/ testing is to ind bugs early and to ind bugs that would be di icult to uncover or isolate with dynamic blac"-bo/ testing. #aving a team o testers concentrate their e orts on the design o the so tware at this early stage o development is highly cost e ective. 2 side bene it o per orming static white-bo/ testing is that it gives the team's blac"-bo/ testers ideas or test cases to apply when they receive the so tware or testing. They may not necessarily understand the details o the code, but by listening to the review comments they can identi y eature areas that sound troublesome or bug-prone. *.T8 ,evelopment teams vary in who has the responsibility or static white-bo/ testing. In some teams the programmers are the ones who organize and run the reviews, inviting the so tware testers as independent observers. In other teams the so tware testers are the ones who per orm this tas", as"ing the programmer who wrote the code and a couple o his peers to assist in the reviews. $ltimately, either approach can wor". It's up to the development team to choose what wor"s best or them. The un ortunate thing about static white-bo/ testing is that it's not always done. !any teams have the misconception that it's too time-consuming, too costly, or not productive. 2ll o these are untruecompared to the alternative o testing, inding, and even not inding bugs at the bac" end o the pro:ect. The problem lies in the perception that a programmer's :ob is to write lines o code and that any tas" that ta"es away rom his e iciency o churning out those lines is slowing down the process. ?ortunately, the tide is changing. !any companies are realizing the bene its o testing early and are hiring and training their programmers and testers to per orm white-bo/ testing. It's not

65

roc"et science =unless you're designing roc"ets>, but getting started re)uires "nowing a ew basic techni)ues. I you're interested in ta"ing it urther, the opportunities are huge. 2ormal Re3iews 2 ormal review is the process under which static white-bo/ testing is per ormed. 2 ormal review can range rom a simple meeting between two programmers to a detailed, rigorous inspection o the so tware's design or its code. There are our essential elements to a ormal review9 Identi y 7roblems. The goal o the review is to ind problems with the so twarenot :ust items that are wrong, but missing items as well. 2ll criticism should be directed at the design or code, not the person who created it. 7articipants shouldn't ta"e any criticism personally. 0eave your egos, emotions, and sensitive eelings at the door. ?ollow -ules. 2 i/ed set o rules should be ollowed. They may set the amount o code to be reviewed =usually a couple hundred lines>, how much time will be spent =a couple hours>, what can be commented on, and so on. This is important so that the participants "now what their roles are and what they should e/pect. It helps the review run more smoothly. 7repare. 8ach participant is e/pected to prepare or and contribute to the review. ,epending on the type o review, participants may have di erent roles. They need to "now what their duties and responsibilities are and be ready to actively ul ill them at the review. !ost o the problems ound through the review process are ound during preparation, not at the actual review. %rite a -eport. The review group must produce a written report summarizing the results o the review and ma"e that report available to the rest o the product development team. It's imperative that others are told the results o the meetinghow many problems were ound, where they were ound, and so on.

%hat ma"es ormal reviews wor" is ollowing an established process. #aphazardly 3getting together to go over some code3 isn't su icient and may actually be detrimental. I a process is run in an ad-hoc ashion, bugs will be missed and the participants will li"ely eel that the e ort was a waste o time. I the reviews are run properly, they can prove to be a great way to ind bugs early. Thin" o them as one o the initial nets =see ?igure 5.1> that catches the big bugs at the beginning o the process. &ure, smaller bugs will still get through, but they'll be caught in the ne/t testing phases with the smaller nets with the tighter weave. 2igure E>1> 2ormal re3iews are the first nets use" in 'at'hing bugs>

In addition to inding problems, holding ormal reviews has a ew indirect results9

66

+ommunications. In ormation not contained in the ormal report is communicated. ?or e/ample, the blac"-bo/ testers can get insight into where problems may lie. Ine/perienced programmers may learn new techni)ues rom more e/perienced programmers. !anagement may get a better eel or how the pro:ect is trac"ing its schedule. Muality. 2 programmer's code that is being gone over in detail, unction by unction, line by line, o ten results in the programmer being more care ul. That's not to say that he would otherwise be sloppy:ust that i he "nows that his wor" is being care ully reviewed by his peers, he might ma"e an e/tra e ort to triple-chec" it to ma"e sure that it's right. Team +amaraderie. I a review is run properly, it can be a good place or testers and programmers to build respect or each other's s"ills and to better understand each other's :obs and :ob needs. &olutions. &olutions may be ound or tough problems, although whether they are discussed depends on the rules or the review. It may be more e ective to discuss solutions outside the review.

These indirect bene its shouldn't be relied on, but they do happen. .n many teams, or whatever reasons, the members end up wor"ing in isolation. ?ormal reviews are a great way to get them in the same room, all discussing the same pro:ect problems. Peer Re3iews The easiest way to get team members together and doing their irst ormal reviews o the so tware is through peer reviews, the least ormal method. &ometimes called buddy reviews, this method is really more o an 3I'll show you mine i you show me yours3 type discussion. 7eer reviews are o ten held with :ust the programmer who designed the architecture or wrote the code and one or two other programmers or testers acting as reviewers. That small group simply reviews the code together and loo"s or problems and oversights. To assure that the review is highly e ective =and doesn't turn into a co ee brea"> all the participants need to ma"e sure that the our "ey elements o a ormal review are in place9 0oo" or problems, ollow rules, prepare or the review, and write a report. (ecause peer reviews are in ormal, these elements are o ten scaled bac". &till, :ust getting together to discuss the code can ind bugs. #alkthroughs %al"throughs are the ne/t step up in ormality rom peer reviews. In a wal"through, the programmer who wrote the code ormally presents =wal"s through> it to a small group o ive or so other programmers and testers. The reviewers should receive copies o the so tware in advance o the review so they can e/amine it and write comments and )uestions that they want to as" at the review. #aving at least one senior programmer as a reviewer is very important. The presenter reads through the code line by line, or unction by unction, e/plaining what the code does and why. The reviewers listen and )uestion anything that loo"s suspicious. (ecause o the larger number o participants involved in a wal"through compared to a peer review, it's much more important or them to prepare or the review and to ollow the rules. It's also very important that a ter the review the presenter write a report telling what was ound and how he plans to address any bugs discovered. !nspe'tions

67

Inspections are the most ormal type o reviews. They are highly structured and re)uire training or each participant. Inspections are di erent rom peer reviews and wal"throughs in that the person who presents the code, the presenter or reader, isn't the original programmer. This orces someone else to learn and understand the material being presented, potentially giving a di erent slant and interpretation at the inspection meeting. The other participants are called inspectors. 8ach is tas"ed with reviewing the code rom a di erent perspective, such as a user, a tester, or a product support person. This helps bring di erent views o the product under review and very o ten identi ies di erent bugs. .ne inspector is even tas"ed with reviewing the code bac"wardthat is, rom the end to the beginningto ma"e sure that the material is covered evenly and completely. &ome inspectors are also assigned tas"s such as moderator and recorder to assure that the rules are ollowed and that the review is run e ectively. 2 ter the inspection meeting is held, the inspectors might meet again to discuss the de ects they ound and to wor" with the moderator to prepare a written report that identi ies the rewor" necessary to address the problems. The programmer then ma"es the changes and the moderator veri ies that they were properly made. ,epending on the scope and magnitude o the changes and on how critical the so tware is, a reinspection may be needed to locate any remaining bugs. Inspections have proven to be very e ective in inding bugs in any so tware deliverable, especially design documents and code, and are gaining popularity as companies and product development teams discover their bene its. Co"ing Stan"ar"s an" *ui"elines In ormal reviews, the inspectors are loo"ing or problems and omissions in the code. There are the classic bugs where something :ust won't wor" as written. These are best ound by care ul analysis o the codesenior programmers and testers are great at this. There are also problems where the code may operate properly but may not be written to meet a speci ic standard or guideline. It's e)uivalent to writing words that can be understood and get a point across but don't meet the grammatical and syntactical rules o the 8nglish language. &tandards are the established, i/ed, have-to- ollow-them rulesthe do's and don'ts. Fuidelines are the suggested best practices, the recommendations, the pre erred way o doing things. &tandards have no e/ceptions, short o a structured waiver process. Fuidelines can be a bit loose. It may sound strange that some piece o so tware may wor", may even be tested and shown to be very stable, but still be incorrect because it doesn't meet some criteria. It's important, though, and there are three reasons or adherence to a standard or guideline9 -eliability. It's been shown that code written to a speci ic standard or guideline is more reliable and secure than code that isn't. -eadabilityS!aintainability. +ode that ollows set standards and guidelines is easier to read, understand, and maintain. 7ortability. +ode o ten has to run on di erent hardware or be compiled with di erent compilers. I it ollows a set standard, it will li"ely be easieror even completely painlessto move it to a di erent plat orm.

The re)uirements or your pro:ect may range rom strict adherence to national or international standards to loose ollowing o internal team guidelines. %hat's important is that your team has some standards or guidelines or programming and that these are veri ied in a ormal review. E(amples of Programming Stan"ar"s an" *ui"elines

68

?igure 5.4 shows an e/ample o a programming standard that deals with the use o the + language -oto, while, and if-el'e statements. Improper use o these statements o ten results in buggy code, and most programming standards e/plicitly set rules or using them. 2igure E>7> 5 sample 'o"ing stan"ar" e(plains how se3eral language 'ontrol stru'tures shoul" be use"> (5"apte" from CII Programming *ui"elines by Thomas Plum an" %an Saks> Copyright 1//1. Plum =all. !n'>

The standard has our main parts9 Title describes what topic the standard covers. &tandard =or guideline> describes the standard or guideline e/plaining e/actly what's allowed and not allowed. Husti ication gives the reasoning behind the standard so that the programmer understands why it's good programming practice. 8/ample shows simple programming samples o how to use the standard. This isn't always necessary.

?igure 5.; is an e/ample o a guideline dealing with + language eatures used in +JJ. *ote how the language is a bit di erent. In this case, it starts out with 3Try to avoid.3 Fuidelines aren't as strict as standards, so there is some room or le/ibility i the situation calls or it. 2igure E>D> 5n e(ample of a programming gui"eline shows how to use 'ertain aspe'ts of C in CII> (5"apte" from CII Programming *ui"elines by Thomas Plum an" %an Saks> Copyright 1//1. Plum =all. !n'>

69

!T+S 5 )5TTER &2 ST6,E There are standards, there are guidelines, and then there is style. ?rom a so tware )uality and testing perspective, style doesn't matter. 8very programmer, :ust li"e every boo" author and artist, has his or her own uni)ue style. The rules may be ollowed, the language usage may be consistent, but it's still easy to tell who created what so tware. That di erentiating actor is style. In programming, it could be how verbose the commenting is or how the variables are named. It could be what indentation scheme is used in the loop constructs. It's the loo" and eel o the code. &ome teams do institute standards and guidelines or style aspects =such as indenting> so that the loo" and eel o the code doesn't become too random. 2s a so tware tester, when per orming ormal reviews on a piece o so tware, test and comment only on things that are wrong, missing, or don't adhere to the team's accepted standards or guidelines. 2s" yoursel i what you're about to report is really a problem or :ust di erence o opinion, a di erence o style. The latter isn't a bug. &btaining Stan"ar"s I your pro:ect, because o its nature, must ollow a set o programming standards, or i you're :ust interested in e/amining your so tware's code to see how well it meets a published standard or guideline, several sources are available or you to re erence. *ational and international standards or most computer languages and in ormation technology can be obtained rom9 2merican *ational &tandards Institute =2*&I>, www.ansi.org International 8ngineering +onsortium =I8+>, www.iec.org

70

International .rganization or &tandardization =I&.>, www.iso.ch *ational +ommittee or In ormation Technology &tandards =*+IT&>, www.ncits.org

There are also documents that demonstrate programming guidelines and best practices available rom pro essional organizations such as 2ssociation or +omputing !achinery =2+!>, www.acm.org Institute o 8lectrical and 8lectronics 8ngineers, Inc =I888>, www.ieee.org

Bou may also obtain in ormation rom the so tware vendor where you purchased your programming so tware. They o ten have published standards and guidelines available or ree or or a small ee. *eneri' Co"e Re3iew Che'klist The rest o this chapter on static white-bo/ testing covers some problems you should loo" or when veri ying so tware or a ormal code review. These chec"lists Q1R are in addition to comparing the code against a standard or a guideline and to ma"ing sure that the code meets the pro:ect's design re)uirements.
Q1R

These chec"list items were adapted rom &o tware Testing in the -eal %orld9 Improving the 7rocess, pp. 19A-4<1. +opyright 199G by 8dward 1it. $sed by permission o 7earson 8ducation 0imited, 0ondon. 2ll rights reserved. To really understand and apply these chec"s, you should have some programming e/perience. I you haven't done much programming, you might ind it use ul to read an introductory boo" such as &ams Teach Boursel (eginning 7rogramming in 44 #ours by &ams 7ublishing be ore you attempt to review program code in detail. %ata Referen'e Errors ,ata re erence errors are bugs caused by using a variable, constant, array, string, or record that hasn't been properly declared or initialized or how it's being used and re erenced. Is an uninitialized variable re erencedI 0oo"ing or omissions is :ust as important as loo"ing or errors. 2re array and string subscripts integer values and are they always within the bounds o the array's or string's dimensionI 2re there any potential 3o by one3 errors in inde/ing operations or subscript re erences to arraysI -emember the code in 0isting G.1 rom +hapter G. Is a variable used where a constant would actually wor" better or e/ample, when chec"ing the boundary o an arrayI Is a variable ever assigned a value that's o a di erent type than the variableI ?or e/ample, does the code accidentally assign a loating-point number to an integer variableI Is memory allocated or re erenced pointersI

71

I a data structure is re erenced in multiple unctions or subroutines, is the structure de ined identically in each oneI

*.T8 ,ata re erence errors are the primary cause o bu er overrunsthe bug behind many so tware security issues. +hapter 1;, 3Testing or &o tware &ecurity,3 covers this topic in more detail.

%ata %e'laration Errors ,ata declaration bugs are caused by improperly declaring or using variables or constants. 2re all the variables assigned the correct length, type, and storage classI ?or e/ample, should a variable be declared as a string instead o an array o charactersI I a variable is initialized at the same time as it's declared, is it properly initialized and consistent with its typeI 2re there any variables with similar namesI This isn't necessarily a bug, but it could be a sign that the names have been con used with those rom somewhere else in the program. 2re any variables declared that are never re erenced or are re erenced only onceI 2re all the variables e/plicitly declared within their speci ic moduleI I not, is it understood that the variable is shared with the ne/t higher moduleI

Computation Errors +omputational or calculation errors are essentially bad math. The calculations don't result in the e/pected result. ,o any calculations that use variables have di erent data types, such as adding an integer to a loating-point numberI ,o any calculations that use variables have the same data type but are di erent lengthsadding a byte to a word, or e/ampleI 2re the compiler's conversion rules or variables o inconsistent type or length understood and ta"en into account in any calculationsI Is the target variable o an assignment smaller than the right-hand e/pressionI Is over low or under low in the middle o a numeric calculation possibleI Is it ever possible or a divisorSmodulus to be zeroI ?or cases o integer arithmetic, does the code handle that some calculations, particularly division, will result in loss o precisionI +an a variable's value go outside its meaning ul rangeI ?or e/ample, could the result o a probability be less than <W or greater than 1<<WI

72

?or e/pressions containing multiple operators, is there any con usion about the order o evaluation and is operator precedence correctI 2re parentheses needed or clari icationI

Comparison Errors 0ess than, greater than, e)ual, not e)ual, true, alse. +omparison and decision errors are very susceptible to boundary condition problems. 2re the comparisons correctI It may sound pretty simple, but there's always con usion over whether a comparison should be less than or less than or e)ual to. 2re there comparisons between ractional or loating-point valuesI I so, will any precision problems a ect their comparisonI Is 1.<<<<<<<1 close enough to 1.<<<<<<<4 to be e)ualI ,oes each (oolean e/pression state what it should stateI ,oes the (oolean calculation wor" as e/pectedI Is there any doubt about the order o evaluationI 2re the operands o a (oolean operator (ooleanI ?or e/ample, is an integer variable containing integer values being used in a (oolean calculationI

Control 2low Errors +ontrol low errors are the result o loops and other control constructs in the language not behaving as e/pected. They are usually caused, directly or indirectly, by computational or comparison errors. I the language contains statement groups such as #e-i%...e%d and do...while, are the e%ds e/plicit and do they match their appropriate groupsI %ill the program, module, subroutine, or loop eventually terminateI I it won't, is that acceptableI Is there a possibility o premature loop e/itI Is it possible that a loop never e/ecutesI Is it acceptable i it doesn'tI I the program contains a multiway branch such as a 'wit)h...)$'e statement, can the inde/ variable ever e/ceed the number o branch possibilitiesI I it does, is this case handled properlyI 2re there any 3o by one3 errors that would cause une/pected low through the loopI

Subroutine Parameter Errors &ubroutine parameter errors are due to incorrect passing o data to and rom so tware subroutines. ,o the types and sizes o parameters received by a subroutine match those sent by the calling codeI Is the order correctI I a subroutine has multiple entry points =yuc">, is a parameter ever re erenced that isn't associated with the current point o entryI

73

I constants are ever passed as arguments, are they accidentally changed in the subroutineI ,oes a subroutine alter a parameter that's intended only as an input valueI ,o the units o each parameter match the units o each corresponding argument8nglish versus metric, or e/ampleI I global variables are present, do they have similar de initions and attributes in all re erencing subroutinesI

!nputJ&utput Errors These errors include anything related to reading rom a ile, accepting input rom a "eyboard or mouse, and writing to an output device such as a printer or screen. The items presented here are very simpli ied and generic. Bou should adapt and add to them to properly cover the so tware you're testing. ,oes the so tware strictly adhere to the speci ied ormat o the data being read or written by the e/ternal deviceI I the ile or peripheral isn't present or ready, is that error condition handledI ,oes the so tware handle the situation o the e/ternal device being disconnected, not available, or ull during a read or writeI 2re all conceivable errors handled by the so tware in an e/pected wayI #ave all error messages been chec"ed or correctness, appropriateness, grammar, and spellingI

&ther Che'ks This best-o -the-rest list de ines a ew items that didn't it well in the other categories. It's not by any means complete, but should give you ideas or speci ic items that should be added to a list tailored or your so tware pro:ect. %ill the so tware wor" with languages other than 8nglishI ,oes it handle e/tended 2&+II charactersI ,oes it need to use $nicode instead o 2&+III I the so tware is intended to be portable to other compilers and +7$s, have allowances been made or thisI 7ortability, i re)uired, can be a huge issue i not planned and tested or. #as compatibility been considered so that the so tware will operate with di erent amounts o available memory, di erent internal hardware such as graphics and sound cards, and di erent peripherals such as printers and modemsI ,oes compilation o the program produce any 3warning3 or 3in ormational3 messagesI They usually indicate that something )uestionable is being done. 7urists would argue that any warning message is unacceptable.

Summary

74

8/amining the codestatic white-bo/ testinghas proven to be an e ective means or inding bugs early. It's a tas" that re)uires a great deal o preparation to ma"e it a productive e/ercise, but many studies have shown that the time spent is well worth the bene its gained. To ma"e it even more attractive, commercial so tware products, "nown as static analyzers, are available to automate a great deal o the wor". The so tware reads in a program's source iles and chec"s them against published standards and your own customizable guidelines. +ompilers have also improved to the point that i you enable all their levels o error chec"ing, they will catch many o the problems listed previously in the generic code review chec"list. &ome will even disallow use o unctions with "nown security issues. These tools don't eliminate the tas"s o code reviews or inspectionsthey :ust ma"e it easier to accomplish and give testers more time to loo" even deeper or bugs. I your team currently isn't doing testing at this level and you have some e/perience at programming, you might try suggesting it as a process to investigate. 7rogrammers and managers may be apprehensive at irst, not "nowing i the bene its are that greatit's hard to claim, or e/ample, that inding a bug during an inspection saved your pro:ect ive days over inding it months later during blac"-bo/ testing. (ut, static white-bo/ testing is gaining momentum, and in some circles, pro:ects can't ship reliable so tware without it. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. *ame several advantages to per orming static white-bo/ testing 2. True or ?alse9 &tatic white-bo/ testing can ind missing items as well as problems. ;. %hat "ey element ma"es ormal reviews wor"I 4. (esides being more ormal, what's the big di erence between inspections and other types o reviewsI G. I a programmer was told that he could name his variables with only eight characters and the irst character had to be capitalized, would that be a standard or a guidelineI 5. &hould you adopt the code review chec"list rom this chapter as your team's standard to veri y its codeI 7. 2 common security issue "nown as a bu er overrun is in a class o errors "nown as whatI They are caused by whatI

Chapter '. Testing the Software with () a* +lasses I* T#I& +#27T8 ,ynamic %hite-(o/ Testing ,ynamic %hite-(o/ Testing 6ersus ,ebugging Testing the 7ieces ,ata +overage

75

+ode +overage

&o ar in 7art II, 3Testing ?undamentals,3 you've learned about three o the our undamental testing techni)ues9 static blac" bo/ =testing the speci ication>, dynamic blac" bo/ =testing the so tware>, and static white bo/ =e/amining the code>. In this chapter, you'll learn the ourth undamental techni)uedynamic white-bo/ testing. Bou'll loo" into the so tware 3bo/3 with your O-ray glasses as you test the so tware. In addition to your O-ray specs, you'll also need to wear your programmer's hati you have one. I you don't own one, don't be scared o . The e/amples used aren't that comple/ and i you ta"e your time, you'll be able to ollow them. Faining even a small grasp o this type o testing will ma"e you a much more e ective blac"-bo/ tester. I you do have some programming e/perience, consider this chapter an introduction to a very wide-open testing ield. !any so tware companies are hiring testers speci ically to per orm low-level testing o their so tware. They're loo"ing or people with both programming and testing s"ills, which is o ten a rare mi/ and highly sought a ter. #ighlights rom this chapter include %hat dynamic white-bo/ testing is The di erence between debugging and dynamic white-bo/ testing %hat unit and integration testing are #ow to test low-level unctions The data areas that need to be tested at a low level #ow to orce a program to operate a certain way %hat di erent methods you can use to measure the thoroughness o your testing

%ynami' #hite-Bo( Testing (y now you should be very amiliar with the terms static, dynamic, white bo/, and blac" bo/. 1nowing that this chapter is about dynamic white-bo/ testing should tell you e/actly what material it covers. &ince it's dynamic, it must be about testing a running program and since it's white-bo/, it must be about loo"ing inside the bo/, e/amining the code, and watching it as it runs. It's li"e testing the so tware with O-ray glasses. ,ynamic white-bo/ testing, in a nutshell, is using in ormation you gain rom seeing what the code does and how it wor"s to determine what to test, what not to test, and how to approach the testing. 2nother name commonly used or dynamic white-bo/ testing is structural testing because you can see and use the underlying structure o the code to design and run your tests. %hy would it be bene icial or you to "now what's happening inside the bo/, to understand how the so tware wor"sI +onsider ?igure 7.1. This igure shows two bo/es that per orm the basic calculator operations o addition, subtraction, multiplication, and division. 2igure 9>1> 6ou woul" 'hoose "ifferent test 'ases if you knew that one bo( 'ontaine" a 'omputer an" the other a person with a pen'il an" paper>

76

I you didn't "now how the bo/es wor"ed, you would apply the dynamic blac"-bo/ testing techni)ues you learned in +hapter G, 3Testing the &o tware with (linders .n.3 (ut, i you could loo" in the bo/es and see that one contained a computer and the other contained a person with a pencil and paper, you would probably choose a completely di erent test approach or each one. . course, this e/ample is very simplistic, but it ma"es the point that "nowing how the so tware operates will in luence how you test. ,ynamic white-bo/ testing isn't limited :ust to seeing what the code does. It also can involve directly testing and controlling the so tware. The our areas that dynamic white-bo/ testing encompasses are ,irectly testing low-level unctions, procedures, subroutines, or libraries. In !icroso t %indows, these are called 2pplication 7rogramming Inter aces =27Is>. Testing the so tware at the top level, as a completed program, but ad:usting your test cases based on what you "now about the so tware's operation. Faining access to read variables and state in ormation rom the so tware to help you determine whether your tests are doing what you thought. 2nd, being able to orce the so tware to do things that would be di icult i you tested it normally. !easuring how much o the code and speci ically what code you 3hit3 when you run your tests and then ad:usting your tests to remove redundant test cases and add missing ones.

8ach area is discussed in the remainder o this chapter. Thin" about them as you read on and consider how they might be used to test so tware that you're amiliar with. %ynami' #hite-Bo( Testing :ersus %ebugging It's important not to con use dynamic white-bo/ testing with debugging. I you've done some programming, you've probably spent many hours debugging code that you've written. The two techni)ues may appear similar because they both involve dealing with so tware bugs and loo"ing at the code, but they're very di erent in their goals =see ?igure 7.4>. 2igure 9>7> %ynami' white-bo( testing an" "ebugging ha3e "ifferent goals but they "o o3erlap in the mi""le>

77

The goal o dynamic white-bo/ testing is to ind bugs. The goal o debugging is to i/ them. They do overlap, however, in the area o isolating where and why the bug occurs. Bou'll learn more about this in +hapter 19, 3-eporting %hat Bou ?ind,3 but or now, thin" o the overlap this way. 2s a so tware tester, you should narrow down the problem to the simplest test case that demonstrates the bug. I it's white-bo/ testing, that could even include in ormation about what lines o code loo" suspicious. The programmer who does the debugging pic"s the process up rom there, determines e/actly what is causing the bug, and attempts to i/ it. *.T8 I you're per orming this low-level testing, you will use many o the same tools that programmers use. I the program is compiled, you will use the same compiler but possibly with di erent settings to enable better error detection. Bou will li"ely use a code-level debugger to single-step through the program, watch variables, set brea" conditions, and so on. Bou may also write your own programs to test separate code modules given to you to validate. Testing the Pie'es -ecall rom +hapter 4, 3The &o tware ,evelopment 7rocess,3 the various models or so tware development. The big-bang model was the easiest but the most chaotic. 8verything was put together at once and, with ingers crossed, the team hoped that it all wor"ed and that a product would be born. (y now you've probably deduced that testing in such a model would be very di icult. 2t most, you could per orm dynamic blac"-bo/ testing, ta"ing the near inal product in one entire blob and e/ploring it to see what you could ind. Bou've learned that this approach is very costly because the bugs are ound late in the game. ?rom a testing perspective, there are two reasons or the high cost9 It's di icult and sometimes impossible to igure out e/actly what caused the problem. The so tware is a huge -ube Foldberg machine that doesn't wor"the ball drops in one side, but buttered toast and hot co ee doesn't come out the other. There's no way to "now which little piece is bro"en and causing the entire contraption to ail. &ome bugs hide others. 2 test might ail. The programmer con idently debugs the problem and ma"es a i/, but when the test is rerun, the so tware still ails. &o many problems were piled one on top the other that it's impossible to get to the core ault.

<nit an" !ntegration Testing The way around this mess is, o course, to never have it happen in the irst place. I the code is built and tested in pieces and gradually put together into larger and larger portions, there won't be any surprises when the entire product is lin"ed together =see ?igure 7.;>. 2igure 9>D> !n"i3i"ual pie'es of 'o"e are built up an" teste" separately. an" then integrate" an" teste" again>

78

Testing that occurs at the lowest level is called unit testing or module testing. 2s the units are tested and the low-level bugs are ound and i/ed, they are integrated and integration testing is per ormed against groups o modules. This process o incremental testing continues, putting together more and more pieces o the so tware until the entire productor at least a ma:or portion o itis tested at once in a process called system testing. %ith this testing strategy, it's much easier to isolate bugs. %hen a problem is ound at the unit level, the problem must be in that unit. I a bug is ound when multiple units are integrated, it must be related to how the modules interact. . course, there are e/ceptions to this, but by and large, testing and debugging is much more e icient than testing everything at once. There are two approaches to this incremental testing9 bottom-up and top-down. In bottom-up testing =see ?igure 7.4>, you write your own modules, called test drivers, that e/ercise the modules you're testing. They hoo" in e/actly the same way that the uture real modules will. These drivers send test-case data to the modules under test, read bac" the results, and veri y that they're correct. Bou can very thoroughly test the so tware this way, eeding it all types and )uantities o data, even ones that might be di icult to send i done at a higher level. 2igure 9>0> 5 test "ri3er 'an repla'e the real software an" more effi'iently test a low-le3el mo"ule>

Top-down testing may sound li"e big-bang testing on a smaller scale. 2 ter all, i the higherlevel so tware is complete, it must be too late to test the lower modules, rightI 2ctually, that's not )uite true. 0oo" at ?igure 7.G. In this case, a low-level inter ace module is used to collect temperature data rom an electronic thermometer. 2 display module sits right above the inter ace, reads the data rom the inter ace, and displays it to the user. To test the top-level display module, you'd need blow torches, water, ice, and a deep reeze to change the temperature o the sensor and have that data passed up the line. 2igure 9>1> 5 test stub sen"s test "ata up to the mo"ule being teste">

79

-ather than test the temperature display module by attempting to control the temperature o the thermometer, you could write a small piece o code called a stub that acts :ust li"e the inter ace module by eeding 3 a"e3 temperature values rom a ile directly to the display module. The display module would read the data and show the temperature :ust as though it was reading directly rom a real thermometer inter ace module. It wouldn't "now the di erence. %ith this test stub con iguration, you could )uic"ly run through numerous test values and validate the operation o the display module. 5n E(ample of )o"ule Testing 2 common unction available in many compilers is one that converts a string o 2&+II characters into an integer value. %hat this unction does is ta"e a string o numbers, or J signs, and possible e/traneous characters such as spaces and letters, and converts them to a numeric value or e/ample, the string 314;4G3 gets converted to the number 14,;4G. It's a airly common unction that's o ten used to process values that a user might type into a dialog bo/ or e/ample, someone's age or an inventory count. The + language unction that per orms this operation is $toi(), which stands or 32&+II to Integer.3 ?igure 7.5 shows the speci ication or this unction. I you're not a + programmer, don't ret. 8/cept or the irst line, which shows how to ma"e the unction call, the spec is in 8nglish and could be used or de ining the same unction or any computer language. 2igure 9>E> The spe'ifi'ation sheet for the C language atoi ! fun'tion>

80

I you're the so tware tester assigned to per orm dynamic white-bo/ testing on this module, what would you doI ?irst, you would probably decide that this module loo"s li"e a bottom module in the program, one that's called by higher up modules but doesn't call anything itsel . Bou could con irm this by loo"ing at the internal code. I this is true, the logical approach is to write a test driver to e/ercise the module independently rom the rest o the program. This test driver would send test strings that you create to the $toi() unction, read bac" the return values or those strings, and compare them with your e/pected results. The test driver would most li"ely be written in the same language as the unctionin this case, +but it's also possible to write the driver in other languages as long as they inter ace to the module you're testing. This test driver can ta"e on several orms. It could be a simple dialog bo/, as shown in ?igure 7.7, that you use to enter test strings and view the results. .r it could be a standalone program that reads test strings and e/pected results rom a ile. The dialog bo/, being user driven, is very interactive and le/ibleit could be given to a blac"-bo/ tester to use. (ut the standalone driver can be very ast reading and writing test cases directly rom a ile. 2igure 9>9> 5 "ialog bo( test "ri3er 'an be use" to sen" test 'ases to a mo"ule being teste">

81

*e/t, you would analyze the speci ication to decide what blac"-bo/ test cases you should try and then apply some e)uivalence partitioning techni)ues to reduce the total set =remember +hapter GI>. Table 7.1 shows e/amples o a ew test cases with their input strings and e/pected output values. This table isn't intended to be a comprehensive list.

Table ".1. Sam#le ASCII to Integer Con$ersion Test Cases In#ut String 313 313 3J13 3<3 3<3 3J<3 31.43 34;3 3abc3 3a14;3 and so on %ut#ut Integer Value 1 1 1 < < < 1 4 < <

0astly, you would loo" at the code to see how the unction was implemented and use your white-bo/ "nowledge o the module to add or remove test cases. *.T8 +reating your blac"-bo/ testing cases based on the speci ication, be ore your white-bo/ cases, is important. That way, you are truly testing what the module is intended to do. I you irst create your test cases based on a white-bo/ view o the module, by e/amining the code, you will be biased into creating test cases based on how the module wor"s. The programmer could have misinterpreted the speci ication and your test cases would then be wrong. They would be precise, per ectly testing the module, but they wouldn't be accurate because they wouldn't be testing the intended operation.

2dding and removing test cases based on your white-bo/ "nowledge is really :ust a urther re inement o the e)uivalence partitions done with inside in ormation. Bour original blac"-bo/ test cases might have assumed an internal 2&+II table that would ma"e cases such as 3a14;3 and 3z14;3 di erent and important. 2 ter e/amining the so tware, you could ind that instead o an 2&+II table, the programmer simply chec"ed or numbers, and J signs, and blan"s. %ith that in ormation, you might decide to remove one o these cases because both o them are in the same e)uivalence partition.

82

%ith close inspection o the code, you could discover that the handling o the J and signs loo"s a little suspicious. Bou might not even understand how it wor"s. In that situation, you could add a ew more test cases with embedded J and signs, :ust to be sure. %ata Co3erage The previous e/ample o white-bo/ testing the $toi() unction was greatly simpli ied and glossed over some o the details o loo"ing at the code to decide what ad:ustments to ma"e to the test cases. In reality, there's )uite a bit more to the process than :ust perusing the so tware or good ideas. The logical approach is to divide the code :ust as you did in blac"-bo/ testinginto its data and its states =or program low>. (y loo"ing at the so tware rom the same perspective, you can more easily map the white-bo/ in ormation you gain to the blac"-bo/ cases you've already written. +onsider the data irst. ,ata includes all the variables, constants, arrays, data structures, "eyboard and mouse input, iles and screen input and output, and IS. to other devices such as modems, networ"s, and so on. %ata 2low ,ata low coverage involves trac"ing a piece o data completely through the so tware. 2t the unit test level this would :ust be through an individual module or unction. The same trac"ing could be done through several integrated modules or even through the entire so tware productalthough it would be more time-consuming to do so. I you test a unction at this low level, you would use a debugger and watch variables to view the data as the program runs =see ?igure 7.A>. %ith blac"-bo/ testing, you only "now what the value o the variable is at the beginning and at the end. %ith dynamic white-bo/ testing you could also chec" intermediate values during program e/ecution. (ased on what you see you might decide to change some o your test cases to ma"e sure the variable ta"es on interesting or even ris"y interim values. 2igure 9>H> 5 "ebugger an" wat'h 3ariables 'an help you tra'e a 3ariable+s 3alues through a program>

Sub-Boun"aries &ub-boundaries were discussed in +hapter G in regard to embedded 2&+II tables and powers-o two. These are probably the most common e/amples o sub-boundaries that can cause bugs,

83

but every piece o so tware will have its own uni)ue sub-boundaries, too. #ere are a ew more e/amples9 2 module that computes ta/es might switch rom using a data table to using a ormula at a certain inancial cut-o point. 2n operating system running low on -2! may start moving data to temporary storage on the hard drive. This sub-boundary may not even be i/ed. It may change depending on how much space remains on the dis". To gain better precision, a comple/ numerical analysis program may switch to a di erent e)uation or solving the problem depending on the size o the number.

I you per orm white-bo/ testing, you need to e/amine the code care ully to loo" or subboundary conditions and create test cases that will e/ercise them. 2s" the programmer who wrote the code i she "nows about any o these situations and pay special attention to internal tables o data because they're especially prone to sub-boundary conditions. 2ormulas an" ECuations 6ery o ten, ormulas and e)uations are buried deep in the code and their presence or e ect isn't always obvious rom the outside. 2 inancial program that computes compound interest will de initely have this ormula somewhere in the so tware9 2L7=1JrSn>nt where 7 L principal amount r L annual interest rate n L number o times the interest is compounded per year t L number o years 2 L amount o money a ter time t 2 good blac"-bo/ tester would hope ully choose a test case o nL<, but a white-bo/ tester, a ter seeing the ormula in the code, would "now to try nL< because that would cause the ormula to blow up with a divide-by-zero error. (ut, what i n was the result o another computationI !aybe the so tware sets the value o n based on other user input or algorithmically tries di erent n values in an attempt to ind the lowest payment. Bou need to as" yoursel i there's any way that n can ever become zero and igure out what inputs to eed the program to ma"e that happen. TI7 &cour your code or ormulas and e)uations, loo" at the variables they use, and create test cases and e)uivalence partitions or them in addition to the normal inputs and outputs o the program.

Error 2or'ing

84

The last type o data testing covered in this chapter is error orcing. I you're running the so tware that you're testing in a debugger, you don't :ust have the ability to watch variables and see what values they holdyou can also orce them to speci ic values. In the preceding compound interest calculation, i you couldn't ind a direct way to set the number o compoundings =n> to zero, you could use your debugger to orce it to zero. The so tware would then have to handle itPor not. *.T8 (e care ul i you use error orcing and ma"e sure you aren't creating a situation that can never happen in the real world. I the programmer chec"ed that n was greater than zero at the top o the unction and n was never used until the ormula, setting it to zero and causing the so tware to ail would be an invalid test case. I you ta"e care in selecting your error orcing scenarios and double-chec" with the programmer to assure that they're valid, error orcing can be an e ective tool. Bou can e/ecute test cases that would otherwise be di icult to per orm. 2&RC!4* ERR&R )ESS5*ES 2 great way to use error orcing is to cause all the error messages in your so tware to appear. !ost so tware uses internal error codes to represent each error message. %hen an internal error condition lag is set, the error handler ta"es the variable that holds the error code, loo"s up the code in a table, and displays the appropriate message. !any errors are di icult to createli"e hoo"ing up 4,<49 printers. (ut i all you want to do is test that the error messages are correct =spelling, language, ormatting, and so on>, using error orcing can be a very e icient way to see all o them. 1eep in mind, though, that you aren't testing the code that detects the error, :ust the code that displays it. Co"e Co3erage 2s with blac"-bo/ testing, testing the data is only hal the battle. ?or comprehensive coverage you must also test the program's states and the program's low among them. Bou must attempt to enter and e/it every module, e/ecute every line o code, and ollow every logic and decision path through the so tware. This type o testing is "nown as code coverage testing. +ode coverage is dynamic white-bo/ testing because it re)uires you to have ull access to the code to view what parts o the so tware you pass through when you run your test cases. The simplest orm o code coverage testing is using your compiler's debugger to view the lines o code you visit as you single-step through the program. ?igure 7.9 shows an e/ample o the 6isual (asic debugger in operation. 2igure 9>/> The "ebugger allows you to single-step through the software to see what lines of 'o"e an" mo"ules you e(e'ute while running your test 'ases>

85

?or very small programs or individual modules, using a debugger is o ten su icient. #owever, per orming code coverage on most so tware re)uires a specialized tool "nown as a code coverage analyzer. ?igure 7.1< shows an e/ample o such a tool. 2igure 9>18> 5 'o"e 'o3erage analyBer pro3i"es "etaile" information about how effe'ti3e your test 'ases are> (This figure is 'opyright an" 'ourtesy of Bullseye Testing Te'hnology>

+ode coverage analyzers hoo" into the so tware you're testing and run transparently in the bac"ground while you run your test cases. 8ach time a unction, a line o code, or a logic decision is e/ecuted, the analyzer records the in ormation. Bou can then obtain statistics that identi y which portions o the so tware were e/ecuted and which portions weren't. %ith this data you'll "now %hat parts o the so tware your test cases don't cover. I a speci ic module is never e/ecuted, you "now that you need to write additional test cases or testing that module's unction. %hich test cases are redundant. I you run a series o test cases and they don't increase the percentage o code covered, they are li"ely in the same e)uivalence partition.

86

%hat new test cases need to be created or better coverage. Bou can loo" at the code that has low coverage, see how it wor"s and what it does, and create new test cases that will e/ercise it more thoroughly.

Bou will also have a general eel or the )uality o the so tware. I your test cases cover 9< percent o the so tware and don't ind any bugs, the so tware is in pretty good shape. I , on the other hand, your tests cover only G< percent o the so tware and you're still inding bugs, you "now you still have wor" to do. -8!I*,8,on't orget about the 7esticide 7arado/ described in +hapter ;, 3The -ealities o &o tware Testing3the more you test the so tware, the more it becomes immune to your tests. I your test cases cover 9< percent o the so tware and you're not inding any bugs, it could still be that your so tware isn't in good shapeit could be that it has become immune. 2dding new test cases may reveal that the ne/t 1< percent is very buggyN Program Statement an" ,ine Co3erage The most straight orward orm o code coverage is called statement coverage or line coverage. I you're monitoring statement coverage while you test your so tware, your goal is to ma"e sure that you e/ecute every statement in the program at least once. In the case o the short program shown in 0isting 7.1, 1<< percent statement coverage would be the e/ecution o lines 1 through 4. ,isting 9>1> !t+s :ery Easy to Test E3ery ,ine of This Simple Program 1 2 3 4 &@BDC <Eello .orld< &@BDC <Che d$te i' <F 8$teG &@BDC <Che tiAe i' <F CiAeG ED8

Bou might thin" this would be the per ect way to ma"e sure that you tested your program completely. Bou could run your tests and add test cases until every statement in the program is touched. $n ortunately, statement coverage is misleading. It can tell you i every statement is e/ecuted, but it can't tell you i you've ta"en all the paths through the so tware. Bran'h Co3erage 2ttempting to cover all the paths in the so tware is called path testing. The simplest orm o path testing is called branch coverage testing. +onsider the program shown in 0isting 7.4. ,isting 9>7> The I& Statement Creates 5nother Bran'h Through the Co"e 1 2 3 4 5 6 7 &@BDC <Eello .orld< B( 8$teG 5 <31-31-2333< CEED &@BDC <E$22" Dew He$r< ED8 B( &@BDC <Che d$te i' <F 8$teG &@BDC <Che tiAe i' <F CiAeG ED8

I you test this program with the goal o 1<< percent statement coverage, you would need to run only a single test case with the 8$teG variable set to Hanuary 1, 4<<<. The program would then e/ecute the ollowing path9

87

0ines 1, 4, ;, 4, G, 5, 7 Bour code coverage analyzer would state that you tested every statement and achieved 1<< percent coverage. Bou could )uit testing, rightI %rongN Bou may have tested every statement, but you didn't test every branch. Bour gut may be telling you that you still need to try a test case or a date that's not Hanuary 1, 4<<<. I you did, the program would e/ecute the other path through the program9 0ines 1, 4, G, 5, 7 !ost code coverage analyzers will account or code branches and report both statement coverage and branch coverage results separately, giving you a much better idea o your test's e ectiveness. Con"ition Co3erage Hust when you thought you had it all igured out, there's yet another complication to path testing. 0isting 7.; shows a slight variation to 0isting 7.4. 2n e/tra condition is added to the B( statement in line 4 that chec"s the time as well as the date. +ondition coverage testing ta"es the e/tra conditions on the branch statements into account. ,isting 9>D> The )ultiple Con"itions in the I& Statement Create )ore Paths Through the Co"e 1 2 3 4 5 6 7 &@BDC <Eello .orld< B( 8$teG 5 <31-31-2333< 0D8 CiAeG 5 <33 33 33< CEED &@BDC <E$22" Dew He$r< ED8 B( &@BDC <Che d$te i' <F 8$teG &@BDC <Che tiAe i' <F CiAeG ED8

In this sample program, to have ull condition coverage testing, you need to have the our sets o test cases shown in Table 7.4. These cases assure that each possibility in the B( statement are covered. Table ".2. Test Cases to Achie$e &ull Con'ition Co$erage (ate) <1-<1-1999 <1-<1-1999 <1-<1-4<<< <1-<1-4<<< Time) 11911911 <<9<<9<< 11911911 <<9<<9<< *ine + ,-ecution 1,4,G,5,7 1,4,G,5,7 1,4,G,5,7 1,4,;,4,G,5,7

I you were concerned only with branch coverage, the irst three conditions would be redundant and could be e)uivalence partitioned into a single test case. (ut, with condition coverage testing, all our cases are important because they e/ercise the di erent conditions o the B( statement in line 4?alse-?alse, ?alse-True, True-?alse, and True-True.

88

2s with branch coverage, code coverage analyzers can be con igured to consider conditions when reporting their results. I you test or condition coverage, you will achieve branch coverage and there ore achieve statement coverage. *.T8 I you manage to test every statement, branch, and condition =and that's impossible e/cept or the smallest o programs>, you still haven't tested the program completely. -emember, all the data errors discussed in the irst part o this chapter are still possible. The program low and the data together ma"e up the operation o the so tware. Summary This chapter showed you how having access to the so tware's source code while the program is running can open up a whole new area o so tware testing. ,ynamic white-bo/ testing is a very power ul approach that can greatly reduce your test wor" by giving you 3inside3 in ormation about what to test. (y "nowing the details o the code, you can eliminate redundant test cases and add test cases or areas you didn't initially consider. 8ither way, you can greatly improve your testing e ectiveness. +hapters 4 through 7 covered the undamentals o so tware testing9 &tatic blac"-bo/ testing involves e/amining the speci ication and loo"ing or problems be ore they get written into the so tware. ,ynamic blac"-bo/ testing involves testing the so tware without "nowing how it wor"s. &tatic white-bo/ testing involves e/amining the details o the written code through ormal reviews and inspections. ,ynamic white-bo/ testing involves testing the so tware when you can see how it wor"s and basing your tests on that in ormation.

In a sense, this is all there is to so tware testing. . course, reading about it in our chapters and putting it into practice are very di erent things. (eing a good so tware tester re)uires lots o dedication and hard wor". It ta"es practice and e/perience to "now when and how to best apply these undamental techni)ues. In 7art III, 32pplying Bour Testing &"ills,3 you'll learn about di erent types o so tware testing and how you can apply the s"ills rom your 3blac" and white testing bo/3 to real-world scenarios. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. %hy does "nowing how the so tware wor"s in luence how and what you should testI 4. %hat's the di erence between dynamic white-bo/ testing and debuggingI ;. %hat are two reasons that testing in a big-bang so tware development model is nearly impossibleI #ow can these be addressedI

4. True or ?alse9 I your product development is in a hurry, you can s"ip module testing
and proceed directly to integration testing. G. %hat's the di erence between a test stub and a test driverI

6. True or ?alse9 2lways design your blac"-bo/ test cases irst. 89

7. . the three code coverage measures described, which one is the bestI %hyI A. %hat's the biggest problem o white-bo/ testing, either static or dynamicI

Chapter H> Configuration Testing I* T#I& +#27T8 2n .verview o +on iguration Testing 2pproaching the Tas" .btaining the #ardware Identi ying #ardware &tandards +on iguration Testing .ther #ardware

0i e could be so simple. 2ll computer hardware could be identical. 2ll so tware could be written by the same company. There wouldn't be con using option buttons to clic" or chec" bo/es to chec". 8verything would inter ace per ectly the irst time, every time. #ow boring. In the real world, G<,<<<-s)uare- oot computer superstores are o ering 7+s, printers, monitors, networ" cards, modems, scanners, digital cameras, peripherals, net-cams, and hundreds o other computer doodads rom thousands o companiesall able to connect to your 7+N I you're :ust getting started at so tware testing, one o your irst tas"s may be con iguration testing. Bou'll be ma"ing sure that your so tware wor"s with as many di erent hardware combinations as possible. I you're not testing so tware or a popular hardware plat orm such as a 7+ or a !acthat is, i you're testing some specialized proprietary systemyou will still need to consider con iguration issues. Bou can easily tailor what you learn in this chapter to your situation. The irst part o this chapter deals with the generalities o 7+ con iguration testing and then moves into the speci ics o testing printers, display adapters =video cards>, and sound cards or a 7+. 2lthough the e/amples are based on des"top computers, you can e/trapolate the methods to :ust about any type o con iguration test problem. *ew and di erent devices are released every day, and it will be your :ob to igure out how to test them. #ighlights o this chapter include %hy con iguration testing is necessary %hy con iguration testing can be a huge :ob 2 basic approach to con iguration testing #ow to ind the hardware you need to test with %hat to do i you're not testing so tware or a des"top computer

5n &3er3iew of Configuration Testing The ne/t time you're in one o those computer superstores, loo" at a ew so tware bo/es and read over the system re)uirements. Bou'll see things such as 7+ with a 7entium 4 processor,

90

1<44/75A ;4-bit color monitor, ;4-bit audio card, game port, and so on. +on iguration testing is the process o chec"ing the operation o the so tware you're testing with all these various types o hardware. +onsider the di erent con iguration possibilities or a standard %indows-based 7+ used in homes and businesses9 The 7+. There are several well-"nown computer manu acturers, such as ,ell, Fateway, #ewlett 7ac"ard, and others. 8ach one builds 7+s using components designed themselves or obtained rom other manu acturers. !any hobbyists even build their own 7+s using o -the-shel components available at computer superstores. +omponents. !ost 7+s are modular and built up rom various system boards, component cards, and other internal devices such as dis" drives, +,--.! drives, ,6, burners, video, sound, a/ modem, and networ" cards =see ?igure A.1>. There are T6 tuner cards and specialized cards or video capture and home automation. There are even inputSoutput cards that can give a 7+ the ability to control a small actoryN These internal devices are built by hundreds o di erent manu acturers. 2igure H>1> 4umerous internal 'omponents make up a PC+s 'onfiguration>

7eripherals. 7eripherals, shown in ?igure A.4, are the printers, scanners, mice, "eyboards, monitors, cameras, :oystic"s, and other devices that plug into your system and operate e/ternally to the 7+. 2igure H>7> 5 PC 'an 'onne't to a wi"e assortment of peripherals>

91

Inter aces. The components and peripherals plug into your 7+ through various types o inter ace connectors =see ?igure A.;>. These inter aces can be internal or e/ternal to the 7+. Typical names or them are I&2, 7+I, $&(, 7&S4, -&S4;4, -H-11, -H-4G, and ?irewire. There are so many di erent possibilities that hardware manu acturers will o ten create the same peripheral with di erent inter aces. It's possible to buy the e/act same mouse in three di erent con igurationsN 2igure H>D> The ba'k of a PC shows numerous interfa'e 'onne'tors for atta'hing peripherals>

92

.ptions and memory. !any components and peripherals can be purchased with di erent hardware options and memory sizes. 7rinters can be upgraded to support e/tra onts or accept more memory to speed up printing. Fraphics cards with more memory can support additional colors and higher resolutions. The system board can have di erent versions o (I.& =its (asic Input .utput &ystem> and, o course, various amounts o memory. ,evice ,rivers. 2ll components and peripherals communicate with the operating system and the so tware applications through low-level so tware called device drivers. These drivers are o ten provided by the hardware device manu acturer and are installed when you set up the hardware. 2lthough technically they are so tware, or testing purposes they are considered part o the hardware con iguration.

I you're a tester gearing up to start con iguration testing on a piece o so tware, you need to consider which o these con iguration areas would be most closely tied to the program. 2 highly graphical computer game will re)uire lots o attention to the video and sound areas. 2 greeting card program will be especially vulnerable to printer issues. 2 a/ or communications program will need to be tested with numerous modems and networ" con igurations. Bou may be wondering why this is all necessary. 2 ter all, there are standards to meet or building hardware, whether it's or an o -the-shel 7+ or a specialized computer in a hospital. Bou would e/pect that i everyone designed their hardware to a set o standards, so tware would :ust wor" with it without any problems. In an ideal world, that would happen, but un ortunately, standards aren't always ollowed. &ometimes, the standards are airly loosecall them guidelines. +ard and peripheral manu acturers are always in tight competition with one another and re)uently bend the rules to s)ueeze in an e/tra eature or to get in a last little bit o per ormance gain. . ten the device drivers are rushed and pac"ed into the bo/ as the hardware goes out the door. The result is so tware that doesn't wor" correctly with certain hardware con igurations. !solating Configuration Bugs Those con iguration bugs can bite hard. -emember the ,isney 0ion 1ing bug described in +hapter 1, 3&o tware Testing (ac"ground3I That was a con iguration problem. The so tware's sound didn't wor" only on a ew, but very popular, hardware con igurations. I you've ever been playing a game or using a graphics program and the colors suddenly go crazy or pieces o windows get le t behind as you drag them, you've probably discovered a display adapter con iguration bug. I you've ever spent hours =or daysN> trying to get an old program to wor" with your new printer, it's probably a con iguration bug. *.T8 The sure way to tell i a bug is a con iguration problem and not :ust a bug that would occur in any con iguration is to per orm the e/act same operation that caused the problem, step by step, on another computer with a completely di erent hardware setup. I the bug doesn't occur, it's very li"ely a speci ic con iguration problem that's revealed by the uni)ue hardware used in the test. 2ssume that you test your so tware on a uni)ue con iguration and discover a problem. %ho should i/ the bugyour team or the hardware manu acturerI That could turn out to be a milliondollar )uestion. ?irst you need to igure out where the problem lies. This is usually a dynamic white-bo/ testing and programmer-debugging e ort. 2 con iguration problem can occur or several reasons, all re)uiring someone to care ully e/amine the code while running the so tware under di erent con igurations to ind the bug9

93

Bour so tware may have a bug that appears under a broad class o con igurations. 2n e/ample is i your greeting card program wor"s ine with laser printers but not with in":et printers. Bour so tware may have a bug speci ic only to one particular con igurationit doesn't wor" on the ."ee,o1ee !odel (-G49 In"Het ,elu/e printer. The hardware device or its device drivers may have a bug that only your so tware reveals. !aybe your so tware is the only one that uses a uni)ue display card setting. %hen your so tware is run with a speci ic video card, the 7+ crashes. The hardware device or its device drivers may have a bug that can be seen with lots o other so twarealthough it may be particularly obvious with yours. 2n e/ample would be i a speci ic printer driver always de aulted to dra t mode and your photo printing so tware had to set it to high-)uality every time it printed.

In the irst two cases, it seems airly straight orward that your pro:ect team is responsible or i/ing the bug. It's your problem. Bou should i/ it. In the last two cases, things get blurry. &ay the bug is in a printer and that printer is the most popular in the world, with tens o millions in use. Bour so tware obviously needs to wor" with that printer. It may ta"e the printer vendor months to i/ the problem =i it does at all> so your team will need to ma"e changes to your so tware, even though the so tware is doing everything right, to wor" around the bug. In the end, it's your team's responsibility to address the problem, no matter where it lies. Bour customers don't care why or how the bug is happening, they :ust want the new so tware they purchased to wor" on their system's con iguration. &2 P<RP,E 2<GG 54% S&<4% C5R%S In 1997 !icroso t released its 2cti!ates (arney character and supporting +,--.! learning so tware or "ids. These animatronic dolls interacted with the so tware through a two-way radio in the doll and another radio connected to a 7+. The 7+'s radio connected to a seldom-used inter ace on most sound cards called an !I,I connector. This inter ace is used or music "eyboards and other musical instruments. !icroso t assumed the connector would be a good choice because most people don't own musical devices. It would li"ely not have anything plugged into it and would be available or use with the 2cti!ates radio. ,uring con iguration testing, a typical amount o bugs showed up. &ome were due to sound card problems, some were in the 2cti!ates so tware. There was one bug, however, that could never )uite be pinned down. It seemed that occasionally, randomly, the 7+ running the so tware would :ust loc" up and would re)uire rebooting. This problem occurred only with the most popular sound card on the mar"eto course. %ith :ust wee"s le t in the schedule, a concerted e ort was put together to resolve the problem. 2 ter a great deal o con iguration testing and debugging, the bug was isolated to the sound card's hardware. It seems that the !I,I connector always had this bug, but, being so seldom used, no one had ever seen it. The 2cti!ates so tware e/posed it or the irst time. There was a mad scramble, lots o denials and inger pointing, and lots o late nights. In the end, the sound card manu acturer conceded that there was a problem and promised to wor" around the bug in updated versions o its device driver. !icroso t included a i/ed driver on the 2cti!ates +,--.! and made changes to the so tware that attempted to ma"e the bug occur less re)uently. ,espite all those e orts, sound card compatibility problems were the top reason that people called in or assistance with the product.

94

SiBing <p the ;ob The :ob o con iguration testing can be a huge underta"ing. &uppose that you're testing a new so tware game that runs under !icroso t %indows. The game is very graphical, has lots o sound e ects, allows multiple players to compete against each other over the phone lines, and can print out game details or strategy planning. 2t the least, you'll need to consider con iguration testing with di erent graphics cards, sound cards, modems, and printers. The %indows 2dd *ew #ardware %izard =see ?igure A.4> allows you to select hardware in each o these categoriesand 4G others. 2igure H>0> The )i'rosoft #in"ows 5"" 4ew =ar"ware #iBar" "ialog bo( allows you to a"" new har"ware to your PC+s 'urrent 'onfiguration>

$nder each hardware category are the di erent manu acturers and models =see ?igure A.G>. 1eep in mind, these are only the models with support built into %indows. !any other models provide their own setup dis"s with their hardware. 2igure H>1> Ea'h type of har"ware has numerous manufa'turers an" mo"els>

95

I you decided to per orm a ull, comprehensive con iguration test, chec"ing every possible ma"e and model combination, you'd have a huge :ob ahead o you. &ay there are appro/imately ;;5 possible display cards, 41< sound cards, 1G<< modems, and 14<< printers. The number o test combinations is ;;5 / 41< / 1G<< / 14<<, or a total in the billionsway too many to considerN I you limited your testing to e/clude combinations, :ust testing each card individually at about ;< minutes per con iguration, you'd be at it or about a year. 1eep in mind that's :ust one pass through the con igurations. It's not uncommon with bug i/es to run two or three con iguration test passes be ore a product is released. The answer to this mess, as you've hope ully deduced, is e)uivalence partitioning. Bou need to igure out a way to reduce the huge set o possible con igurations to the ones that matter the most. Bou'll assume some ris" by not testing everything, but that's what so tware testing is all about. 5pproa'hing the Task The decision-ma"ing process that goes into deciding what devices to test with and how they should be tested is a airly straight orward e)uivalence partition pro:ect. %hat's important, and what ma"es the e ort a success or not, is the in ormation you use to ma"e the decisions. I you're not e/perienced with the hardware that your so tware runs on, you should learn as much as you can and bring in other e/perienced testers or programmers to help you. 2s" a lot o )uestions and ma"e sure you get your plan approved. The ollowing sections show the general process that you should use when planning your con iguration testing. %e'i"e the Types of =ar"ware 6ou+ll 4ee" ,oes your application printI I so, you'll need to test printers. I it has sound, you'll need to test sound cards. I it's a photo or graphics program, you'll li"ely need scanners and digital cameras. 0oo" closely at your so tware eature set to ma"e sure that you cover everything. 7ut your so tware dis" on a table and as" yoursel what hardware pieces you need to put together to ma"e it wor". &4,!4E RE*!STR5T!&4 2n e/ample o a eature that you can easily overloo" when selecting what hardware to test with is online registration. !any programs today allow users to register their so tware during installation via modem or broadband connections. $sers type in their name, address, and other personal data, clic" a button, and the modem dials out to a computer at the so tware company where it downloads the in ormation and completes the registration. The so tware may not do anything else with online communications. (ut, i it has online registration, you will need to consider modems and networ" communications as part o your con iguration testing.

%e'i"e #hat =ar"ware Bran"s. )o"els. an" %e3i'e %ri3ers 5re 53ailable I you're putting out a cutting-edge graphics program, you probably don't need to test that it prints well on a 19A7 blac"-and-white dot-matri/ printer. %or" with your sales and mar"eting people to create a list o hardware to test with. I they can't or won't help, chec" out the recent e)uipment reviews rom 7+ !agazine or !ac %orld to get an idea o what hardware is available and what is =and was> popular. (oth magazines, as well as others, have annual reviews o printers, sound cards, display adapters and other peripherals. ,o some research to see i some o the devices are clones o each other and there ore e)uivalent alling under the same e)uivalence partition. ?or e/ample, a printer manu acturer

96

may license their printer to another company that puts a di erent cover and label on it. ?rom your standpoint, it's the same printer. ,ecide what device drivers you're going to test with. Bour options are usually the drivers included with the operating system, the drivers included with the device, or the latest drivers available on the hardware or operating system company's website. $sually, all three are di erent. 2s" yoursel what customers have or what they can get. %e'i"e #hi'h =ar"ware 2eatures. )o"es. an" &ptions 5re Possible +olor printers can print in blac" and white or color, they can print in di erent )uality modes, and can have settings or printing photos or te/t. ,isplay cards, as shown in ?igure A.5, can have di erent color settings and screen resolutions. 2igure H>E> The "isplay properties of number of 'olors an" s'reen area are possible 'onfigurations for a "isplay 'ar">

8very device has options, and your so tware may not need to support all o them. 2 good e/ample o this is computer games. !any re)uire a minimum number o display colors and resolution. I the con iguration has less than that, they simply won't run. Pare %own the !"entifie" =ar"ware Configurations to a )anageable Set Fiven that you don't have the time or budget to test everything, you need to reduce the thousands o potential con igurations into the ones that matterthe ones you're going to test. .ne way to do this is to put all the con iguration in ormation into a spreadsheet with columns or the manu acturer, model, driver versions, and options. ?igure A.7 shows an e/ample o a table that identi ies various printer con igurations. Bou and your team can review the chart and decide which con iguration you want to test. 2igure H>9> &rganiBe your 'onfiguration information into a sprea"sheet>

97

*otice that ?igure A.7 also has columns or in ormation about the device's popularity, its type, and its age. %hen creating your e)uivalence partitions, you might decide that you want to test only the most popular printers or ones that are less than ive years old. %ith the type in ormationin this case, laser or in":etyou could decide to test 7G percent lasers and 4G percent in":ets. *.T8 $ltimately, the decision-ma"ing process that you use to e)uivalence partition the con igurations into smaller sets is up to you and your team. There is no right ormula. 8very so tware pro:ect is di erent and will have di erent selection criteria. Hust ma"e sure that everyone on the pro:ect team, especially your pro:ect manager, is aware o what con igurations are being tested =and not tested> and what variables went into selecting them.

!"entify 6our Software+s <niCue 2eatures That #ork with the =ar"ware Configurations The "ey word here is uni)ue. Bou don't want to, nor do you need to, completely test your so tware on each con iguration. Bou need to test only those eatures that are di erent rom each other =di erent e)uivalence partitions> that interact with the hardware. ?or e/ample, i you're testing a word processor such as %ord7ad =see ?igure A.A>, you don't need to test the ile save and load eature in each con iguration. ?ile saving and loading has nothing to do with printing. 2 good test would be to create a document that contains di erent =selected by e)uivalence partitioning, o course> onts, point sizes, colors, embedded pictures, and so on. Bou would then attempt to print this document on each chosen printer con iguration. 2igure H>H> 6ou 'an use a sample "o'ument ma"e up of "ifferent fonts an" styles to 'onfiguration test a printer>

98

&electing the uni)ue eatures to try isn't as easy as it sounds. Bou should irst ma"e a blac"-bo/ pass by loo"ing at your product and pulling out the obvious ones. Then tal" with others on your team, especially the programmers, to get a white-bo/ view. Bou may be surprised at what eatures are even slightly tied to the con iguration. %esign the Test Cases to Run on Ea'h Configuration Bou'll learn the details o writing test cases in +hapter 1A, 3%riting and Trac"ing Test +ases,3 but, or now, consider that you'll need to write down the steps re)uired to test each con iguration. This can be as simple as 1> &elect and set up the ne/t test con iguration rom the list. 7> D> 0> 1> E> &tart the so tware. 0oad in the ile )o%fi-te't.do). +on irm that the ile is displayed correctly. 7rint the document. +on irm that there are no error messages and that the printed document matches the standard. 0og any discrepancies as a bug.

9>

In reality, the steps would be much more involved, including more detail and speci ics on e/actly what to do and what to loo" or. The goal is to create steps that anyone can run. Bou'll learn more about writing test cases in +hapter 1A.

99

E(e'ute the Tests on Ea'h Configuration Bou need to run the test cases and care ully log and report your results =see +hapter 19, 3-eporting %hat Bou ?ind3> to your team and to the hardware manu acturers i necessary. 2s described earlier in this chapter, it's o ten di icult and time-consuming to identi y the speci ic source o con iguration problems. Bou'll need to wor" closely with the programmers and whitebo/ testers to isolate the cause and decide i the bugs you ind are due to your so tware or to the hardware. I the bug is speci ic to the hardware, consult the manu acturer's website or in ormation on reporting problems to them. (e sure to identi y yoursel as a so tware tester and what company you wor" or. !any companies have separate sta set up to assist so tware companies writing so tware to wor" with their hardware. They may as" you to send copies o your test so tware, your test cases, and supporting details to help them isolate the problem. Rerun the Tests <ntil the Results Satisfy 6our Team It's not uncommon or con iguration testing to run the entire course o a pro:ect. Initially a ew con igurations might be tried, then a ull test pass, then smaller and smaller sets to con irm bug i/es. 8ventually you will get to a point where there are no "nown bugs or to where the bugs that still e/ist are in uncommon or unli"ely test con igurations. 2t that point, you can call your con iguration testing complete. &btaining the =ar"ware .ne thing that hasn't been mentioned so ar is where you obtain all this hardware. 8ven i you ta"e great pains, and ris", to e)uivalence partition your con igurations to the barest minimum, you still could re)uire dozens o di erent hardware setups. It would be an e/pensive proposition to go out and buy everything at retail, especially i you will use the hardware only once or the one test pass. #ere are a ew ideas or overcoming this problem9 (uy only the con igurations that you can or will use most o ten. 2 great plan is or every tester on the team to have di erent hardware. This may drive your purchasing department and the group that maintains your company's 7+s crazy =they li"e everyone to have e/actly the same con iguration> but it's a very e icient means o always having di erent con igurations available to test on. 8ven i your test team is very small, three or our people having :ust a ew con igurations would be a great help. +ontact the hardware manu acturers and as" i they will lend or even give you the hardware. I you e/plain that you're testing new so tware and you want to assure that it wor"s on their hardware, many will do this or you. They have an interest in the outcome, too, so tell them that you'll urnish them with the results o the tests and, i you can, a copy o the inished so tware. It's good to build up these relationships, especially i you ind a bug and need a contact person at the hardware company to report it to. &end a memo or email to everyone in your company as"ing what hardware they have in their o ice or even at homeand i they would allow you to run a ew tests on it. To per orm the con iguration testing, you may need to drive around town, but it's a whole lot cheaper than attempting to buy all the hardware. C&42!*<R5T!&4 TEST!4* :CRS The !icroso t 2cti!ates product line o animatronic dolls not only inter aced with a 7+, but also a 6+-. +oded commands, invisible to a viewer, were mi/ed in with the

100

video on the tape. 2 special bo/ connected to the 6+- decoded the commands and sent them by radio to the doll. The test team obviously needed to per orm con iguration testing on 6+-s. They had many 7+ con igurations but no 6+-s. They ound two ways to get the :ob done9 o o They as"ed about ;<< employees to bring in their 6+-s or a day o testing. The program manager awarded gi t certi icates as a means o persuading people to bring them in. They paid the manager o a local electronics superstore to stay at the store a ter hours =actually, all night> while they pulled each 6+- o the shel , connected their e)uipment, and ran the tests. They dusted and cleaned the 6+-s and bought the manager dinner to show their than"s.

%hen it was all over, they had tested about 1G< uni)ue 6+-s, which they determined was a very good e)uivalence partition o the 6+-s in people's homes. I you have the budget, wor" with your pro:ect manager to contract out your test wor" to a pro essional con iguration and compatibility test lab. These companies do nothing but con iguration testing and have every piece o 7+ hardware "nown to man. ."ay, maybe not that much, but they do have a lot. These labs can help you, based on their e/perience, select the correct hardware to test. Then, they will allow you to come in and use their e)uipment, or they will provide a complete turn-"ey service. Bou provide the so tware, the step-by-step test process, and the e/pected results. They'll ta"e it rom there, running the tests and reporting what passed and what ailed. . course this can be costly, but much less so than buying the hardware yoursel or worse, not testing and having customers ind the problems. !"entifying =ar"ware Stan"ar"s I you're interested in per orming a little static blac"-bo/ analysisthat is, reviewing the speci ications that the hardware companies use to create their productsyou can loo" in a couple o places. 1nowing some details o the hardware speci ications can help you ma"e more in ormed e)uivalence partition decisions. ?or 2pple hardware, visit the 2pple #ardware website at developer.apple.comShardware. There you'll ind in ormation and lin"s about developing and testing hardware and device drivers or 2pple computers. 2nother 2pple lin", developer.apple.comStesting, points you to speci ic testing in ormation, including lin"s to test labs that per orm con iguration testing. ?or 7+s, the best lin" is www.microso t.comSwhdcSsystemSplat orm. This site provides technical implementation guidelines, tips, and tools or developers and testers developing hardware or use with %indows. !icroso t also publishes a set o standards or so tware and hardware to receive the %indows logo. That in ormation is at msdn.microso t.comScerti ication and www.microso t.comSwhdcSwh)l. Configuration Testing &ther =ar"ware &o, what i you're not testing so tware that runs on a 7+ or a !acI %as this chapter a waste o your timeI *o wayN 8verything you learned can be applied to testing generic or proprietary systems, too. It doesn't matter what the hardware is or what it's connected toE i it can have variations such as memory size, +7$ speed, etc, or, i it connects to another piece o hardware, so tware con iguration issues need to be tested.

101

I you're testing so tware or an industrial controller, a networ", medical devices, or a phone system, as" yoursel the same )uestions that you would i you were testing so tware or a des"top computer9 %hat e/ternal hardware will operate with this so twareI %hat models and versions o that hardware are availableI %hat eatures or options does that hardware supportI

+reate e)uivalence partitions o the hardware based on input rom the people who wor" with the e)uipment, your pro:ect manager, or your sales people. ,evelop test cases, collect the selected hardware, and run the tests. +on iguration testing ollows the same testing techni)ues that you've already learned. Summary This chapter got you thin"ing about how to approach con iguration testing. It's a :ob that new so tware testers are re)uently assigned because it is easily de ined, is a good introduction to basic organization s"ills and e)uivalence partitioning, is a tas" that will get you wor"ing with lots o other pro:ect team members, and is one or which your manager can readily veri y the results. The downside is that it can become overwhelming. I you're assigned to per orm con iguration testing or your pro:ect, ta"e a deep breath, reread this chapter, care ully plan your wor", and ta"e your time. %hen you're done, your boss will have another :ob or you9 compatibility testing, the sub:ect o the ne/t chapter. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. %hat's the di erence between a component and a peripheralI 4. #ow can you tell i a bug you ind is a general problem or a speci ic con iguration problemI ;. #ow could you guarantee that your so tware would never have a con iguration problemI

4. &ome companies purchase generic hardware and put their names on it, selling it as
their own. Bou'll o ten see this on lower-priced peripherals sold in computer superstores. The same 3cloned3 peripheral might be sold under di erent names in di erent stores. True or ?alse9 .nly one version o a cloned sound card needs to be considered when selecting the con igurations to test. G. In addition to age and popularity, what other criteria might you use to e)uivalence partition hardware or con iguration testingI 5. Is it acceptable to release a so tware product that has con iguration bugsI

Chapter /> Compatibility Testing I* T#I& +#27T8-

102

+ompatibility Testing .verview 7lat orm and 2pplication 6ersions &tandards and Fuidelines ,ata &haring +ompatibility

In +hapter A, 3+on iguration Testing,3 you learned about hardware con iguration testing and how to assure that so tware wor"s properly with the hardware it was designed to run on and connect with. This chapter deals with a similar area o interaction testingchec"ing that your so tware operates correctly with other so tware. Testing whether one program plays well with others has become increasingly important as consumers demand the ability to share data among programs o di erent types and rom di erent vendors and ta"e advantage o the ability to run multiple programs at once. It used to be that a program could be developed as a standalone application. It would be run in a "nown, understood, benign environment, isolated rom anything that could corrupt it. Today, that program li"ely needs to import and e/port data to other programs, run with di erent operating systems and %eb browsers, and interoperate with other so tware being run simultaneously on the same hardware. The :ob o so tware compatibility testing is to ma"e sure that this interaction wor"s as users would e/pect. The highlights o this chapter include %hat it means or so tware to be compatible #ow standards de ine compatibility %hat plat orms are and what they mean or compatibility %hy being able to trans er data among so tware applications is the "ey to compatibility

Compatibility Testing &3er3iew &o tware compatibility testing means chec"ing that your so tware interacts with and shares in ormation correctly with other so tware. This interaction could occur between two programs simultaneously running on the same computer or even on di erent computers connected through the Internet thousands o miles apart. The interaction could also be as simple as saving data to a loppy dis" and hand-carrying it to another computer across the room. 8/amples o compatible so tware are +utting te/t rom a web page and pasting it into a document opened in your word processor &aving accounting data rom one spreadsheet program and then loading it into a completely di erent spreadsheet program #aving photograph touchup so tware wor" correctly on di erent versions o the same operating system #aving your word processor load in the names and addresses rom your contact management program and print out personalized invitations and envelopes $pgrading to a new database program and having all your e/isting databases load in and wor" :ust as they did with the old program

103

%hat compatibility means or your so tware depends on what your team decides to speci y and what levels o compatibility are re)uired by the system that your so tware will run on. &o tware or a standalone medical device that runs its own operating system, stores its data on its own memory cartridges, and doesn't connect to any other device would have no compatibility considerations. #owever, the i th version o a word processor =see ?igure 9.1> that reads and writes di erent iles rom other word processors, allows multiuser editing over the Internet, and supports inclusion o embedded pictures and spreadsheets rom various applications has a multitude o compatibility considerations. 2igure />1> Compatibility a'ross "ifferent software appli'ations 'an Cui'kly be'ome 3ery 'ompli'ate">

I you're assigned the tas" o per orming so tware compatibility testing on a new piece o so tware, you'll need to get the answers to a ew )uestions9 %hat other plat orms =operating system, web browser, or other operating environment> and other application so tware is your so tware designed to be compatible withI I the so tware you're testing is a plat orm, what applications are designed to run under itI %hat compatibility standards or guidelines should be ollowed that de ine how your so tware should interact with other so twareI %hat types o data will your so tware use to interact and share in ormation with other plat orms and so twareI

Faining the answers to these )uestions is basic static testingboth blac"-bo/ and white-bo/. It involves thoroughly analyzing the speci ication or the product and any supporting speci ications. It could also entail discussions with the programmers and possibly close review o the code to assure that all lin"s to and rom your so tware are identi ied. The rest o this chapter discusses these )uestions in more detail. Platform an" 5ppli'ation :ersions &electing the target plat orms or the compatible applications is really a program management or a mar"eting tas". &omeone who's very amiliar with the customer base will decide whether your so tware is to be designed or a speci ic operating system, web browser, or some other plat orm. They'll also identi y the version or versions that the so tware needs to be compatible with. ?or e/ample, you've probably seen notices such as these on so tware pac"ages or startup screens9 %or"s best with 2.0 9.<

104

-e)uires %indows O7 or greater ?or use with 0inu/ 4.5.1< This in ormation is part o the speci ication and tells the development and test teams what they're aiming or. 8ach plat orm has its own development criteria and it's important, rom a pro:ect management standpoint, to ma"e this plat orm list as small as possible but still ill the customer's needs. Ba'kwar" an" 2orwar" Compatibility Two terms you'll hear regarding compatibility testing are bac"ward compatible and orward compatible. I something is bac"ward compatible, it will wor" with previous versions o the so tware. I something is orward compatible, it will wor" with uture versions o the so tware. The simplest demonstration o bac"ward and orward compatibility is with a .t1t or te/t ile. 2s shown in ?igure 9.4, a te/t ile created using *otepad 9A running under %indows 9A is bac"ward compatible all the way bac" to !&-,.& 1.<. It's also orward compatible to %indows O7 service pac" 4 and li"ely will be beyond that. 2igure />7> Ba'kwar" an" forwar" 'ompatibility "efine what 3ersions will work with your software or "ata files>

*.T8 It's not a re)uirement that all so tware or iles be bac"ward or orward compatible. That's a product eature decision your so tware designers need to ma"e. Bou should, though, provide input on how much testing will be re)uired to chec" orward and bac"ward compatibility or the so tware.

105

The !mpa't of Testing )ultiple :ersions Testing that multiple versions o plat orms and so tware applications wor" properly with each other can be a huge tas". +onsider the situation o having to compatibility test a new version o a popular operating system. The programmers have made numerous bug i/es and per ormance improvements and have added many new eatures to the code. There could be tens or hundreds o thousands o e/isting programs or the current versions o the .&. The pro:ect's goal is to be 1<< percent compatible with them. &ee ?igure 9.;. 2igure />D> !f you 'ompatibility test a new platform. you must 'he'k that e(isting software appli'ations work 'orre'tly with it>

This is a big :ob, but it's :ust another e/ample o how e)uivalence partitioning can be applied to reduce the amount o wor". *.T8 To begin the tas" o compatibility testing, you need to e)uivalence partition all the possible so tware combinations into the smallest, e ective set that veri ies that your so tware interacts properly with other so tware. In short, you can't test all the thousands o so tware programs on your operating system, so you need to decide which ones are the most important to test. The "ey word is important. The criteria that might go into deciding what programs to choose could be 7opularity. $se sales data to select the top 1<< or 1,<<< most popular programs. 2ge. Bou might want to select programs and versions that are less than three years old. Type. (rea" the so tware world into types such as painting, writing, accounting, databases, communications, and so on. &elect so tware rom each category or testing. !anu acturer. 2nother criteria would be to pic" so tware based on the company that created it.

Hust as in hardware con iguration testing, there is no right 3te/tboo"3 answer. Bou and your team will need to decide what matters most and then use that criteria to create e)uivalence partitions o the so tware you need to test with. The previous e/ample dealt with compatibility testing a new operating system plat orm. The same issues apply to testing a new application =see ?igure 9.4>. Bou need to decide what plat orm versions you should test your so tware on and what other so tware applications you should test your so tware with.

106

2igure />0> Compatibility testing a new appli'ation may reCuire you to test it on multiple platforms an" with multiple appli'ations>

Stan"ar"s an" *ui"elines &o ar in this chapter you've learned about selecting the so tware that you'll compatibility test with your program. *ow, it's time to loo" at how you'll approach the actual testing. Bour irst stop should be researching the e/isting standards and guidelines that might apply to your so tware or the plat orm. There are really two levels o these re)uirements9 high-level and low-level. It may be a misnomer to re er to them as high and low, but in a sense, that's really what they are. #ighlevel standards are the ones that guide your product's general operation, its loo" and eel, its supported eatures, and so on. 0ow-level standards are the nitty-gritty details, such as the ile ormats and the networ" communications protocols. (oth are important and both need to be tested to assure compatibility. =igh-,e3el Stan"ar"s an" *ui"elines %ill your so tware run under %indows, !ac, or 0inu/ operating systemsI Is it a web applicationI I so, what browsers will it run onI 8ach o these is considered a plat orm and most have their own set o standards and guidelines that must be ollowed i an application is to claim that it's compatible with the plat orm. 2n e/ample o this is the +erti ied or !icroso t %indows logo =see ?igure 9.G>. To be awarded this logo, your so tware must undergo and pass compatibility testing by an independent testing laboratory. The goal is to assure that the so tware runs stably and reliably on the operating system. 2igure />1> The Certifie" for )i'rosoft #in"ows logo signifies that the software meets all the 'riteria "efine" by the gui"elines>

107

2 ew e/amples o the logo re)uirements are that the so tware &upports mice with more than three buttons &upports installation on dis" drives other than +9 and ,9 &upports ilenames longer than the ,.& A.; ormat ,oesn't read, write, or otherwise use the old system iles wi%.i%i, '"'teA.i%i, $utoe1e).#$t, or )o%fi-.'"'

These may sound li"e simple, matter-o - act re)uirements, but they're only our items out o a 1<<J page document. It's a great deal o wor" to assure that your so tware complies with all the logo re)uirements, but it ma"es or much more compatible so tware. *.T8 The details o the %indows logo can be obtained at msdn.microso t.comScerti ication. ,etails or using the 2pple !ac logo are at developer.apple.comStesting.

,ow-,e3el Stan"ar"s an" *ui"elines 0ow-level standards are, in a sense, more important than the high-level standards. Bou could create a program that would run on %indows that didn't have the loo" and eel o other %indows so tware. It wouldn't be granted the +erti ied or !icroso t %indows logo. $sers might not be thrilled with the di erences rom other applications, but they could use the product. I , however, your so tware is a graphics program that saves its iles to dis" as .2i)t iles =a standard !acintosh ile ormat or graphics> but the program doesn't ollow the standard or .2i)t iles, your users won't be able to view the iles in any other program. Bour so tware wouldn't be compatible with the standard and would li"ely be a short-lived product. &imilarly, communications protocols, programming language synta/, and any means that programs use to share in ormation must adhere to published standards and guidelines. These low-level standards are o ten ta"en or granted, but rom a tester's perspective must be tested. Bou should treat low-level compatibility standards as an e/tension o the so tware's speci ication. I the so tware spec states, 3The so tware will save and load its graphics iles as .#A2, .I2-, and .-if ormats,3 you need to ind the standards or these ormats and design tests to con irm that the so tware does indeed adhere to them. *.T8 ,on't necessarily trust your team's interpretation o the standards or guidelines. 0oo" them up yoursel and develop your tests directly rom the source. -emember the di erence between precision and accuracyI Bou don't want your product's compatibility code to be per ectly precise, but totally inaccurate. %ata Sharing Compatibility The sharing o data among applications is what really gives so tware its power. 2 well-written program that supports and adheres to published standards and allows users to easily trans er data to and rom other so tware is a great compatible product. The most amiliar means o trans erring data rom one program to another is saving and loading dis" iles. 2s discussed in the previous section, adhering to the low-level standards or the dis" and ile ormats is what ma"es this sharing possible. .ther means, though, are sometimes ta"en or granted but still need to be tested or compatibility. #ere are a ew e/amples9 ?ile save and ile load are the data-sharing methods that everyone is aware o . Bou save your data to a loppy dis" =or some other means o magnetic or optical storage>

108

and then hand carry it over to another computer running di erent so tware and load it in. The data ormat o the iles needs to meet standards or it to be compatible on both computers. ?ile e/port and ile import are the means that many programs use to be compatible with older versions o themselves and with other programs. ?igure 9.5 shows the !icroso t %ord ?ile .pen dialog bo/ and some o the 4; di erent ile ormats that can be imported into the word processor. 2igure />E> )i'rosoft #or" 'an import 7D "ifferent file formats>

To test the ile import eature, you would need to create test documents in each compatible ile ormatprobably using the original so tware that wrote that ormat. Those documents would need to have e)uivalence partitioned samples o the possible te/t and ormatting to chec" that the importing code properly converts it to the new ormat. +ut, copy, and paste are the most amiliar methods or sharing data among programs without trans erring the data to a dis". In this case, the trans er happens in memory through an intermediate program called the +lipboard. ?igure 9.7 shows how this trans er occurs. 2igure />9> The System Clipboar" is a temporary hol"ing pla'e for "ifferent types of "ata that+s being 'opie" from one appli'ation to another>

109

The +lipboard is designed to hold several di erent data types. +ommon ones in %indows are te/t, pictures, and sounds. These data types can also be di erent ormats or e/ample, the te/t can be plain old te/t, #T!0, or rich te/t. 7ictures can be bitmaps, meta iles, or .tifs. %henever a user per orms a cut or copy, the data that's chosen is placed in the +lipboard. %hen he does a paste, it's copied rom the +lipboard to the destination so tware. &ome applications may only accept certain data types or ormats being pasted into them or e/ample, a painting program may accept pictures, but not te/t. I you're compatibility testing a program, you need to ma"e sure that its data can be properly copied in and out o the +lipboard to other programs. This eature is so transparent and so re)uently used, people orget that there's a lot o code behind ma"ing sure that it wor"s and is compatible across lots o di erent so tware. ,,8 =pronounced ,-,-8>, +.! = or +omponent .b:ect !odel>, and .08 =pronounced ohlay> are the methods in %indows o trans erring data between two applications. ,,8 stands or ,ynamic ,ata 8/change and .08 stands or .b:ect 0in"ing and 8mbedding. .ther plat orms support similar methods. There's no need to get into the gory details o these technologies in this boo", but the primary di erence between these two methods and the +lipboard is that with ,,8 and .08 data can low rom one application to the other in real time. +utting and copying is a manual operation. %ith ,,8 and .08, the trans er can happen automatically. 2n e/ample o how these might be used could be a written report done in a word processor that has a pie-chart created by a spreadsheet program. I the report's author copied and pasted the chart into the report, it would be a snapshot in time o the data. I , however, the author lin"ed the pie chart into the report as an ob:ect, when the underlying numbers or the chart change, the new graphics will automatically appear in the report. This is all pretty ancy, yes, but it's also a testing challenge to ma"e sure that all the ob:ect lin"ing, embedding, and data e/changing happens correctly. Summary This chapter introduced you to the basics o compatibility testing. In reality, an entire boo" could be written on the sub:ect, and a single chapter doesn't do the topic :ustice. 8very plat orm and every application is uni)ue, and the compatibility issues on one system can be totally di erent than on another. 2s a new so tware tester, you may be assigned a tas" o compatibility testing your so tware. That may seem strange, given that it's potentially such a large and comple/ tas", but you'll li"ely be assigned :ust a piece o the entire :ob. I your pro:ect is a new operating system, you

110

may be as"ed to compatibility test :ust word processors or graphics programs. I your pro:ect is an applications program, you may be as"ed to compatibility test it on several di erent plat orms. 8ach is a manageable tas" that you can easily handle i you approach your testing with these three things in mind9 8)uivalence partition all the possible choices o compatible so tware into a manageable set. . course, your pro:ect manager should agree with your list and understand the ris" involved in not testing everything. -esearch the high-level and low-level standards and guidelines that might apply to your so tware. $se these as e/tensions o your product's speci ication. Test the di erent ways that data can low between the so tware programs you're testing. This data e/change is what ma"es one program compatible with another.

AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N

1. True or ?alse9 2ll so tware must undergo some level o compatibility testing. 2. True or ?alse9 +ompatibility is a product eature and can have di erent levels o
compliance. ;. I you're assigned to test compatibility o your product's data ile ormats, how would you approach the tas"I 4. #ow can you test orward compatibilityI

Chapter 18> 2oreign-,anguage Testing I* T#I& +#27T8 !a"ing the %ords and 7ictures !a"e &ense Translation Issues 0ocalization Issues +on iguration and +ompatibility Issues #ow !uch &hould Bou TestI

&i eres luente en mXs de un idioma y competente probando programas de computadora, usted tiene una habilidad muy deseada en el mercado. %enn &ie eine zuverlYZig &o tware 7r[ erin sind, und lieZend eine remd sprache, ausser 8nglish, sprechen "\nnen, dann "\nnen &ie gut verdienen.

111

Translated roughly rom &panish and Ferman, the preceding two sentences read9 I you are a competent so tware tester and are luent in a language other than 8nglish, you have a very mar"etable s"ill set. !ost so tware today is released to the entire world, not :ust to a certain country or in a speci ic language. !icroso t shipped %indows O7 with support or 1<5 di erent languages and dialects, rom 2 ri"aans to #ungarian to Tulu. !ost other so tware companies do the same, realizing that the $.&. 8nglish mar"et is less than hal o their potential customers. It ma"es business sense to design and test your so tware or worldwide distribution. This chapter covers what's involved in testing so tware written or other countries and languages. It might seem li"e a straight orward process, but it's not, and you'll learn why. #ighlights o this chapter include %hy :ust translating is not enough #ow words and te/t are a ected %hy ootballs and telephones are important The con iguration and compatibility issues #ow large o a :ob testing another language is

)aking the #or"s an" Pi'tures )ake Sense #ave you ever read a user's manual or an appliance or a toy that was poorly converted word or word rom another languageI 37ut in a bolt number ive past through green bar and tighten no loose to nut.3 Fot itI That's a poor translation, and it's what so tware can loo" li"e to a non-8nglish spea"er i little e ort is put into building the so tware or oreign languages. It's easy to individually convert all the words, but to ma"e the overall instructions meaning ul and use ul re)uires much more wor" and attention. Food translators can do that. I they're luent in both languages, they can ma"e the oreign te/t read as well as the original. $n ortunately, what you'll ind in the so tware industry is that even a good translation isn't su icient. Ta"e &panish, or e/ample. It should be a simple matter to convert 8nglish te/t to &panish, rightI %ell, which &panish are you re erring toI &panish rom &painI %hat about &panish rom +osta -ica, 7eru, or the ,ominican -epublicI They're all &panish, but they're di erent enough that so tware written or one might not be received well by the others. 8ven 8nglish has this problem. There's not :ust 2merican 8nglish, there's also +anadian, 2ustralian, and (ritish 8nglish. It would probably seem strange to you to see the words colour, neighbour, and rumour in your word processor. %hat needs to be accounted or, besides the language, is the region or localethe user's country or geographic area. The process o adapting so tware to a speci ic locale, ta"ing into account its language, dialect, local conventions, and culture, is called localization or sometimes internationalization. Testing the so tware is called localization testing. Translation !ssues 2lthough translation is :ust a part o the overall localization e ort, it's an important one rom a test standpoint. The most obvious problem is how to test something that's in another language. %ell, you or someone on your test team will need to be at least semi- luent in the language you're testing, being able to navigate the so tware, read any te/t it displays, and type the necessary commands to run your tests. It might be time to sign up or the community college course in &lovenian you always wanted to ta"e. *.T8

112

It's important that you or someone on your test team be at least a little amiliar with the language you're testing. . course, i you're shipping your program in ;4 di erent languages, they may be di icult. The solution is to contract out this wor" to a localization testing company. *umerous such companies worldwide can per orm testing in nearly any language. ?or more in ormation, search the Internet or 3localization testing.3 It's not a re)uirement that everyone on the test team spea" the language that the so tware is being localized intoE you probably need :ust one person. !any things can be chec"ed without "nowing what the words say. It would be help ul, sure, to "now a bit o the language, but you'll see that you might be able to do a air amount o the testing without being completely luent. Te(t E(pansion The most straight orward e/ample o a translation problem that can occur is due to something called te/t e/pansion. 2lthough 8nglish may appear at times to be wordy, it turns out that when 8nglish is translated into other languages, o ten more characters are necessary to say the same thing. ?igure 1<.1 shows how the size o a button needs to e/pand to hold the translated te/t o two common computer words. 2 good rule o thumb is to e/pect up to 1<< percent increase in size or individual wordson a button, or e/ample. 8/pect a G< percent increase in size or sentences and short paragraphstypical phrases you would see in dialog bo/es and error messages. 2igure 18>1> #hen translate" into other languages. the wor"s )inimiBe an" )a(imiBe 'an 3ary greatly in siBe often for'ing the <! to be re"esigne" to a''ommo"ate them>

(ecause o this e/pansion, you need to care ully test areas o the so tware that could be a ected by longer te/t. 0oo" or te/t that doesn't wrap correctly, is truncated, or is hyphenated incorrectly. This could occur anywhereonscreen, in windows, bo/es, buttons, and so on. 2lso loo" or cases where the te/t had enough room to e/pand, but did so by pushing something else out o the way. 2nother possibility is that this longer te/t can cause a ma:or program ailure or even a system crash. 2 programmer could have allocated enough internal memory or the 8nglish te/t messages, but not enough or the translated strings. The 8nglish version o the so tware will wor" ine but the Ferman version will crash when the message is displayed. 2 white-bo/ tester could catch this problem without "nowing a single word o the language. 5SC!!. %BCS. an" <ni'o"e +hapter G, 3Testing the &o tware with (linders .n,3 brie ly discussed the 2&+II character set. 2&+II can represent only 4G5 di erent charactersnot nearly enough to represent all the possible characters in all languages. %hen so tware started being developed or di erent languages,

113

solutions needed to be ound to overcome this limitation. 2n approach common in the days o !&-,.&, but still in use today, is to use a techni)ue called code pages. 8ssentially, a code page is a replacement 2&+II table, with a di erent code page or each language. I your so tware runs in Muebec on a ?rench 7+, it could load and use a code page that supports ?rench characters. -ussian uses a di erent code page or its +yrillic characters, and so on. This solution is ine, although a bit clun"y, or languages with less than 4G5 characters, but Hapanese, +hinese, and other languages with thousands o symbols cause problems. 2 system called ,(+& = or ,ouble-(yte +haracter &et> is used by some so tware to provide more than 4G5 characters. $sing 4 bytes instead o 1 byte allows or up to 5G,G;5 di erent characters. +ode pages and ,(+& are su icient in many situations but su er rom a ew problems. !ost important is the issue o compatibility. I a #ebrew document is loaded onto a Ferman computer running a (ritish word processor, the result can be gibberish. %ithout the proper code pages or the proper conversion rom one to the other, the characters can't be interpreted correctly, or even at all. The solution to this mess is the $nicode standard.

$nicode provides a uni)ue number or every character, no matter what the plat orm, no matter what the program, no matter what the language. 3%hat is $nicodeI3 rom the $nicode +onsortium website, www.unicode.org (ecause $nicode is a worldwide standard supported by the ma:or so tware companies, hardware manu acturers, and other standards groups, it's becoming more commonplace. !ost ma:or so tware applications support it. ?igure 1<.4 shows many o the di erent characters supported. I it's at all possible that your so tware will ever be localized, you and the programmers on your pro:ect should cut your ties to 3ol' 2&+II3 and switch to $nicode to save yoursel time, aggravation, and bugs. 2igure 18>7> This )i'rosoft #or" "ialog shows support for the <ni'o"e stan"ar">

=ot -eys an" Short'uts In 8nglish, it's &earch. In ?rench, it's -Cchercher. I the hot"ey or selecting &earch in the 8nglish version o your so tware is 2ltJ&, that will need to change in the ?rench version.

114

In localized versions o your so tware, you'll need to test that all the hot"eys and shortcuts wor" properly and aren't too di icult to use or e/ample, re)uiring a third "eypress. 2nd, don't orget to chec" that the 8nglish hot"eys and shortcuts are disabled. E(ten"e" Chara'ters 2 common problem with localized so tware, and even non-localized so tware, is in its handling o e/tended characters. -e erring bac" to that ancient 2&+II table, e/tended characters are the ones that all outside the normal 8nglish alphabet o 2T and az. 8/amples o these would be the accented characters such as the C in HosC or the ] in 8l *i]o. They also include the many symbol characters such as that aren't on your typical "eyboard. I your so tware is properly written to use $nicode or even i it correctly manages code pages or ,(+&, this shouldn't be an issue, but a tester should never assume anything, so it's worthwhile to chec". The way to test this is to loo" or all the places that your so tware can accept character input or send output. In each place, try to use e/tended characters to see i they wor" :ust as regular characters would. ,ialog bo/es, logins, and any te/t ields are air game. +an you send and receive e/tended characters through a modemI +an you name your iles with them or even have the characters in the ilesI %ill they print out properlyI %hat happens i you cut, copy, and paste them between your program and another oneI TI7 The simplest way to ensure that you test or proper handling o e/tended characters is to add them to your e)uivalence partition o the standard characters that you test. 2long with those bug-prone characters sitting on the 2&+II table boundaries, throw in an ^, an _ and a Z.

Computations on Chara'ters -elated to e/tended characters are problems with how they're interpreted by so tware that per orms calculations on them. Two e/amples o this are word sorting and upper- and lowercase conversion. ,oes your so tware sort or alphabetize word listsI !aybe in a list bo/ o selectable items such as ilenames or website addressesI I so, how would you sort the ollowing wordsI 1opi`ren -eiste armlich -eis"orn -eiZaus reiten reiZen 2rg rCsumC "opie`n -eisschnaps resume

I you're testing so tware to be sold to one o the many 2sian cultures, are you aware that the sort order is based on the order o the brush stro"es used to paint the characterI The preceding list would li"ely have a completely di erent sort order i written in !andarin +hinese. ?ind out

115

what the sorting rules are or the language you're testing and develop tests to speci ically chec" that the proper sort order occurs. The other area where calculation on e/tended characters brea"s down is with upper- and lowercase conversion. It's a problem because the 3tric"3 solution that many programmers learn in school is to simply add or subtract ;4 to the 2&+II value o the letter to convert it between cases. 2dd ;4 to the 2&+II value o 2 and you get the 2&+II value o a. $n ortunately, that doesn't wor" or e/tended characters. I you tried this techni)ue using the 2pple !ac e/tended character set, you'd convert b =2&+II 1;4> to c =2&+II 154> instead o ] =2&+II 1G<>not e/actly what you'd e/pect. &orting and alphabetizing are :ust two e/amples. +are ully loo" at your so tware to determine i there are other situations where calculations are per ormed on letters or words. &pellchec"ing perhapsI Rea"ing ,eft to Right an" Right to ,eft 2 huge issue or translation is that some languages, such as #ebrew and 2rabic, read rom right to le t, not le t to right. Imagine lipping your entire user inter ace into a mirror image o itsel . Than" ully, most ma:or operating systems provide built-in support or handling these languages. %ithout this, it would be a nearly impossible tas". 8ven so, it's still not a simple matter o translating the te/t. It re)uires a great deal o programming to ma"e use o the .&'s eatures to do the :ob. ?rom a testing standpoint, it's probably sa e to consider it a completely new product, not :ust a localization. Te(t in *raphi's 2nother translation problem occurs when te/t is used in graphics. &ee ?igure 1<.; or several e/amples. 2igure 18>D> #or" 7888 has e(amples of te(t in bitmaps that woul" be "iffi'ult to translate>

The icons in ?igure 1<.; are the standard ones or selecting (old, Italic, $nderline, and ?ont +olor. &ince they use the 8nglish letters (, I, $, and 2, they'll mean nothing to someone rom Hapan who doesn't read 8nglish. They might pic" up on the meaning based on their loo"the ( is a bit dar", the I is leaning, and the $ has a line under itbut so tware isn't supposed to be a puzzle. The impact o this is that when the so tware is localized, each icon will have to be changed to re lect the new languages. I there were many o these icons, it could get prohibitively e/pensive to localize the program. 0oo" or te/t-in-graphic bugs early in the development cycle so they don't ma"e it through to the end. -eep the Te(t out of the Co"e The inal translation problem to cover is a white-bo/ testing issue"eep the te/t out o the code. %hat this means is that all te/t strings, error messages, and really anything that could possibly be translated should be stored in a separate ile independent o the source code. Bou should never see a line o code such as9 &ri%t <Eello .orld<

116

!ost localizers are not programmers, nor do they need to be. It's ris"y and ine icient to have them modi ying the source code to translate it rom one language to another. %hat they should modi y is a simple te/t ile, called a resource ile, that contains all the messages the so tware can display. %hen the so tware runs, it re erences the messages by loo"ing them up, not "nowing or caring what they say. I the message is in 8nglish or ,utch, it gets displayed :ust the same. That said, it's important or white-bo/ testers to search the code to ma"e sure there are no embedded strings that weren't placed in the e/ternal ile. It would be pretty embarrassing to have an important error message in a &panish program appear in 8nglish. 2nother variation o this problem is when the code dynamically generates a te/t message. ?or e/ample, it might piece together snippets o te/t to create a larger message. The code could ta"e three strings9 1. 3Bou pressed the3 4. a variable string containing the name o the "ey :ust pressed ;. 3"ey :ust in timeN3 and put them together to create a message. I the variable string had the value 3stop nuclear reaction,3 the total message would read9 Bou pressed the stop nuclear reaction "ey :ust in timeN The problem is that the word order is not the same in all languages. 2lthough it pieces together nicely in 8nglish, with each phrase translated separately, it could be gibberish when stuc" together in !andarin +hinese or even Ferman. ,on't let strings crop into the code and don't let them be built up into larger strings by the code. ,o'aliBation !ssues 2s mentioned previously, translation issues are only hal the problem. Te/t can easily be translated and allowances made or di erent characters and lengths o strings. The di iculty occurs in changing the so tware so that it's appropriate or the oreign mar"et. -8!I*,8-emember those terms rom +hapter ;, 3The -ealitites o &o tware Testing39 precision, accuracy, and reliability and )ualityI %ell translated and tested so tware is precise and reliable, but, i the programmers don't consider localization issues, it's probably not accurate or o high )uality. It might loo" and eel great, read per ectly, and never crash, but to someone rom another locale, it might :ust seem plainold wrong. 2ssuring that the product is correctly localized gets you to this ne/t step. Content %hat would you thin" o a new so tware encyclopedia or the $.&. 8nglish mar"et i it had the content shown in ?igure 1<.4I 2igure 18>0> These 'ontent samples woul" seem strange in an 5meri'an English en'y'lope"ia>

117

In the $nited &tates, a soccer ball isn't the same thing as a ootballN Bou don't drive on the le tN These may not seem right to you, but in other countries they would be per ectly accurate. I you're testing a product that will be localized, you need to care ully e/amine the content to ma"e sure it's appropriate to the area where it will be used. +ontent is all the other 3stu 3 besides the code that goes into the product =see +hapter 4, 3The &o tware ,evelopment 7rocess3>. The ollowing list shows various types o content that you should care ully review or localization issues. ,on't consider it a complete listE there can be many more e/amples depending on the product. Thin" about what other items in your so tware might be problematic i it was sent to another country. &ample documents 7ictures 6ideo Icons &ounds #elp iles

!aps with disputed boundaries !ar"eting material 7ac"aging %eb lin"s

2 *.&8 T.. 0.*F In 199;, !icroso t released two products or "ids called +reative %riter and ?ine 2rtist. These products used a helper character named !cTee to guide the "ids through the so tware. 2 great deal o research went into the design o !cTee to select his loo", color, mannerisms, personality, and so on. #e turned out to be a rather strange loo"ing ellow with buc" teeth, dar" purple s"in, and a big nose. $n ortunately, a ter a great deal o wor" was done drawing the animations that would appear on the screen, a call came in rom one o !icroso t's oreign o ices. They had received a preliminary version o the so tware and a ter reviewing it said that it was unacceptable. The reason9 !cTee's nose was too long. In their culture, people with large noses weren't common and, right or wrong, they associated having a large nose with lots o negative stereotypes. They said that the product wouldn't sell i it was localized or their locale. It would have been way too costly to create two di erent !cTees, one or each mar"et, so the artwor" completely to that point was thrown out, and !cTee had his irst nose :ob. The bottom line is that the content that goes with the so tware, whether it's te/t, graphics, sounds, or whatever, is especially prone to having localization issues. Test the content with an eye or these types o problems and, i you're not e/perienced with the culture o the locale that the so tware is destined or, be sure to call in someone who is. %ata 2ormats

118

,i erent locales use di erent ormats or data units such as currency, time, and measurement. Hust as with content, these are localization, not translation, issues. 2n 2merican 8nglish publishing program that wor"s with inches couldn't simply undergo a te/t translation to use centimeters. It would re)uire code changes to alter the underlying ormulas, gridlines, and so on. Table 1<.1 shows many o the di erent categories o units that you'll need to become amiliar with i you're testing localized so tware. Table 1..1. (ata &ormat Consi'erations for *ocali/e' Software 0nit !easurements *umbers +urrency ,ates Times +alendars 2ddresses Telephone numbers 7aper sizes Consi'erations !etric or 8nglish9 meters vs. yards +omma, decimal, or space separatorsE how negatives are shownE K symbol or numberE 1.4<<,<< vs. 14<<.<< or 1<< vs. =1<<> ,i erent symbols and where they're placed9 ;<I vs. I;< .rder o month, day, yearE separatorsE leading zerosE long and short ormats9 ddSmmSyy vs. mmSddSyy or !ay G, 4<<G vs. 1G de mayo 4<<G 14-hour or 44-hour, separators ;9;<pm vs. 1G9;< ,i erent calendars and starting days9 In some countries &unday is not the irst day o the wee" .rder o linesE postal code used9 9A<74 vs. T4* <85 7arenthesis or dash separators9 =44G> GGG-1414 vs. 44G-GGG-1414 vs. 44G.GGG.1414 ,i erent paper and envelope sizes9 $& 0etter vs. 24

?ortunately, most operating systems designed or use in multiple locales support these di erent units and their ormats. ?igure 1<.G shows an e/ample rom %indows. #aving this built-in support ma"es it easier, but by no means oolproo , or programmers to write localized so tware. 2igure 18>1> The #in"ows Regional Settings options allow a user to sele't how numbers. 'urren'y. times. an" "ates will be "isplaye">

119

*.T8 #ow a unit is displayed isn't necessarily how it's treated internally by the so tware. ?or e/ample, the ,ate tab on the -egional &ettings program shows a short date style o mSdSyy. That doesn't imply that the operating system handles only a 4-digit year =and hence is a B41 bug>. In this case, the setting means only a 4-digit year is displayed. The operating system still supports a 4-digit year or computationsmore things to consider when testing. I you're testing localized so tware, you'll need to become very amiliar with the units o measure used by the target locale. To properly test the so tware, you'll need to create di erent e)uivalence partitions o test data rom the ones you create or testing the original version o the so tware. Configuration an" Compatibility !ssues The in ormation covered in +hapters A, 3+on iguration Testing,3 and 9, 3+ompatibility Testing,3 on con iguration and compatibility testing is very important when testing localized versions o so tware. The problems that can crop up when so tware interacts with di erent hardware and so tware are ampli ied by all the new and di erent combinations. 7er orming this testing isn't necessarily more di icult, :ust a bit larger o a tas". It can also ta/ your logistical s"ills to locate and ac)uire the oreign version o hardware and so tware to test with. 2oreign Platform Configurations %indows O7 supports 1<5 di erent languages and 55 di erent "eyboard layouts. It does this, as shown in ?igure 1<.5, through the 1eyboard 7roperties dialog via +ontrol 7anel. The drop-down list or languages runs rom 2 ri"aans to $"rainian and includes eight di erent versions o 8nglish other than 2merican 8nglish =2ustralian, (ritish, +anadian, +aribbean, Irish, Hamaican, *ew Tealand, and &outh 2 rican>, ive di erent Ferman dialects, and 4< di erent &panish dialects. 2igure 18>E> #in"ows supports the use of "ifferent keyboar"s an" languages through the -eyboar" Properties "ialog>

120

?igure 1<.7 shows e/amples o three di erent "eyboard layouts designed or di erent countries. Bou'll notice that each has "eys speci ic to its own language, but also has 8nglish characters. This is airly common, since 8nglish is o ten spo"en as a second language in many countries, and allows the "eyboard to be used with both native and 8nglish language so tware. 2igure 18>9> The 5rabi'. 2ren'h. an" Russian keyboar"s support 'hara'ters spe'ifi' to those languages> (www>fingertipsoft>'om

121

1eyboards are probably the piece o hardware with the largest language dependencies, but depending on what you're testing, there can be many others. 7rinters, or e/ample, would need to print all the characters your so tware sends to them and properly ormat the output on the various paper sizes used in di erent countries. I your so tware uses a modem, there might be issues related to the phone lines or communication protocol di erences. (asically, any peripheral that your so tware could potentially wor" with needs to be considered or a place in your e)uivalence partitions or plat orm con iguration and compatibility testing. *.T8 %hen designing your e)uivalence partitions, don't orget that you should consider all the hardware and so tware that can ma"e up the plat orm. This includes the hardware, device drivers or the hardware, and the operating system. -unning a ?rench printer on a !ac, with a (ritish operating system, and a Ferman version o your so tware might be a per ectly legitimate con iguration or your users. %ata Compatibility Hust as with plat orm con iguration testing, compatibility testing o data ta"es on a whole new meaning when you add localization to the e)uation. ?igure 1<.A shows how comple/ it can get moving data rom one application to another. In this e/ample, a Ferman application that uses metric units and e/tended characters can move data to a di erent ?rench program by saving and loading to dis" or using cut and paste. That ?rench application can then e/port the data or import to yet another 8nglish application. That 8nglish program, which uses 8nglish units and non-e/tended characters, can then move it all bac" to original Ferman program. 2igure 18>H> %ata 'ompatibility testing of lo'aliBe" software 'an get fairly 'omple(>

122

,uring this round and round o data trans ers, with all the conversions and handling o measurement units and e/tended characters, there are numerous places or bugs. &ome o these bugs might be due to design decisions. ?or e/ample, what should happen to data moved rom one application to another i it needs to change ormatsI &hould it be automatically converted, or should the user be prompted or a decisionI &hould it show an error or should the data :ust move and the units changeI These important )uestions need to be answered be ore you can start testing the compatibility o your localized so tware. 2s soon as you have those speci ications, your compatibility testing should proceed as it normally would:ust with more test cases in your e)uivalence partitions. =ow )u'h Shoul" 6ou Test$ The big uncertainty that looms over localization testing is in determining how much o the so tware you should test. I you spent si/ months testing the 2merican 8nglish version, should you spend si/ months testing a version localized into ?renchI &hould you spend even more because o additional con iguration and compatibility issuesI This comple/ issue comes down to two )uestions9 %as the pro:ect intended to be localized rom the very beginningI %as programming code changed to ma"e the localized versionI

I the so tware was designed rom the very beginning to account or all the things discussed in this chapter, the ris" is much smaller that a localized version will be very buggy and re)uire lots o testing. I , on the other hand, the so tware was written speci ically or the $.&. 8nglish mar"et and then it was decided to localize it into another language, it would probably be wise to treat the so tware as a completely new release re)uiring ull testing. *.T8 The amount o localization testing re)uired is a ris"-based decision, :ust as all testing is. 2s you gain e/perience in testing, you'll learn what variables go into the decision-ma"ing process. The other )uestion deals with what needs to change in the overall so tware product. I the localization e ort involves changing only content such as graphics and te/tnot codethe test e ort can sometimes be :ust a validation o the changes. I , however, because o poor design or other problems, the underlying code must change, the testing needs ta"e that into account and chec" unctionality as well as content. I& IT 0.+20IT2(08I

123

.ne method used by teams who "now they are going to localize their product is to test or localizability. That is, they test the irst version o the product, assuming that it will eventually be localized. The white-bo/ testers e/amine the code or te/t strings, proper handling o units o measure, e/tended characters, and other code-level issues. They may even create their own 3 a"e3 localized version. The blac"-bo/ testers care ully review the spec and the product itsel or localizing problems such as te/t in graphics and con iguration issues. They can use the 3 a"e3 version to test or compatibility. 8ventually, when the product is localized, many o the problems that would have shown up later have already been ound and i/ed, ma"ing the localization e ort much less pain ul and costly. Summary #a dn egy rXtermett Cs "Cpzett so tver ismerI, Cs olyC"onyan beszCl egy nyelvet az 2ngolon "ev[l, dn egy nagyon piac"Cpes sza""Cpzett szemCly. That's the same irst sentence o this chapteronly written in #ungarian this time. ,on't worry i you can't read it. Bou've learned in this chapter that "nowing the language is only part o the overall testing re)uired or a localized product. !uch wor" can be done by chec"ing the product or localizability and or testing language-independent areas. I you are luent in a language other than 8nglish, "eep reading this boo", and learn all you can about so tware testing. %ith the global economy and the worldwide adoption o technology and computers you will, as the #ungarian phrase roughly says, 3have a very mar"etable s"ill set.3 ?or more in ormation on localization programming and testing or %indows, visit www.microso t.comSglobaldev. ?or the !ac, consult the 2pple website, developer.apple.comSintlSlocalizationStools.html. 0inu/ programmers and testers can ind localization in ormation at www.linu/.comShowtosS#.%T.-I*,8OSother-lang.shtml. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. %hat's the di erence between translation and localizationI 4. ,o you need to "now the language to be able to test a localized productI ;. %hat is te/t e/pansion and what common bugs can occur because o itI 4. Identi y several areas where e/tended characters can cause problems. G. %hy is it important to "eep te/t strings out o the codeI 5. *ame a ew types o data ormats that could vary rom one localized program to another.

Chapter 11> <sability Testing I* T#I& +#27T8 $ser Inter ace Testing %hat !a"es a Food $II

124

Testing or the ,isabled9 2ccessibility Testing

&o tware is written to be used. That sounds pretty obvious, but it's sometimes orgotten in the rush to design, develop, and test a comple/ product. &o much time and e ort is spent on the technology aspects o writing the code that the development team ignores the most important aspect o so twarethat someone will eventually use the stu . It really doesn't matter whether the so tware is embedded in a microwave oven, a telephone switching station, or an Internet stoc" trading website. 8ventually the bits and bytes bubble up to where a live person will interact with it. $sability is how appropriate, unctional, and e ective that interaction is. Bou may have heard the term ergonomics, the science o designing everyday things so that they're easy and unctional to use. 2n ergonomist's main concern is in achieving usability. *ow, you're not going to get the "nowledge o a our-year ergonomics degree in the 1G or so pages o this chapter, nor do you need to. -emember rom +hapter 1, 3&o tware Testing (ac"ground,3 the i th rule o what constitutes a bug9 The so tware is di icult to understand, hard to use, slow, orin the so tware tester's eyeswill be viewed by the end user as :ust plain not right. That's your blan" chec" or usability testing. Bou're li"ely the irst person, other than the programmers, to use the so tware. Bou've become amiliar with the speci ication and investigated who the customers will be. I you have problems using the so tware while you're testing it, odds are the customers will, too. (ecause there are so many di erent types o so tware, it's impossible to go into detail about usability issues or all o them. $sability o a nuclear reactor shutdown se)uence is pretty di erent rom usability o a voicemail menu system. %hat you'll learn in this chapter are the basics o what to loo" orwith a bias toward so tware that you use on your 7+ every day. Bou can then ta"e those ideas and apply them to whatever so tware you have to test. #ighlights o this chapter include %hat usability testing involves %hat to loo" or when testing a user inter ace %hat special usability eatures are needed by the disabled

<ser !nterfa'e Testing The means that you use to interact with a so tware program is called its user inter ace or $I. 2ll so tware has some sort o $I. 7urists might argue that this isn't true, that so tware such as what's in your car to control the uelSair ratio in the engine doesn't have a user inter ace. In truth, it doesn't have a conventional $I, but the e/tra pressure you need to apply to the gas pedal and the audible sputtering you hear rom the tailpipe is indeed a user inter ace. The computer $I we're all amiliar with has changed over time. The original computers had toggle switches and light bulbs. 7aper tape, punch cards, and teletypes were popular user inter aces in the '5<s and '7<s. 6ideo monitors and simple line editors such as !&-,.& came ne/t. *ow we're using personal computers with sophisticated graphical user inter aces =F$Is>. &oon we'll be spea"ing and listening to our 7+s, carrying on verbal conversations as we do with peopleN 2lthough these $Is were very di erent, technically they all provided the same interaction with the computerthe means to give it input and receive output. #hat )akes a *oo" <!$ !any so tware companies spend large amounts o time and money researching the best way to design the user inter aces or their so tware. They use special usability labs run by ergonomic specialists. The labs are e)uipped with one-way mirrors and video cameras to record e/actly how people use their so tware. 8verything the users =sub:ects> do rom what "eys they press, how they use the mouse, what mista"es they ma"e, and what con uses them is analyzed to ma"e improvements to the $I.

125

Bou may be wondering what a so tware tester could possibly contribute with such a detailed and scienti ic process. (y the time the so tware is speci ied and written, it should have the per ect $I. (ut, i that's the case, why are there so many 6+-s blin"ing 149<<I ?irst, not every so tware development team designs their inter ace so scienti ically. !any $Is are :ust thrown together by the programmerswho may be good at writing code, but aren't necessarily ergonomics e/perts. .ther reasons might be that technological limitations or time constraints caused the $I to be sacri iced. 2s you learned in +hapter 1<, 3?oreign-0anguage Testing,3 the reason might be that the so tware wasn't properly localized. In the end, the so tware tester needs to assume the responsibility o testing the product's usability, and that includes its user inter ace. Bou might not eel that you're properly trained to test a $I, but you are. -emember, you don't have to design it. Bou :ust have to pretend you're the user and ind problems with it. #ere's a list o seven important traits common to a good $I. It doesn't matter i the $I is on a digital watch or is the !ac .& O inter ace, they all still apply. ?ollows standards and guidelines Intuitive +onsistent ?le/ible +om ortable +orrect $se ul

I you read a $I design boo", you may also see other traits being listed as important. !ost o them are inherent or ollow rom these seven. ?or e/ample, 3easy to learn3 isn't listed above, but i something is intuitive and consistent, it's probably easy to learn. 2s a tester, i you concentrate on ma"ing sure your so tware's $I meets these criteria, you'll have a darn good inter ace. 8ach trait is discussed in detail in the ollowing sections. 2ollows Stan"ar"s an" *ui"elines The single most important user inter ace trait is that your so tware ollows e/isting standards and guidelinesor has a really good reason not to. I your so tware is running on an e/isting plat orm such as !ac or %indows, the standards are set. 2pple's are de ined in the boo" !acintosh #uman Inter ace Fuidelines, published by 2ddison-%esley, also available online at developer.apple.comSdocumentationSmacS #IFuidelinesS#IFuidelines-4.html. !icroso t's are in the boo" !icroso t %indows $ser 8/perience, published by !icroso t 7ress, with the online version at msdn.microso t.comSlibrarySde ault.aspIurlLSlibrarySenusSdnwueShtmlSwelcome.asp. 8ach boo" goes into meticulous detail about how so tware that runs on each plat orm should loo" and eel to the user. 8verything is de ined rom when to use chec" bo/es instead o an option button =when both states o the choice are clearly opposite and unambiguous> to when it's proper to use the in ormation, warning, and critical messages as shown in ?igure 11.1. 2igure 11>1> %i" you e3er noti'e that there are three "ifferent le3els of messages in #in"ows$ #hen an" how to use ea'h one is "efine" in the user interfa'e stan"ar"s for #in"ows>

126

*.T8 I you're testing so tware that runs on a speci ic plat orm, you need to treat the standards and guidelines or that plat orm as an addendum to your product's speci ication. +reate test cases based on it :ust as you would rom the product's spec. These standards and guidelines were developed =hope ully> by e/perts in so tware usability. They have accounted or a great deal o ormal testing, e/perience, and trial and error to devise rules that wor" well or their users. I your so tware strictly ollows the rules, most o the other traits o a good $I will happen automatically. *ot all o them will because your team may want to improvise on them a bit, or the rules may not per ectly it with your so tware. In those cases, you need to really pay attention to usability issues. It's also possible that your plat orm doesn't have a standard, or maybe your so tware is the plat orm. In those situations, your design team will be the ones creating the usability standards or your so tware. Bou won't be able to ta"e or granted the rules that someone else has already igured out, and the remaining traits o a good user inter ace will be even more important or you to ollow. !ntuiti3e In 197G the !IT& =!icro Instrumentation Telemetry &ystems> 2ltair AA<< was released as one o the irst personal computers. Its user inter ace =see ?igure 11.4> was nothing but switches and lightsnot e/actly intuitive to use. 2igure 11>7> The )!TS 5ltair HH88 an" its less-than-intuiti3e user interfa'e> (Photo 'ourtesy of the Computer )useum of 5meri'a. www>'omputer-museum>org>

127

The 2ltair was designed or computer hobbyists, people who are a lot more orgiving o user inter ace issues. Today, users want much more out o their so tware than what the 2ltair AA<< provided. 8veryone rom grandmothers to little "ids to 7h.,.s are using computers in their daily lives. The computers with the most intuitive $Is are the ones that people don't even realize they're using. %hen you're testing a user inter ace, consider the ollowing things and how they might apply to gauging how intuitive your so tware is9 Is the user inter ace clean, unobtrusive, not busyI The $I shouldn't get in the way o what you want to do. The unctions you need or the response you're loo"ing or should be obvious and be there when you e/pect them. Is the $I organized and laid out wellI ,oes it allow you to easily get rom one unction to anotherI Is what to do ne/t obviousI 2t any point can you decide to do nothing or even bac" up or bac" outI 2re your inputs ac"nowledgedI ,o the menus or windows go too deepI Is there e/cessive unctionalityI ,oes the so tware attempt to do too much, either as a whole or in partI ,o too many eatures complicate your wor"I ,o you eel li"e you're getting in ormation overloadI I all else ails, does the help system really help youI

Consistent +onsistency within your so tware and with other so tware is a "ey attribute. $sers develop habits and e/pect that i they do something a certain way in one program, another will do the same operation the same way. ?igure 11.; shows an e/ample o how two %indows applications, which should be ollowing a standard, aren't consistent. In *otepad, ?ind is accessed through the &earch menu or by pressing ?;. In %ord7ad, a very similar program, it's accessed through the 8dit menu or by pressing +trlJ?. 2igure 11>D> #in"ows 4otepa" an" #or"Pa" are in'onsistent in how the 2in" feature is a''esse">

128

Inconsistencies such as this rustrate users as they move rom one program to another. It's even worse i the inconsistency is within the same program. I there's a standard or your so tware or your plat orm, ollow it. I not, pay particular attention to your so tware's eatures to ma"e sure that similar operations are per ormed similarly. Thin" about a ew basic items as you review your product9 &hortcut "eys and menu selections. In a voicemail system, pressing <, not other numbers, is almost always the 3get-out3 button that connects you to a real person. In %indows, pressing ?1 should always get you help. Terminology and naming. 2re the same terms used throughout the so twareI 2re eatures named consistentlyI ?or e/ample, is ?ind always called ?ind, or is it sometimes called &earchI 2udience. ,oes the so tware consistently tal" to the same audience levelI 2 un greeting card program with a color ul user inter ace shouldn't display error messages o arcane technobabble. 7lacement or buttons such as .1 and +ancel. ,id you ever notice that in %indows, .1 is always on the top or le t and +ancel on the right or bottomI The !ac .& places .1 on the right. 1eyboard e)uivalents to onscreen buttons should also be consistent. ?or e/ample, the 8sc "ey always does a cancel and 8nter does an .1.

2le(ible $sers li"e choicesnot too many, but enough to allow them to select what they want to do and how they want to do it. The %indows +alculator =see ?igure 11.4> has two views9 &tandard and &cienti ic. $sers can decide which one they need or their tas" or the one they're most com ortable using. 2igure 11>0> The #in"ows Cal'ulator shows its fle(ibility by ha3ing two "ifferent 3iews>

129

. course, with le/ibility comes comple/ity. In the +alculator e/ample you'll have a much larger test e ort than i there's :ust one view. The test impact o le/ibility is elt most in the areas covered in +hapter G, 3Testing the &o tware with (linders .n,3 with states and with data9 &tate :umping. ?le/ible so tware provides more options and more ways to accomplish the same tas". The result is additional paths among the di erent states o the so tware. Bour state transition diagrams can become much more comple/ and you'll need to spend more time deciding which interconnecting paths should be tested. &tate termination and s"ipping. This is most evident when so tware has power-user modes where a user who's very amiliar with the so tware can s"ip numerous prompts or windows and go directly to where they want to go. 2 voicemail system that allows you to directly punch in your party's e/tension is an e/ample. I you're testing so tware that allows this, you'll need to ma"e sure that all the state variables are correctly set i all the intermediate states are s"ipped or terminated early. ,ata input and output. $sers want di erent ways to enter their data and see their results. To put te/t into a %ord7ad document, you can type it, paste it, load it rom si/ di erent ile ormats, insert it as an ob:ect, or drag it with the mouse rom another program. The !icroso t 8/cel spreadsheet program allows you to view your data in 14 di erent standard and 4< di erent custom graphs. %ho even "new there were that many possibilitiesI Testing all the di erent ways to get data in and out o your so tware can very )uic"ly increase the e ort necessary and ma"e or tough choices when creating your e)uivalence partitions.

Comfortable &o tware should be com ortable to use. It shouldn't get in the way or ma"e it di icult or a user to do his wor". &o tware com ort is a pretty touchy- eely concept. -esearchers have spent their careers trying to ind the right ormula to ma"e so tware com ortable. It can be a di icult concept to )uanti y, but you can loo" or a ew things that will give you a better idea o how to identi y good and bad so tware com ort9 2ppropriateness. &o tware should loo" and eel proper or what it's doing and who it's or. 2 inancial business application should probably not go crazy with loud colors and sound e ects. 2 space game, on the other hand, will have much more leeway with the rules. &o tware should neither be too garish nor too plain or the tas" it's intended to per orm. 8rror handling. 2 program should warn users be ore a critical operation and allow users to restore data lost because o a mista"e. 7eople ta"e the $ndoS-edo eature or granted today, but it wasn't long ago that these eatures didn't e/ist.

130

7er ormance. (eing ast isn't always a good thing. !ore than one program has lashed error messages too )uic"ly to read. I an operation is slow, it should at least give the user eedbac" on how much longer it will ta"e and show that it's still wor"ing and hasn't rozen. &tatus bars, as shown in ?igure 11.G, are a popular way to accomplish this. 2igure 11>1> Status bars show how mu'h of the work has been 'omplete" an" how mu'h is left to go>

Corre't The com ort trait is admittedly a bit uzzy and o ten can be le t to interpretation. +orrectness, though, isn't. %hen you're testing or correctness, you're testing whether the $I does what it's supposed to do. ?igure 11.5 is an e/ample o a $I that isn't correct. 2igure 11>E> This software has a 'ompletely useless 5bort button>

This igure shows a message bo/ rom a popular page-scanning program or %indows. The bo/ appears when a scan is started and is supposed to provide a way or the user to stop the scan mid-process. $n ortunately, it doesn't wor". *ote that the cursor is an hourglass. 2n hourglass means =according to the %indows standard> that the so tware is busy and can't accept any input. Then why is the 2bort button thereI Bou can repeatedly clic" the 2bort button during the entire scan, which can ta"e a minute or more, and nothing happens. The scan continues uninterrupted until it completes. I clic"ing the 2bort button with the hourglass cursor did stop the scan, would that be a bugI Bou bet it wouldN +orrectness problems such as this are usually obvious and will be ound in your normal course o testing against the product speci ication. Bou should pay attention to some areas in particular, however9 !ar"eting di erences. 2re there e/tra or missing unctions, or unctions that per orm operations di erent rom what the mar"eting material saysI *otice that you're not comparing the so tware to the speci icationyou're comparing it to the sales in ormation. They're usually di erent. 0anguage and spelling. &ome programmers are poor spellers and writers and o ten create very interesting user messages. The ollowing is an order con irmation message rom a popular e-commerce websitehope ully i/ed by the time you read this9

131

I there are any discrepancies with the in ormation below, please contact us immediately to ensure timely delivery o the products that you ordered. (ad media. !edia is any supporting icons, images, sounds, or videos that go with your so tware's $I. Icons should be the same size and have the same palette. &ounds should all be o the same ormat and sampling rate. The correct ones should be displayed when chosen rom the $I. %B&I%BF =what you see is what you get>. !a"e sure that what the $I displays is really what you have. %hen you clic" the &ave button, is the document onscreen e/actly what's saved to dis"I %hen you load it bac", does it per ectly compare with the originalI %hen you print it, does the output per ectly match what's previewed on the screenI

<seful The inal trait o a good user inter ace is whether it's use ul. -emember, here you're not concerned with whether the so tware itsel is use ul, :ust whether the particular eature is. 2 popular term used in the so tware industry to describe unnecessary or gratuitous eatures is dancing bologna. Thin" o a sausage bouncing around on the screencompletely unnecessaryN %hen you're reviewing the product speci ication, preparing to test, or actually per orming your testing, as" yoursel i the eatures you see actually contribute to the so tware's value. ,o they help users do what the so tware is intended to doI I you don't thin" they're necessary, do some research to ind out why they're in the so tware. It's possible that there are reasons you're not aware o , or it could :ust be dancing bologna. Those super luous eatures, whether they be in a solitaire program or a heart monitor are bad or the user and mean e/tra testing or you. Testing for the %isable": 5''essibility Testing 2 serious topic that alls under the area o usability testing is that o accessibility testing or testing or the disabled. 2 1997 government &urvey o Income and 7rogram 7articipation =&I77> used by the $.&. +ensus (ureau ound that about G; million people =nearly 4<W o the population> in the country had some sort o disability. Table 11.1 shows a more detailed brea"down. Table 11.1. Peo#le with (isabilities Age <44 Percentage of Peo#le with (isabilities 1AW

4G44 1;W 4GG4 4;W GG54 ;5W 5G59 4GW 7<74 47W 7G79 GAW A<J 74W

132

+utting the data another way, reveals that 7.7 million people have di iculty seeing the words and letters in a newspaper. 1.A million people are legally blind and A million people have di iculty hearing. %ith our aging population and the penetration o technology into nearly every aspect o our lives, the usability o so tware becomes more important every day. 2lthough there are many types o disabilities, the ollowing ones ma"e using computers and so tware especially di icult9 6isual impairments. +olor blindness, e/treme near and ar sightedness, tunnel vision, dim vision, blurry vision, and cataracts are e/amples o visual limitations. 7eople with one or more o these would have their own uni)ue di iculty in using so tware. Thin" about trying to see where the mouse pointer is located or where te/t or small graphics appear onscreen. %hat i you couldn't see the screen at allI #earing impairments. &omeone may be partially or completely dea , have problems hearing certain re)uencies, or pic"ing a speci ic sound out o bac"ground noise. &uch a person may not be able to hear the sounds or voices that accompany an onscreen video, audible help, or system alerts. !otion impairments. ,isease or in:ury can cause a person to lose ine, gross, or total motor control o his hands or arms. It may be di icult or impossible or some people to properly use a "eyboard or a mouse. ?or e/ample, they may not be able to press more than one "ey at a time or may ind it impossible to press a "ey only once. 2ccurately moving a mouse may not be possible. +ognitive and language. ,ysle/ia and memory problems may ma"e it di icult or someone to use comple/ user inter aces. Thin" o the issues outlined previously in this chapter and how they might impact a person with cognitive and language di iculties.

,egal ReCuirements ?ortunately, developing so tware with a user inter ace that can be used by the disabled isn't :ust a good idea, a guideline, or a standardit's o ten the law. In the $nited &tates, three laws apply to this area and other countries are considering and adopting similar laws9 The 2mericans with ,isability 2ct states that businesses with 1G or mores employees must ma"e reasonable accommodations or employees, or potential employees, with disabilities. The 2,2 has recently been applied to commercial Internet websites, mandating that they be made accessible to the public who uses them. &ection G<A o the -ehabilitation 2ct is very similar to the 2,2 and applies to any organization that receives ederal unding. &ection 4GG o the Telecommunications 2ct re)uires that all hardware and so tware that trans ers in ormation over the Internet, a networ", or the phone lines be made so that it can be used by people with disabilities. I it's not directly usable, it must be compatible =see +hapter A, 3+on iguration Testing,3 and +hapter 9, 3+ompatibility Testing3> with e/isting hardware and so tware accessibility aids.

5''essibility 2eatures in Software &o tware can be made accessible in one o two ways. The easiest is to ta"e advantage o support built into its plat orm or operating system. %indows, !ac .&, Hava, and 0inu/ all support accessibility to some degree. Bour so tware only needs to adhere to the plat orm's standards or communicating with the "eyboard, mouse, sound card, and monitor to be

133

accessibility enabled. ?igure 11.7 shows an e/ample o the %indows accessibility settings control panel. 2igure 11>9> The #in"ows a''essibility features are set from this 'ontrol panel>

I the so tware you're testing doesn't run on these plat orms or is its own plat orm, it will need to have its own accessibility eatures speci ied, programmed, and tested. The latter case is obviously a much larger test e ort than the irst, but don't ta"e built-in support or granted, either. Bou'll need to test accessibility eatures in both situations to ma"e sure that they comply. *.T8 I you're testing usability or your product, be sure to create test cases speci ically or accessibility. Bou'll eel good "nowing that this area is thoroughly tested. 8ach plat orm is slightly di erent in the eatures that it o ers, but they all strive to ma"e it easier or applications to be accessibility enabled. %indows provides the ollowing capabilities9 &tic"y1eys allows the &hi t, +trl, or 2lt "eys to stay in e ect until the ne/t "ey is pressed. ?ilter1eys prevents brie , repeated =accidental> "eystro"es rom being recognized. Toggle1eys plays tones when the +aps 0oc", &croll 0oc", or *um0oc" "eyboard modes are enabled. &ound&entry creates a visual warning whenever the system generates a sound. &how&ounds tells programs to display captions or any sounds or speech they ma"e. These captions need to be programmed into your so tware. #igh +ontrast sets up the screen with colors and onts designed to be read by the visually impaired. ?igure 11.A shows an e/ample o this.

134

2igure 11>H> The #in"ows "esktop 'an be swit'he" to this high 'ontrast mo"e for easier 3iewing by the 3isually impaire">

!ouse1eys allows use o "eyboard "eys instead o the mouse to navigate. &erial1eys sets up a communications port to read in "eystro"es rom an e/ternal non"eyboard device. 2lthough the .& should ma"e these devices loo" li"e a standard "eyboard, it would be a good idea to add them to your con iguration testing e)uivalence partitions.

?or more in ormation about the accessibility eatures built into the popular .& plat orms, consult the ollowing websites9 www.microso t.comSenable www.apple.comSaccessibility www-;.ibm.comSable www.linu/.orgSdocsSldpShowtoS2ccessibility-#.%T.

Summary -emember this rom our de inition o a bug in +hapter 1I The so tware is di icult to understand, hard to use, slow, orin the so tware tester's eyeswill be viewed by the end user as :ust plain not right. 2s a so tware tester chec"ing the usability o a so tware product, you're li"ely the irst person to use the product in a meaning ul way, the irst person to see it all come together in its proposed inal orm. I it's hard to use or doesn't ma"e sense to you, customers will have the same issues. 2bove all, don't let the vagueness or sub:ectivity o usability testing hinder your test e ort. It's vague and sub:ective by nature. 8ven the e/perts who design the user inter aces will admit to thatwell, some o them will. I you're testing a new product's $I, re er to the lists in this chapter that de ine what ma"es or a good one. I it doesn't meet these criteria, it's a bug, and i it's a usability bug, it might :ust be the law.

135

AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N

1. True or ?alse9 2ll so tware has a user inter ace and there ore must be tested or
usability. 4. Is user inter ace design a science or an artI

;. I there's no de initive right or wrong user inter ace, how can it be testedI 4. 0ist some e/amples o poorly designed or inconsistent $Is in products you're amiliar with. G. %hat our types o disabilities could a ect so tware usabilityI 5. I you're testing so tware that will be accessibility enabled, what areas do you need to pay close attention toI

Chapter 17> Testing the %o'umentation I* T#I& +#27T8 Types o &o tware ,ocumentation The Importance o ,ocumentation Testing %hat to 0oo" or %hen -eviewing ,ocumentation The -ealities o ,ocumentation Testing

In +hapter 4, 3The &o tware ,evelopment 7rocess,3 you learned that there's a great deal o wor" and a great number o non-so tware pieces that ma"e up a so tware product. !uch o that non-so tware is its documentation. In simpler days, so tware documentation was at most a re$dAe ile copied onto the so tware's loppy dis" or a short 1-page insert put into the bo/. *ow it's much, much more, sometimes re)uiring more time and e ort to produce than the so tware itsel . 2s a so tware tester, you typically aren't constrained to :ust testing the so tware. Bour responsibility will li"ely cover all the parts that ma"e up the entire so tware product. 2ssuring that the documentation is correct is your :ob, too. In this chapter you'll learn about testing so tware documentation and how to include it in your overall so tware test e ort. #ighlights o this chapter include The di erent types o so tware documentation %hy documentation testing is important %hat to loo" or when testing documentation

Types of Software %o'umentation I your so tware's documentation consisted o nothing but a simple re$dAe ile, testing it wouldn't be a big deal. Bou'd ma"e sure that it included all the material that it was supposed

136

to, that everything was technically accurate, and = or good measure> you might run a spell chec" and a virus scan on the dis". That would be it. (ut, the days o documentation consisting o :ust a re$dAe ile are gone. Today, so tware documentation can ma"e up a huge portion o the overall product. &ometimes, it can seem as i the product is nothing but documentation with a little bit o so tware thrown in. #ere's a list o so tware components that can be classi ied as documentation. .bviously, not all so tware will have all the components, but it's possible9 7ac"aging te/t and graphics. This includes the bo/, carton, wrapping, and so on. The documentation might contain screen shots rom the so tware, lists o eatures, system re)uirements, and copyright in ormation. !ar"eting material, ads, and other inserts. These are all the pieces o paper you usually throw away, but they are important tools used to promote the sale o related so tware, add-on content, service contracts, and so on. The in ormation or them must be correct or a customer to ta"e them seriously. %arrantySregistration. This is the card that the customer ills out and sends in to register the so tware. It can also be part o the so tware, being displayed onscreen or the user to read, ac"nowledge, and complete online. 8$02. 7ronounced 3you-la,3 it stands or 8nd $ser 0icense 2greement. This is the legal document that the customer agrees to that says, among other things, that he won't copy the so tware nor sue the manu acturer i he's harmed by a bug. The 8$02 is sometimes printed on the envelope containing the mediathe loppy or +,. It also may pop up onscreen during the so tware's installation. 2n e/ample is shown in ?igure 14.1. 2igure 17>1> The E<,5 is part of the software+s "o'umentation an" e(plains the legal terms of use for the software>

137

0abels and stic"ers. These may appear on the media, on the bo/, or on the printed material. There may also be serial number stic"ers and labels that seal the 8$02 envelope. ?igure 14.4 shows an e/ample o a dis" label and all the in ormation that needs to be chec"ed. 2igure 17>7> There+s lots of "o'umentation on this "isk label for the software tester to 'he'k>

Installation and setup instructions. &ometimes this in ormation is printed directly on the discs, but it also can be included on the +, sleeve or as a +, :ewel bo/ insert. I it's comple/ so tware, there could be an entire installation manual. $ser's manual. The use ulness and le/ibility o online manuals has made printed manuals much less common than they once were. !ost so tware now comes with a small, concise 3getting started3type manual with the detailed in ormation moved to online ormat. The online manuals can be distributed on the so tware's media, on a website, or a combination o both. .nline help. .nline help o ten gets intertwined with the user's manual, sometimes even replacing it. .nline help is inde/ed and searchable, ma"ing it much easier or users to ind the in ormation they're loo"ing or. !any online help systems allow natural language )ueries so users can type Cell Ae how to )o2" te1t froA o%e 2ro-r$A to $%other and receive an appropriate response. Tutorials, wizards, and +(T =+omputer (ased Training>. These tools blend programming code and written documentation. They're o ten a mi/ture o both content and highlevel, macro-li"e programming and are o ten tied in with the online help system. 2 user can as" a )uestion and the so tware then guides him through the steps to

138

complete the tas". !icroso t's . ice 2ssistant, sometimes re erred to as the 3paper clip guy3 =see ?igure 14.;>, is an e/ample o such a system. 2igure 17>D> The )i'rosoft &ffi'e 5ssistant is an e(ample of a 3ery elaborate help an" tutorial system>

&amples, e/amples, and templates. 2n e/ample o these would be a word processor with orms or samples that a user can simply ill in to )uic"ly create pro essionalloo"ing results. 2 compiler could have snippets o code that demonstrate how to use certain aspects o the language. 8rror messages. These have already been discussed a couple times in this boo" as an o ten neglected area, but they ultimately all under the category o documentation.

The !mportan'e of %o'umentation Testing &o tware users consider all these individual non-so tware components parts o the overall so tware product. They don't care whether the pieces were created by a programmer, a writer, or a graphic artist. %hat they care about is the )uality o the entire pac"age. *.T8 I the installation instructions are wrong or i an incorrect error message leads them astray, users will view those as bugs with the so twareones that should have been ound by a so tware tester. Food so tware documentation contributes to the product's overall )uality in three ways9 It improves usability. -emember rom +hapter 11, 3$sability Testing,3 all the issues related to a product's usabilityI !uch o that usability is related to the so tware documentation. It improves reliability. -eliability is how stable and consistent the so tware is. ,oes it do what the user e/pects and when he e/pects itI I the user reads the documentation,

139

uses the so tware, and gets une/pected results, that's poor reliability. 2s you'll see in the rest o this chapter, testing the so tware and the documentation against each other is a good way to ind bugs in both o them. It lowers support costs. In +hapter 4 you learned that problems ound by a customer can cost 1< to 1<< times as much as i they were ound and i/ed early in the product's development. The reason is that users who are con used or run into une/pected problems will call the company or help, which is e/pensive. Food documentation can prevent these calls by ade)uately e/plaining and leading users through di icult areas.

*.T8 2s a so tware tester, you should treat the so tware's documentation with the same level o attention and give it the same level o e ort that you do the code. They are one in the same to the user. I you're not as"ed to test the documentation, be sure to raise this as an issue and wor" to have it included in your overall testing plan. #hat to ,ook for #hen Re3iewing %o'umentation Testing the documentation can occur on two di erent levels. I it's non-code, such as a printed user's manual or the pac"aging, testing is a static process much li"e what's described in +hapters 4, 38/amining the &peci ication,3 and 5, 38/amining the +ode.3 Thin" o it as technical editing or technical proo reading. I the documentation and code are more closely tied, such as with a hyperlin"ed online manual or with a help ul paper clip guy, it becomes a dynamic test e ort that should be chec"ed with the techni)ues you learned in +hapters G, 3Testing the &o tware with (linders .n,3 and 7, 3Testing the &o tware with O--ay Flasses.3 In this situation, you really are testing so tware. *.T8 %hether or not the documentation is code, a very e ective approach to testing it is to treat it :ust li"e a user would. -ead it care ully, ollow every step, e/amine every igure, and try every e/ample. I there is sample code, type it in and ma"e sure it wor"s as described. %ith this simple real-world approach, you'll ind bugs both in the so tware and the documentation. Table 14.1 is a simple chec"list to use as a basis or building your documentation test cases. Table 12.1. A (ocumentation Testing Chec1list 2hat to Chec1 2hat to Consi'er Feneral 2reas 2udience Terminology ,oes the documentation spea" to the correct level o audience, not too novice, not too advancedI Is the terminology proper or the audienceI 2re the terms used consistentlyI I acronyms or abbreviations are used, are they standard ones or do they need to be de inedI !a"e sure that your company's acronyms don't accidentally ma"e it through. 2re all the terms inde/ed and cross-re erenced correctlyI 2re the appropriate topics coveredI 2re any topics missingI #ow about topics that shouldn't be included, such as a eature that was cut rom the product and no one told the manual writer. Is the material covered in the proper depthI

+ontent and sub:ect matter

+orrectness Hust the acts Is all the in ormation actually and technically correctI 0oo" or mista"es caused by the writers wor"ing rom outdated specs or sales people in lating

140

Table 12.1. A (ocumentation Testing Chec1list 2hat to Chec1 2hat to Consi'er the truth. +hec" the table o contents, the inde/, and chapter re erences. Try the website $-0s. Is the product support phone number correctI Try it. &tep by step -ead all the te/t care ully and slowly. ?ollow the instructions e/actly. 2ssume nothingN -esist the temptation to ill in missing stepsE your customers won't "now what's missing. +ompare your results to the ones shown in the documentation.

+orrectness ?igures and screen captures &amples and e/amples &pelling and grammar +hec" igures or accuracy and precision. ,o they represent the correct image and is the image correctI !a"e sure that any screen captures aren't rom prerelease so tware that has since changed. 2re the igure captions correctI 0oad and use every sample :ust as a customer would. I it's code, type or copy it in and run it. There's nothing more embarrassing than samples that don't wor"and it happens all the timeN In an ideal world, these types o bugs wouldn't ma"e it through to you. &pelling and grammar chec"ers are too commonplace not to be used. It's possible, though, that someone orgot to per orm the chec" or that a specialized or technical term slipped through. It's also possible that the chec"ing had to be done manually, such as in a screen capture or a drawn igure. ,on't ta"e it or granted.

?inally, i the documentation is so tware driven, test it as you would the rest o the so tware. +hec" that the inde/ list is complete, that searching inds the correct results, and that the hyperlin"s and hotspots :ump to the correct pages. $se e)uivalence partition techni)ues to decide what test cases to try. The Realities of %o'umentation Testing To close this chapter, it's important or you to learn a ew things that ma"e documentation development and testing a bit di erent rom so tware development. +hapter ; was titled 3The -ealities o &o tware Testing.3 Bou might call these issues the realities o documentation testing9 ,ocumentation o ten gets the least attention, budget, and resources. There seems to be the mentality that it's a so tware pro:ect irst and oremost and all the other stu is less important. In reality, it's a so tware product that people are buying and all that other stu is at least as important as the bits and bytes. I you're responsible or testing an area o the so tware, ma"e sure that you budget time to test the documentation that goes along with that code. Five it the same attention that you do the so tware and i it has bugs, report them. It's possible that the people writing the documentation aren't e/perts in what the so tware does. Hust as you don't have to be an accounting e/pert to test a spreadsheet program, the writer doesn't have to be an e/pert in the so tware's eatures to write its documentation. 2s a result, you can't rely on the person creating the content to ma"e sense out o poorly written specs or comple/ or unclear product eatures. %or" closely with writers to ma"e sure they have the in ormation they need and that they're up-todate with the product's design. !ost importantly, tell them about di icult-to-use or

141

di icult-to-understand areas o the code that you discover so they can better e/plain those areas in the documentation. 7rinted documentation ta"es time to produce, sometimes wee"s or even months. (ecause o this time di erence, a so tware product's documentation may need to be inalizedloc"ed downbe ore the so tware is completed. I the so tware unctionality changes or bugs are discovered during this critical period, the documentation can't be changed to re lect them. That's why the re$dAe ile was invented. It's how those lastminute changes are o ten communicated to users. The solution to this problem is to have a good development model, ollow it, hold your documentation release to the last possible minute, and release as much documentation as possible, in electronic ormat, with the so tware.

Summary #ope ully this chapter opened your eyes to how much more there can be to a so tware product than the code the programmers write. The so tware's documentation, in all its orms, created by writers, illustrators, inde/ers, and so on, can easily ta"e more e ort to develop and test than the actual so tware. ?rom the user's standpoint, it's all the same product. 2n online help inde/ that's missing an important term, an incorrect step in the installation instructions, or a blatant misspelling are bugs :ust li"e any other so tware ailure. I you properly test the documentation, you'll ind the bugs be ore your users do. In the ne/t chapter you'll learn about applying your testing s"ills to an area that's in the news almost dailyso tware security. It's an area that will re)uire particular attention in every tas" you per orm as a so tware tester rom early code and speci ication reviews to testing the documentation. AuiB These )uiz )uestions are provided or your urther understanding. &ee 2ppendi/ 2, 32nswers to Muiz Muestions,3 or the answersbut don't pee"N 1. &tart up %indows 7aint =see ?igure 14.4> and loo" or several e/amples o documentation that should be tested. %hat did you indI 2igure 17>0> #hat e(amples of "o'umentation 'an you fin" in #in"ows Paint$

142

4. The %indows 7aint #elp Inde/ contains more than 4<< terms rom airbrush tool to zooming in or out. %ould you test that each o these ta"es you to the correct help topicsI %hat i there were 1<,<<< inde/ed termsI 3. True or ?alse9 Testing error messages alls under documentation testing. 4. In what three ways does good documentation contribute to the product's overall )ualityI Chapter 1D> Testing for Software Se'urity I* T#I& +#27T8 %arFamesthe !ovie $nderstanding the !otivation Threat !odeling Is &o tware &ecurity a ?eatureI Is &ecurity 6ulnerability a (ugI $nderstanding the (u er .verrun $sing &a e &tring ?unctions +omputer ?orensics

It seems as though a day doesn't go by without a news story about yet another computer security issue. #ac"ers, viruses, worms, spyware, bac"doors, Tro:an horses, and denial-o service attac"s have become common terms. 8ven average computer users have been impacted beyond the nuisance level, losing important data and valuable time restoring their systems a ter an attac". 2 Hanuary 14, 4<<G, story in the 02 Times by Hoseph !enn, titled, 3*o !ore Internet or Them,3 reveals that many people have had enough. They're rustrated, angry, and they are 3plugging out3disconnecting their computers rom the Internet in an e ort to regain control o their 7+s. That's not a good sign or the health and growth o the computer industry.

143

?or these reasons, so tware security is on every programmer's mind =or at least it should be i she wants to stay employed> and is touching all aspects o the so tware development process. &o tware testing has yet another area to be concerned with, and this chapter will give you an introduction to this important and timely topic. #ighlights o this chapter include %hy someone would want to brea" into a computer %hat types o brea"-ins are common #ow to wor" with your design team to identi y security issues %hy so tware security problems are nothing more than so tware bugs %hat you, as a so tware tester, can do to ind security vulnerabilities #ow the new ield o computer orensics is related to so tware security testing

#ar*amesthe )o3ie .ne o the irst mainstream e/amples o computer hac"ing that brought it into the public eye was the 19A; movie %arFames. In this movie, !athew (roderic"'s teenage character, ,avid 0ightman, uses his I!&2I A<A< home computer and a ;<< baud acoustic modem to hac" into the $& government's *.-2, computer system. .nce in, he stumbles across a war game scenario called 3Flobal Thermonuclear %ar3 that he thin"s is :ust a game but turns out to be a real-li e simulation that almost starts %orld %ar III. #ow did he gain accessI 7retty simply. #e programmed his computer to se)uentially dial phone numbers rom, say GGG-<<<< to GGG-9999, and listen or another computer modem to answer. I a person answered, the computer :ust hung up. #is program ran unattended while he was at school and created a list o 3hot3 numbers. 2 ter school, with that narrowed down list, ,avid called the numbers individually to see which ones would allow him to log-in. In the case o the *.-2, computer, he did some sleuthing to uncover the name o one o the government programmers and ound the name o his deceased son. Typing the son's name, H.&#$2, into the login password ield gained him ull access to the military's 3high security3 computer system. !ore than 4< years later, the technology has changed, but the processes and techni)ues remain the same. &o tware and computer systems are now much more interconnected and, un ortunately, there are many more hac"ers. Bou might say that %arFames has ta"en on a new meaning rom what it had in the moviethe hac"ers are playing games and the so tware industry is ighting a war against them. &o tware security, or more correctly, the lac" o it, has become a huge issue. #5R %R!:!4* %ith the popularity o wireless 3%i?i3 networ"s in metropolitan areas, resource ul hac"ers have discovered that they can search or open computer networ"s much li"e !athew (roderic"'s character in %arFames. They simply drive around city streets, sit outside corporate buildings, and wal" through co ee shops and restaurants with laptops or ine/pensive 3sni ers3 loo"ing or unprotected wireless networ"s. The techni)ue, named a ter the movie, is "nown as 3war driving.3 <n"erstan"ing the )oti3ation In the %arFames e/ample, the hac"er was motivated by curiosity and the desire to use a secure computer with more capability than his own. 2lthough he elt that his actions weren't malicious because he didn't plan to harm the system, his actions resulted in a cascade e ect

144

with serious conse)uences. 2s a so tware tester it's important to understand why someone may want to brea" into your so tware. $nderstanding their intent will aid you in thin"ing about where the security vulnerabilities might be in the so tware you're testing. 2 secure product is a product that protects the con identiality, integrity, and availability o the customers' in ormation, and the integrity and availability o processing resources, under control o the system's owner or administrator. www.microso t.comStechnetScommunitySchatsStransSsecuritySsec<514.msp/ 2 security vulnerability is a law in a product that ma"es it in easibleeven when using the product properlyto prevent an attac"er rom usurping privileges on the user's system, regulating its operation, compromising data on it, or assuming ungranted trust. www.microso t.comStechnetSarchiveScommunityScolumnsSsecuritySessaysSvul nrbl.msp/ #ac"er9 .ne who is pro icient at using or programming a computerE a computer bu . .ne who uses programming s"ills to gain illegal access to a computer networ" or ile. www.dictionary.com The ive motives that a hac"er might have to gain access to a system are +hallengeS7restige. The simplest and most benign orm o hac"ing is when someone brea"s into a system purely or the challenge o the tas" and the prestige =among his ellow hac"ers> o succeeding. There's no intent o anything more sinister. %ar driving is such an activity. 2lthough this may not sound li"e much o a problem, imagine i a loc" pic"er practiced his cra t by unloc"ing random doors throughout your neighborhood each night and then bragging to his riends which homes had loc"s that were the easiest to pic". *o one would eel secureand rightly so. +uriosity. The ne/t level up the scale is curiosity. #ere, the hac"er doesn't stop at :ust gaining access. .nce inside, he wants to loo" around to see what's there. +uriosity is the motive. The hac"er will peruse the system loo"ing or something interesting. 2 so tware system could have a security vulnerability that allows a hac"er to gain access =challengeSprestige> but still be secure enough to prevent the hac"er rom loo"ing at any o its valuable data. $seS0everage. This is the level where the hac"er does more than :ust brea"ing and entering. #ere the hac"er will actually attempt to use the system or his own purpose. The %arFames e/ample we've discussed is a $seS0everage attac". 2 present-day e/ample is when a home 7+ is attac"ed with an email virus, which then uses the email addresses stored on that 7+ and its computing power to re-send thousands more instances o the virus. The hac"er is able to accomplish much more using the distributed power o many computers than i using :ust his own. 2dditionally, the hac"er may be able to better cover his trac"s by using these 3hac"ed3 computers to do the damage.

145

6andalize. %hen you thin" o vandalizing, remember the three ,'s9 ,e acing, ,estruction, and ,enial o &ervice. ,e acing is changing the appearance o a website to promote the opinion or thoughts o the hac"ersee ?igure 1;.1. ,estruction ta"es the orm o deleting or altering o data stored on the system. 2n e/ample might be a college student changing his grades or deleting the inal e/am. ,enial o service is preventing or hindering the hac"ed system rom per orming its intended operation. 2n e/ample o this would be looding an ecommerce website with so much tra ic that it's incapable o handling its standard transactions, loc"ing out paying customers rom ma"ing purchases. 8ven worse, the hac"er could crash the system resulting in data loss and days o downtime. 2igure 1D>1> The "efa'ing of a website is ?ust one type of "amage that a ha'ker 'an infli't>

&teal. 7robably the most severe orm o hac"ing is stealing, outright the t. The intent is to ind something o value that can be used or sold. +redit card numbers, personal in ormation, goods and services, even login I,s and email addresses, all have value to the hac"er. In 4<<;, a 44-year-old computer hac"er gained access to and stole 94 million 2.0 screen names =login I,s>. The list o names was later sold several times to various spammers or amounts up to @1<<,<<<N *ot bad or a ew days' wor".

2re you a raid yetI Bou should be. 8very day there are countless hac"ers out there attempting to brea" into theoretically secure systems. 2nd, many o them succeed. *ow that you "now why a hac"er may want to brea" into a system, we'll continue our discussion o so tware security testing by outlining what you can do to assist your development team in creating a secure so tware product. Threat )o"eling In !ichael #oward and ,avid 0e(lanc's e/cellent boo", %riting &ecure +ode =!icroso t 7ress, 4<<;, second edition>, they describe a process "nown as threat modeling or evaluating a so tware system or security issues. Thin" o threat modeling as a variation o the ormal reviews described in +hapter 5, 38/amining the +ode.3 The goal in this case, though, is or the review team to loo" or areas o the product's eature set that are susceptible to security vulnerabilities. %ith that in ormation, the team can then choose to ma"e changes to the product, spend more e ort designing certain eatures, or concentrate testing on potential trouble spots. $ltimately, such an understanding will result in a more secure product.

146

*.T8 $nless everyone on the product development teamand that means everyone9 pro:ect managers, programmers, testers, technical writers, mar"eting, and product supportunderstands and agrees to the possible threats, your team will not be able to create a secure product. 7er orming the threat modeling is not the so tware tester's responsibility. This responsibility should all under the pro:ect manager's tas" list and involve all o the pro:ect team members or input. ?or this reason, we're only going to discuss the process in generalities. ?or more detailed in ormation on holding a threat modeling review, visit msdn.microso t.comSlibrary and search or 3threat modeling.3 ?igure 1;.4 shows a physical diagram o a ictional comple/ ecommerce system. *ote that the system has multiple entry methods or user access. $sers can access the system via 7,2s, wireless connections, and standard 7+s. The system is connected to the Internet and has server so tware rom di erent vendors. The system stores payment in ormation in a database that's directly accessible via an internal connection. The preliminary design o such a system re)uires comprehensive threat modeling to assure that it remains secure when put into operation. 2igure 1D>7> 5 'omple( system reCuires 'omprehensi3e threat mo"eling to i"entify se'urity 3ulnerabilities>

It really doesn't matter whether the so tware or system your team is developing is more or less comple/ than what's shown in ?igure 1;.4. The steps o the threat modeling process are the same. The ollowing steps were adapted rom msdn.microso t.comSlibrary. 2ssemble the threat modeling team. %e've already touched on this. (esides the standard team members, a vital addition would be someone with an e/tensive so tware security bac"ground. ?or a small team, this could be an outside consultant brought in :ust or the design stage. In a large company, this person could come rom a pool o e/perts who 3 loat3 rom one pro:ect to another o ering their "nowledge and e/pertise. It's important or the team to understand that their initial goal is not to solve the security issuesE it's to identi y them. 0ater meetings can be held with smaller speci ic groups = or e/ample, :ust a ew programmers and testers> to isolate the threats and devise solutions. Identi y the 2ssets. Thin" o all the things that your system has that might be o value to an intruder. Is there personal in ormation o your clientsI !aybe credit card

147

numbersI #ow about computing resourcesI +ould someone want to covertly use your system to send spam to your clientsI %ould you lose business i your company's website was de aced or replacedI +reate an 2rchitecture .verview. In this stage you'll want to identi y the technology you plan to use or your so tware and how it's interconnected. Bour team will create an architectural diagram that shows the ma:or technological pieces and how they communicate. 2n important aspect o this is identi ying the trust boundaries between the di erent technologies and the authentication and authorizations that must occur to gain access to the data. ,ecompose the 2pplication. This is a ormalized process to identi y how and where the data lows through the system. Ideally, it would be based on the design's data low diagrams and state transition maps. I these diagrams don't e/ist, they will need to be created. Thin" about the data as containers and what secures the containers. %hat are the means or entering the container to see the dataI Is the data encryptedI Is it password protectedI Identi y the Threats. .nce you thoroughly understand what the components =assets, architecture, and data> are, your team can move on to identi ying the threats. 8ach o the components should be considered threat targets and you should assume that they will be attac"ed. Thin" about whether each component could be improperly viewed. +ould it be modi iedI +ould a hac"er prevent authorized users rom using the componentI +ould someone gain access and ta"e control o the systemI ,ocument the Threats. 8ach threat must be documented and trac"ed to ensure that it is addressed. ,ocumentation is a simple matter o describing the threat =in one sentence or less>, the target, what orm the attac" might ta"e, and what countermeasures the system will use to prevent an attac". In +hapter 19, 3-eporting %hat Bou ?ind,3 you'll learn how to properly trac" the bugs you discover. $sing a similar system may help your team manage the security threats they identi y. -an" the threats. ?inally, it's important to understand that all threats are not created e)ual. Bour so tware's data may be protected with a ,epartment o ,e ense 14A-bit encryption that would ta"e decades on a super computer to brea". %ould the data be threatened even with that level o securityI &ure it would. %ould that threat ran" higher than an auto-answer phone modem connected to the administrator's computerI 7robably not. Bour team will need to loo" at each threat and determine its ran"ing. 2 simple way to do this is to use the ,-82, ?ormula de ined at msdn.microso t.comSlibrary under the chapter 3Improving %eb 2pplication &ecurity3. ,-82, stands or o ,amage 7otential #ow much damage =physical, inancial, integrity, and so on> could be done i this area is hac"edI o -eproducibility #ow easy is it or a hac"er to e/ploit the vulnerability consistentlyI %ould every attempt be success ulI .ne in 1<< attemptsI .ne in a millionI o 8/ploitability #ow technically di icult is it to gain access to the system or dataI Is it a ew lines o basic macro code that can be emailed around the Internet or does it re)uire someone with e/pert programming s"illsI o 2 ected $sers I the hac" succeeds, how many users will be a ectedI %ould it be a single user or, in the case o the stolen 2.0 screen names, 94 millionI o ,iscoverability %hat's the li"elihood that a hac"er will discover the vulnerabilityI 2 secret 3bac"door3 login may not sound discoverable, until a disgruntled employee is ired and posts the in ormation on the %eb. 2 simple 1L0ow, 4L!edium, and ;L#igh system applied to each o the ive categories and then summed up or a value between G and 1G would wor" to arrive at a numerical ran"ing or each threat. Bour team could plan to design and test the most severe issues irst, and then continue on to the other lower-ran"ed threats as time permits. In

148

+hapter 19 you'll learn more about rating and ran"ing so tware bugs and some techni)ues to use to ma"e sure they get i/ed. !s Software Se'urity a 2eature$ !s Se'urity :ulnerability a Bug$ (y now you've hope ully come to the conclusion that so tware security can be viewed as simply another eature o a so tware product or system. !ost people probably don't want their personal inancial data made public by a hac"er. They would consider their spreadsheet so tware's ability to "eep that in ormation private a necessary eature. That eature is one that would go toward ma"ing the so tware a good )uality product =remember our discussion o )uality versus reliability in +hapter ;, 3The -ealities o &o tware Testing3I> I the so tware 3 ailed3 and allowed a hac"er to see personal ban" balances and credit card in o, most users would consider that a so tware bug, pure and simple. -8!I*,8It's a good idea, here, to reiterate our de inition o a bug rom +hapter 1, 3&o tware Testing (ac"ground39 1. The so tware doesn't do something that the product speci ication says it should do. 4. The so tware does something that the product speci ication says it shouldn't do. ;. The so tware does something that the product speci ication doesn't mention. 4. The so tware doesn't do something that the product speci ication doesn't mention but should. G. The so tware is di icult to understand, hard to use, slow, orin the so tware tester's eyeswill be viewed by the end user as :ust plain not right. 2nd, to revisit our discussion o test-to-pass and test-to- ail rom +hapter G, 3Testing the &o tware with (linders .n.3 %hen you test-to-pass, you really only ensure that the so tware minimally wor"s. Bou don't push its capabilities. Bou don't see what you can do to brea" it. Bou treat it with "id gloves, applying the simplest and most straight orward test cases. 2 ter you assure yoursel that the so tware does what it's speci ied to do in ordinary circumstances, it's time to put on your snea"y, conniving, devious hat and attempt to ind bugs by trying things that should orce them outtest-to- ail. 2s a so tware tester, you may be responsible or testing the so tware's overall security or you may :ust be responsible or testing that your assigned eatures are secure. *o matter, you'll need to consider all ive de initions o a bug. Bou won't necessarily be given a product speci ication that e/plicitly de ines how so tware security is to be addressed in your so tware. *or will you be able to assume that the threat model is complete and accurate. ?or these reasons, you'll need to put on your 3test-to- ail3 hat and attac" the so tware much li"e a hac"er wouldassuming that every eature has a security vulnerability and that it's your :ob to ind it and e/ploit it. TI7 Testing or security bugs is a test-to- ail activity and one that will o ten cover areas o the product that haven't been completely understood or speci ied. <n"erstan"ing the Buffer &3errun It would be impossible in one chapter, or even one boo", to ade)uately cover all the possible means o attac"ing a so tware product. 2 ter all, a spreadsheet shared over your home's wireless networ" is very di erent rom a multiplayer video game played over the %eb or a distributed ,epartment o ,e ense computer system. The operating systems and the technologies are uni)ue and there ore will usually have di erent security vulnerabilities. There

149

is one common problem, however, that is a security issue in any so tware productthe bu er overrun. In the Feneric +ode -eview +hec"list in +hapter 5, you learned about ,ata -e erence 8rrorsbugs caused by using a variable, constant, array, string, or record that hasn't been properly declared or initialized or how it's being used and re erenced.3 2 bu er overrun is such an error. It is the result o poor programming, enabled by many languages such as + and +JJ, that lac" sa e string handling unctions. +onsider the sample + code in 0isting 1;.1. ,isting 1D>1> E(ample of a Simple Buffer &3erflow 1 ,oid A"7uffer*o2"()h$r * 2Jour)eJtr) K 2 )h$r 28e'tJtr[133]F 3 i%t %Lo)$lV$r1 5 123F 4 i%t %Lo)$lV$r2 5 456F 5 'tr)2"(28e'tJtr9 2Jour)eJtr)F ... 6 M 7 ,oid A"V$lid$te() 8 K 9 /* 13 0''uAe thi' fu%)tio%+' )ode ,$lid$te' $ u'er 2$''word 11 $%d -r$%t' $))e'' to Aillio%' of 2ri,$te )u'toAer re)ord' 12 /* 13 M ,o you see the problemI The size o the input string, 2Jour)eJtr, is un"nown. The size o the destination string, 28e'tJtr is 1<< bytes. %hat happens i the source string's length is greater than 1<<I 2s the code is written, the source string is copied right into the destination string, no matter what the length. I it's more than 1<< bytes, it will ill the destination string and then continue overwriting the values stored in the local variables. %orse, however, is i the source string is long enough, it could also overwrite the return address o the unction A"7uffer*o2" and the contents o the e/ecutable code in the unction A"V$lid$te(). In this e/ample, a competent hac"er could enter a super long password, stu ed with hand-written assembly code instead o alphanumeric 2&+II characters, and override the intended password validation per ormed in A"V$lid$tepossibly gaining access to the system. &uddenly, those code reviews described in +hapter 5 ta"e on a whole new meaningN *.T8 This is a greatly simpli ied e/ample o a bu er overrun to demonstrate the potential problem. 8/actly what data or program code gets overwritten, or even i it will be overwritten or e/ecuted at all, depends on the compiler and the +7$. (ut, o course, the hac"ers "now that. (u er overruns caused by improper handling o strings are by ar the most common coding error that can result in a security vulnerability, but any o the error classes described in +hapter 5 are potential problems. 2s a so tware tester, your :ob is to ind these types o bugs as early as possible. 2 code review would ind them early in the development cycle, but there's an even better meansand that's to prevent them rom happening in the irst place. <sing Safe String 2un'tions In 4<<4, !icroso t began an initiative to identi y the common + and +JJ unctions that were prone to bu er overrun coding errors. These unctions are not 3bad3 by themselves, but to be used securely they re)uire e/tensive error chec"ing on the part o the programmer. I that error chec"ing is neglected =and it o ten is>, the code will have a security vulnerability. Fiven the ris" o this oversight, it was decided that it would be best to develop and promote a new set o unctions to replace those prone to problems with a robust, thoroughly tested, and documented set o new ones.

150

These new unctions, called &a e &tring ?unctions, are available rom !icroso t or the %indows O7 &71 and later versions o the %indows ,,1 and 7lat orm &,1. !any other commercial and reeware libraries that implement 3sa e strings3 are available or common operating systems, processors, and compilers. The ollowing list =adapted rom the article, 3$sing &a e &tring ?unctions,3 on msdn.microso t.comSlibrary> e/plains some o the bene its o using the new unctions9 8ach unction receives the size o the destination bu er as input. The unction can thus ensure that it does not write past the end o the bu er. The unctions null-terminate all output strings, even i the operation truncates the intended result. The code per orming an operation on the returned string can sa ely assume that it will eventually encounter a nulldenoting the string's end. The data prior to the null will be valid and the string won't run on, inde initely. 2ll unctions return an *T&T2T$& value, with only one possible success code. The calling unction can easily determine i the unction succeeded in per orming its operation. 8ach unction is available in two versions. .ne supports single-byte 2&+II characters and the other, double-byte $nicode characters. -emember rom +hapter 1<, 3?oreign0anguage Testing,3 that to support all the letters and symbols in multiple oreign languages, characters need to ta"e up more than one byte o space.

Table 1;.1 shows a list o the unsa e unctions and the sa e unctions that replace them. %hen you and your team are per orming code reviews or white-bo/ testing, be on the loo"out or the unsa e unctions and how they are used. .bviously, your team's programmers should be using the sa e versions, but, i not, your code reviews will need to be per ormed with much more rigor to ensure that any possible security vulnerabilities are addressed. Table 13.1. The %l' 40nsecure4 C String &unctions an' Their 5ew 4Secure4 Re#lacements %l' 40nsafe4 &unctions 'tr)$t w)')$t 5ew 4Safe4 &unctions @tlJtri%-*#*$t @tlJtri%-*#*$tE1 @tlJtri%-*)h*$t @tlJtri%-*)h*$tE1 @tlJtri%-*#*$tD @tlJtri%-*#*$tDE1 @tlJtri%-*)h*$tD @tlJtri%-*)h*$tDE1 @tlJtri%-*#*o2" @tlJtri%-*#*o2"E1 @tlJtri%-*)h*o2" @tlJtri%-*)h*o2"E1 @tlJtri%-*#*o2"D @tlJtri%-*#*o2"DE1 @tlJtri%-*)h*o2"D Pur#ose +oncatenate two strings.

'tr%)$t w)'%)$t

+oncatenate two byte-counted strings, while limiting the size o the appended string.

Jtr)2" w)')2"

+opy a string into a bu er.

'tr%)2" w)'%)2"

+opy a byte-counted string into a bu er, while limiting the size o the copied string.

151

Table 13.1. The %l' 40nsecure4 C String &unctions an' Their 5ew 4Secure4 Re#lacements %l' 40nsafe4 &unctions 5ew 4Safe4 &unctions @tlJtri%-*)h*o2"DE1 'trle% w)'le% '2ri%tf 'w2ri%tf N'%2ri%tf N'%w2ri%tf ,'2ri%tf ,'w2ri%tf N,'%2ri%tf N,'%w2ri%tf @tlJtri%-*#Le%-th @tlJtri%-*)hLe%-th @tlJtri%-*#&ri%tf @tlJtri%-*#&ri%tfE1 @tlJtri%-*)h&ri%tf @tlJtri%-*)h&ri%tfE1 ,etermine the length o a supplied string. Pur#ose

+reate a ormatted te/t string that is based on a ormat string and a set o additional unction arguments.

@tlJtri%-*#V&ri%tf +reate a ormatted te/t string that is based on a @tlJtri%-*#V&ri%tfE1 ormat string and one additional unction argument. @tlJtri%-*)hV&ri%tf @tlJtri%-*)hV&ri%tfE1

T=E ;PE* :!R<S %hat could be more secure than a pictureI 2 ter all, it's data, not e/ecutable code. That alse assumption was bro"en in &eptember o 4<<4 when a virus was discovered that was embedded in several pornographic H78F images posted to an Internet newsgroup. %hen viewed, a virus was downloaded to the user's 7+. *o one thought it was possible, but it was. The problem lied in an e/ploitation o a bu er over low. The H78F ile ormat, besides storing the picture elements, also allows or the storing o embedded comments. !any so tware pac"ages or editing and organizing pictures use this ield or annotating the picture3.ur amily at the beach,3 3#ouse or &ale,3 and so orth. This comment ield starts with a he/ value o </???8 ollowed by a twobyte value. This value speci ies the length o the comment, plus 4 bytes = or the ield length>. $sing this encoding method, a comment o up to 5G,G;; bytes would be valid. I there is no comment, then the ield is supposed to contain a value o 4. The problem is that i the value is an illegal entry o < or 1, a bu er over low occurs. It turns out that the code used to interpret the H78F data and turn it into a viewable picture normalized the length by subtracting o the 4 bytes be ore it read out the actual comment. I the length byte was set to <, subtracting o 4 yielded a length o -4. The code was written to handle positive integers and interpreted the negative 4 as a positive 4F(. The ne/t 4F( o 3comment3 data was then loaded, improperly overwriting valid data and program. I that 3comment3 data was care ully cra ted, hand coded, assembly, it could be used to gain access to the viewer's 7+. !icroso t had to issue a critical update to all the components that loaded and viewed H78F images.

152

&o tware vulnerabilities can occur where you never e/pect them. ?or this reason, it's imperative to consider so tware security in all aspects o a so tware product or system. ?rom a user perspective, security is measure o )uality. They may not as" or it, but they "now they want it, and will consider lac" o so tware security a bug =a huge one> i they don't have it. %e'll close out this chapter with a brie visit to another aspect o computer security, one that's related to privacy, computer orensics. Computer 2orensi's $p to this point, we've discussed so tware security rom an active standpoint. %e loo"ed at it rom the perspective that a hac"er may try to actively manipulate your so tware by inding security vulnerabilities and e/ploiting them to create access to data or to control the system. 2nother perspective is that it doesn't have to be that di icult. &ometimes, the data is :ust lying around or the viewing by those who "now where to loo". The irst e/ample o this is with eatures that we're all amiliar with in browsing the %eb. ?igure 1;.; show an e/ample o Internet 8/plorer's drop-down address bar displaying the history list o websites that have been recently visited. ?or most users this is not a problemE it's actually use ul, allowing you to go bac" and )uic"ly return to sites you've visited without retyping their ull $-0s. (ut, what i this screen shot came rom a public access terminalI The person behind you in line could "now what you were viewing through a single mouse clic". 2igure 1D>D> The list of websites you+3e 3iewe" 'oul" be a se'urity 3ulnerability>

*.T8 ,ata that 3stays around3 and isn't deleted rom user to user is "nown as latent data. It should be considered a potential security vulnerability and needs to be discussed in any threat modeling your team does. It may not be considered a problem or your product. .r, it could be a ma:or issue. 2nother e/ample o latent data is the Foogle Toolbar 2uto?ill eature shown in ?igure 1;.4. This eature allows you to store in ormation such as your name, address, phone number, email address, and so on so that when a blan" orm is displayed =such as on an ecommerce order page> you can populate all the ields with a single clic". That's a great eature that many o us re)uently use. (ut, i you're testing a product or so tware security, you need to thin" li"e a user and decide i such a eature needs some type o 3override3 to hide or delete the data so it's not accessible to others.

153

2igure 1D>0> The *oogle Toolbar 5uto2ill feature stores information for Cui'kly filling in web forms> Se'urity 3ulnerability$

2 more comple/ e/ample o latent data is one used by computer security e/perts to discover evidence that could be used in a crime investigation. %hen data is written to a dis", it is written in bloc"s. The size o these bloc"s, called sectors, varies depending on the operating system. !&,.&S%indows uses G14-byte bloc"s. ,epending on the ile system being used, the sectors are written out in groups called clusters. The %indows ?2T ilesystem uses clusters o 4<4A bytes, each made up o our G14-byte clusters. ?igure 1;.G shows what two clusters might loo" li"e on a dis" drive a ter a te/t ile called readme.doc has been written to it. The ile is 44<< bytes in length and is shown by the white area spanning &ector 1 through the midpoint o &ector G. &o, i the ile is 44<< bytes, what's in the gray area rom the &ector G through &ector AI That in ormation is "nown as latent data. 2igure 1D>1> The "ata in the file rea"me>"o' is not the only "ata written to "isk>

154

I the ile is 44<< bytes long, it will ta"e up 4.; G14-byte bloc"s =44<<SG14L4.;>. The data at the end o &ector G is called -2! slac" because what is contained there is in ormation that happened to reside in the system's random access memory when the ile was created. It could be nothing, or it could be administration passwords or credit card numbers. There's no way to "now, but what is a given is that data other than what was in the ile was written rom the computer's memory onto its dis" drive. The remainder o the gray area, rom &ector 5 through the end o &ector A, is "nown as dis" slac". It e/ists because the ile system writes in 4<4A-byte clusters and our ile only contained enough data to partially ill two o them. The data stored here is typically data that e/isted on the drive be ore the ile was written. It could be the remnants o another ile or a previous, longer, version o readme.doc. This latent data could be benign, or it could contain in ormation that was intentionally deleted or is very private. *.T8 2lthough this e/ample uses a dis" drive to illustrate the concept o latent data, the security issues o -2! slac" and dis" slac" apply to writable +,s, ,6,s, memory cards, and virtually any sort o storage media. &imple, publicly available, tools can easily view and e/tract latent data rom a dis". &ome can even piece together scattered data ragments into their original complete iles. %hen you're testing a so tware product, you'll need to wor" with your team to decide i latent data is a security vulnerability issue. I it is, you and your team will need to devise ways to prevent it rom occurring. Chapter 10> #ebsite Testing I* T#I& +#27T8 %eb 7age ?undamentals (lac"-(o/ Testing Fray-(o/ Testing %hite-(o/ Testing +on iguration and +ompatibility Testing $sability Testing Introducing 2utomation

The testing techni)ues that you've learned in previous chapters have been airly generic. They've been presented by using small programs such as %indows %ord7ad, +alculator, and 7aint to demonstrate testing undamentals and how to apply them. This inal chapter o 7art III, 32pplying Bour Testing &"ills,3 is geared toward testing a speci ic type o so twareInternet web pages. It's a airly timely topic, something that you're li"ely amiliar with, and a good realworld e/ample to apply the techni)ues that you've learned so ar. %hat you'll ind in this chapter is that website testing encompasses many areas, including con iguration testing, compatibility testing, usability testing, documentation testing, security, and, i the site is intended or a worldwide audience, localization testing. . course, blac"bo/, white-bo/, static, and dynamic testing are always a given. This chapter isn't meant to be a complete how-to guide or testing Internet websites, but it will give you a straight orward practical e/ample o testing something real and give you a good head start i your irst :ob happens to be loo"ing or bugs in someone's website.

155

#ighlights o this chapter include %hat undamental parts o a web page need testing %hat basic white-bo/ and blac"-bo/ techni)ues apply to web page testing #ow con iguration and compatibility testing apply %hy usability testing is the primary concern o web pages #ow to use tools to help test your website

#eb Page 2un"amentals In the simplest terms, Internet web pages are :ust documents o te/t, pictures, sounds, video, and hyperlin"smuch li"e the +,--.! multimedia titles that were popular in the mid 199<s. 2s in those programs, web users can navigate rom page to page by clic"ing hyperlin"ed te/t or pictures, searching or words or phrases, and viewing the in ormation they ind. The Internet, though, has introduced two twists to the multimedia document concept that revolutionizes the technology9 $nli"e data that is stored solely on a +,--.!, web pages aren't constrained to a single 7+. $sers can lin" to and search worldwide across the entire Internet or in ormation on any website. %eb page authoring isn't limited to programmers using e/pensive and technical tools. The average person can create a simple web page almost as easily as writing a letter in a word processor.

(ut, :ust as giving someone a paint brush doesn't ma"e him an artist, giving someone the ability to create web pages doesn't ma"e him an e/pert in multimedia publishing. +ouple that with the technology e/plosion that continually adds new website eatures, and you have the per ect opportunity or a so tware tester. ?igure 14.1 shows a popular news website that demonstrates many o the possible web page eatures. 2 partial list o them includes Te/t o di erent sizes, onts, and colors =o"ay, you can't see the colors in this boo"> Fraphics and photos #yperlin"ed te/t and graphics -otating advertisements Te/t that has ,rop-down selection bo/es ?ields in which the users can enter data 2igure 10>1> 5 typi'al web page has many testable features>

156

2 great deal o comple/9

unctionality also isn't as obvious, eatures that ma"e the website much more

+ustomizable layout that allows users to change where in ormation is positioned onscreen +ustomizable content that allows users to select what news and in ormation they want to see ,ynamic drop-down selection bo/es ,ynamically changing te/t ,ynamic layout and optional in ormation based on screen resolution +ompatibility with di erent web browsers, browser versions, and hardware and so tware plat orms 0ots o hidden ormatting, tagging, and embedded in ormation that enhances the web page's usability

Franted, short o a secure e-commerce website, this is probably one o the more comple/ and eature-rich web pages on the Internet. I you have the tester mentality =and hope ully you've gained it by reading this ar in the boo">, loo"ing at such a web page should whet your appetite to :ump in and start inding bugs. The remainder o this chapter will give you clues on where to loo". Bla'k-Bo( Testing -emember all the way bac" to +hapters 4 through 7, the ones that covered the undamentals o testingI In those vitally important chapters, you learned about blac"-bo/, white-bo/, static, and dynamic testingthe raw s"ills o a so tware tester. %eb pages are the per ect means to practice what you've learned. Bou don't have to go out and buy di erent programsyou can simply :ump to a web page, one o your avorites or a completely new one, and begin testing.

157

The easiest place to start is by treating the web page or the entire website as a blac" bo/. Bou don't "now anything about how it wor"s, you don't have a speci ication, you :ust have the website in ront o you to test. %hat do you loo" orI ?igure 14.4 shows a screen image o 2pple's website, www.apple.com, a airly straight orward and typical website. It has all the basic elementste/t, graphics, hyperlin"s to other pages on the site, and hyperlin"s to other websites. 2 ew o the pages have orm ields in which users can enter in ormation and a ew pages play videos. .ne interesting thing about this site that's not so common is that it's localized or 47 di erent locales, rom 2sia to the $1. 2igure 10>7> #hat woul" you test in a straightforwar" website su'h as this$

I you have access to the Internet, ta"e some time now and e/plore 2pple's website. Thin" about how you would approach testing it. %hat would you testI %hat would your e)uivalence partitions beI %hat would you choose not to testI 2 ter e/ploring a bit, what did you decideI #ope ully you realized that it's a pretty big :ob. I you loo"ed at the site map =www.apple.comS indSsitemap.html>, you ound lin"s to more than 1<< di erent sub-sites, each one with several pages. *.T8 %hen testing a website, you irst should create a state table =see +hapter G, 3Testing the &o tware with (linders .n3>, treating each page as a di erent state with the hyperlin"s as the lines connecting them. 2 completed state map will give you a better view o the overall tas". Than" ully, most o the pages are airly simple, made up o :ust te/t, graphics, lin"s, and the occasional orm. Testing them isn't di icult. The ollowing sections give some ideas o what to loo" or. Te(t %eb page te/t should be treated :ust li"e documentation and tested as described in +hapter 14, 3Testing the ,ocumentation.3 +hec" the audience level, the terminology, the content and sub:ect matter, the accuracyespecially o in ormation that can become outdatedand always, always chec" spelling. *.T8 ,on't rely on spell chec"ers to be per ect, especially when they're used on web page content. They might only chec" the regular te/t but not what's contained in the graphics, scrolling

158

mar)uees, orms, and so on. Bou could per orm what you thin" is a complete spell chec" and still have misspellings on the page. I there is contact in ormation such as email addresses, phone numbers, or postal addresses, chec" them to ma"e sure that they're correct. !a"e sure that the copyright notices are correct and dated appropriately. Test that each page has a correct title. This te/t appears in the browser's title bar =upper-le t corner o ?igure 14.4> and what is listed by de ault when you add the page to your avorites or boo"mar"s. 2n o ten overloo"ed type o te/t is called 20T te/t, or 20Ternate te/t. ?igure 14.; shows an e/ample o 20T te/t. %hen a user puts the mouse cursor over a graphic on the page he gets a pop-up description o what the graphic represents. %eb browsers that don't display graphics use 20T te/t. 2lso, with 20T te/t blind users can use graphically rich websitesan audible reader interprets the 20T te/t and reads it out through the computer's spea"ers. 2igure 10>D> 5,T te(t pro3i"es te(tual "es'riptions of graphi's images on web pages>

*.T8 *ot all browsers support display o the 20T te/t. &ome only show simple TIT08 te/t in the tooltipor nothing at all. &ince this could prevent the blind rom using the %eb, it should be considered a serious accessibility bug. +hec" or te/t layout issues by resizing your browser window to be very small or very large. This will reveal bugs where the designer or programmer assumed a i/ed page width or height. It will also reveal hard-coded ormatting such as line brea"s that might loo" good with certain layouts but not with others. =yperlinks 0in"s can be tied to te/t or graphics. 8ach lin" should be chec"ed to ma"e sure that it :umps to the correct destination and opens in the correct window. I you don't have a speci ication or the website, you'll need to test whether the :ump wor"ed correctly. !a"e sure that hyperlin"s are obvious. Te/t lin"s are usually underlined, and the mouse pointer should change =usually to a hand pointer> when it's over any "ind o hyperlin"te/t or graphic.

159

I the lin" opens up an email message, ill out the message, send it, and ma"e sure you get a response. 0oo" or orphan pages, which are part o the website but can't be accessed through a hyperlin" because someone orgot to hoo" them up. Bou'll li"ely need to get a list rom the web page's designer o the e/pected pages and compare that with your own state table. *raphi's !any possible bugs with graphics are covered later under usability testing, but you can chec" a ew obvious things with a simple blac"-bo/ approach. ?or e/ample, do all graphics load and display properlyI I a graphic is missing or is incorrectly named, it won't load and the web page will display an error where the graphic was to be placed =see ?igure 14.4>. 2igure 10>0> !f a graphi' 'an+t loa" onto a web page an error is put in its lo'ation>

I te/t and graphics are intermi/ed on the page, ma"e sure that the te/t wraps properly around the graphics. Try resizing the browser's window to see i strange wrapping occurs around the graphic. #ow's the per ormance o loading the pageI 2re there so many graphics on the page, resulting in a large amount o data to be trans erred and displayed, that the website's per ormance is too slowI %hat happens i you test on a slow dialup connection instead o your company's ast local area networ"I 2orms ?orms are the te/t bo/es, list bo/es, and other ields or entering or selecting in ormation on a web page. ?igure 14.G shows a simple e/ample rom 2pple's website. It's a signup orm or potential !ac developers. There are ields or entering your irst name, middle initial, last name, and email address. There's an obvious bug on this pagehope ully it's i/ed by the time you read this. 2igure 10>1> )ake sure your website+s form fiel"s are positione" properly> 4oti'e in this 5pple %e3eloper signup form that the mi""le initial ()>!> fiel" is mispla'e">

160

Test orms :ust as you would i they were ields in a regular so tware programremember +hapter GI 2re the ields the correct sizeI ,o they accept the correct data and re:ect the wrong dataI Is there proper con irmation when you inally press 8nterI 2re optional ields truly optional and the re)uired ones truly re)uiredI %hat happens i you enter 9999999999999999999999999999I &b?e'ts an" &ther Simple )is'ellaneous 2un'tionality Bour website may contain eatures such as a hit counter, scrolling mar)uee te/t, changing advertisements, or internal site searches =not to be con used with search engines that search the entire %eb>. %hen planning your tests or a website, ta"e care to identi y all the eatures present on each page. Treat each uni)ue eature as you would a eature in a regular program and test it individually with the standard testing techni)ues that you've learned. ,oes it have its own statesI ,oes it handle dataI +ould it have ranges or boundariesI %hat test cases apply and how should they be e)uivalence classedI 2 web page is :ust li"e any other so tware. *ray-Bo( Testing Bou're already amiliar with blac"-bo/ and white-bo/ testing, but another type o testing, graybo/ testing, is a mi/ture o the twohence the name. %ith gray-bo/ testing, you straddle the line between blac"-bo/ and white-bo/ testing. Bou still test the so tware as a blac"-bo/, but you supplement the wor" by ta"ing a pee" =not a ull loo", as in white-bo/ testing> at what ma"es the so tware wor". %eb pages lend themselves nicely to gray-bo/ testing. !ost web pages are built with #T!0 =#yperte/t !ar"up 0anguage>. 0isting 14.1 shows a ew lines o the #T!0 used to create the web page shown in ?igure 14.5. ,isting 10>1> Sample =T), Showing Some of #hat+s Behin" a #eb Page #eb Page 2un"amentals In the simplest terms, Internet web pages are :ust documents o te/t, pictures, sounds, video, and hyperlin"smuch li"e the +,--.! multimedia titles that were popular in the mid 199<s. 2s in those programs, web users can navigate rom page to page by clic"ing hyperlin"ed te/t or pictures, searching or words or phrases, and viewing the in ormation they ind.

161

The Internet, though, has introduced two twists to the multimedia document concept that revolutionizes the technology9 $nli"e data that is stored solely on a +,--.!, web pages aren't constrained to a single 7+. $sers can lin" to and search worldwide across the entire Internet or in ormation on any website. %eb page authoring isn't limited to programmers using e/pensive and technical tools. The average person can create a simple web page almost as easily as writing a letter in a word processor.

(ut, :ust as giving someone a paint brush doesn't ma"e him an artist, giving someone the ability to create web pages doesn't ma"e him an e/pert in multimedia publishing. +ouple that with the technology e/plosion that continually adds new website eatures, and you have the per ect opportunity or a so tware tester. ?igure 14.1 shows a popular news website that demonstrates many o the possible web page eatures. 2 partial list o them includes Te/t o di erent sizes, onts, and colors =o"ay, you can't see the colors in this boo"> Fraphics and photos #yperlin"ed te/t and graphics -otating advertisements Te/t that has ,rop-down selection bo/es ?ields in which the users can enter data 2igure 10>1> 5 typi'al web page has many testable features>

2 great deal o comple/9

unctionality also isn't as obvious, eatures that ma"e the website much more

162

+ustomizable layout that allows users to change where in ormation is positioned onscreen +ustomizable content that allows users to select what news and in ormation they want to see ,ynamic drop-down selection bo/es ,ynamically changing te/t ,ynamic layout and optional in ormation based on screen resolution +ompatibility with di erent web browsers, browser versions, and hardware and so tware plat orms 0ots o hidden ormatting, tagging, and embedded in ormation that enhances the web page's usability

Franted, short o a secure e-commerce website, this is probably one o the more comple/ and eature-rich web pages on the Internet. I you have the tester mentality =and hope ully you've gained it by reading this ar in the boo">, loo"ing at such a web page should whet your appetite to :ump in and start inding bugs. The remainder o this chapter will give you clues on where to loo". Bla'k-Bo( Testing -emember all the way bac" to +hapters 4 through 7, the ones that covered the undamentals o testingI In those vitally important chapters, you learned about blac"-bo/, white-bo/, static, and dynamic testingthe raw s"ills o a so tware tester. %eb pages are the per ect means to practice what you've learned. Bou don't have to go out and buy di erent programsyou can simply :ump to a web page, one o your avorites or a completely new one, and begin testing. The easiest place to start is by treating the web page or the entire website as a blac" bo/. Bou don't "now anything about how it wor"s, you don't have a speci ication, you :ust have the website in ront o you to test. %hat do you loo" orI ?igure 14.4 shows a screen image o 2pple's website, www.apple.com, a airly straight orward and typical website. It has all the basic elementste/t, graphics, hyperlin"s to other pages on the site, and hyperlin"s to other websites. 2 ew o the pages have orm ields in which users can enter in ormation and a ew pages play videos. .ne interesting thing about this site that's not so common is that it's localized or 47 di erent locales, rom 2sia to the $1. 2igure 10>7> #hat woul" you test in a straightforwar" website su'h as this$

163

I you have access to the Internet, ta"e some time now and e/plore 2pple's website. Thin" about how you would approach testing it. %hat would you testI %hat would your e)uivalence partitions beI %hat would you choose not to testI 2 ter e/ploring a bit, what did you decideI #ope ully you realized that it's a pretty big :ob. I you loo"ed at the site map =www.apple.comS indSsitemap.html>, you ound lin"s to more than 1<< di erent sub-sites, each one with several pages. *.T8 %hen testing a website, you irst should create a state table =see +hapter G, 3Testing the &o tware with (linders .n3>, treating each page as a di erent state with the hyperlin"s as the lines connecting them. 2 completed state map will give you a better view o the overall tas". Than" ully, most o the pages are airly simple, made up o :ust te/t, graphics, lin"s, and the occasional orm. Testing them isn't di icult. The ollowing sections give some ideas o what to loo" or. Te(t %eb page te/t should be treated :ust li"e documentation and tested as described in +hapter 14, 3Testing the ,ocumentation.3 +hec" the audience level, the terminology, the content and sub:ect matter, the accuracyespecially o in ormation that can become outdatedand always, always chec" spelling. *.T8 ,on't rely on spell chec"ers to be per ect, especially when they're used on web page content. They might only chec" the regular te/t but not what's contained in the graphics, scrolling mar)uees, orms, and so on. Bou could per orm what you thin" is a complete spell chec" and still have misspellings on the page. I there is contact in ormation such as email addresses, phone numbers, or postal addresses, chec" them to ma"e sure that they're correct. !a"e sure that the copyright notices are correct and dated appropriately. Test that each page has a correct title. This te/t appears in the browser's title bar =upper-le t corner o ?igure 14.4> and what is listed by de ault when you add the page to your avorites or boo"mar"s. 2n o ten overloo"ed type o te/t is called 20T te/t, or 20Ternate te/t. ?igure 14.; shows an e/ample o 20T te/t. %hen a user puts the mouse cursor over a graphic on the page he gets a pop-up description o what the graphic represents. %eb browsers that don't display graphics

164

use 20T te/t. 2lso, with 20T te/t blind users can use graphically rich websitesan audible reader interprets the 20T te/t and reads it out through the computer's spea"ers. 2igure 10>D> 5,T te(t pro3i"es te(tual "es'riptions of graphi's images on web pages>

*.T8 *ot all browsers support display o the 20T te/t. &ome only show simple TIT08 te/t in the tooltipor nothing at all. &ince this could prevent the blind rom using the %eb, it should be considered a serious accessibility bug.

+hec" or te/t layout issues by resizing your browser window to be very small or very large. This will reveal bugs where the designer or programmer assumed a i/ed page width or height. It will also reveal hard-coded ormatting such as line brea"s that might loo" good with certain layouts but not with others. =yperlinks 0in"s can be tied to te/t or graphics. 8ach lin" should be chec"ed to ma"e sure that it :umps to the correct destination and opens in the correct window. I you don't have a speci ication or the website, you'll need to test whether the :ump wor"ed correctly. !a"e sure that hyperlin"s are obvious. Te/t lin"s are usually underlined, and the mouse pointer should change =usually to a hand pointer> when it's over any "ind o hyperlin"te/t or graphic. I the lin" opens up an email message, ill out the message, send it, and ma"e sure you get a response. 0oo" or orphan pages, which are part o the website but can't be accessed through a hyperlin" because someone orgot to hoo" them up. Bou'll li"ely need to get a list rom the web page's designer o the e/pected pages and compare that with your own state table. *raphi's

165

!any possible bugs with graphics are covered later under usability testing, but you can chec" a ew obvious things with a simple blac"-bo/ approach. ?or e/ample, do all graphics load and display properlyI I a graphic is missing or is incorrectly named, it won't load and the web page will display an error where the graphic was to be placed =see ?igure 14.4>. 2igure 10>0> !f a graphi' 'an+t loa" onto a web page an error is put in its lo'ation>

I te/t and graphics are intermi/ed on the page, ma"e sure that the te/t wraps properly around the graphics. Try resizing the browser's window to see i strange wrapping occurs around the graphic. #ow's the per ormance o loading the pageI 2re there so many graphics on the page, resulting in a large amount o data to be trans erred and displayed, that the website's per ormance is too slowI %hat happens i you test on a slow dialup connection instead o your company's ast local area networ"I 2orms ?orms are the te/t bo/es, list bo/es, and other ields or entering or selecting in ormation on a web page. ?igure 14.G shows a simple e/ample rom 2pple's website. It's a signup orm or potential !ac developers. There are ields or entering your irst name, middle initial, last name, and email address. There's an obvious bug on this pagehope ully it's i/ed by the time you read this. 2igure 10>1> )ake sure your website+s form fiel"s are positione" properly> 4oti'e in this 5pple %e3eloper signup form that the mi""le initial ()>!> fiel" is mispla'e">

Test orms :ust as you would i they were ields in a regular so tware programremember +hapter GI 2re the ields the correct sizeI ,o they accept the correct data and re:ect the wrong dataI Is there proper con irmation when you inally press 8nterI 2re optional ields truly optional and the re)uired ones truly re)uiredI %hat happens i you enter 9999999999999999999999999999I

166

&b?e'ts an" &ther Simple )is'ellaneous 2un'tionality Bour website may contain eatures such as a hit counter, scrolling mar)uee te/t, changing advertisements, or internal site searches =not to be con used with search engines that search the entire %eb>. %hen planning your tests or a website, ta"e care to identi y all the eatures present on each page. Treat each uni)ue eature as you would a eature in a regular program and test it individually with the standard testing techni)ues that you've learned. ,oes it have its own statesI ,oes it handle dataI +ould it have ranges or boundariesI %hat test cases apply and how should they be e)uivalence classedI 2 web page is :ust li"e any other so tware. *ray-Bo( Testing Bou're already amiliar with blac"-bo/ and white-bo/ testing, but another type o testing, graybo/ testing, is a mi/ture o the twohence the name. %ith gray-bo/ testing, you straddle the line between blac"-bo/ and white-bo/ testing. Bou still test the so tware as a blac"-bo/, but you supplement the wor" by ta"ing a pee" =not a ull loo", as in white-bo/ testing> at what ma"es the so tware wor". %eb pages lend themselves nicely to gray-bo/ testing. !ost web pages are built with #T!0 =#yperte/t !ar"up 0anguage>. 0isting 14.1 shows a ew lines o the #T!0 used to create the web page shown in ?igure 14.5. ,isting 10>1> Sample =T), Showing Some of #hat+s Behin" a #eb Page

2igure 10>E> Part of this web page is 'reate" by the =T), in ,isting 10>1>

167

*.T8 I you're not amiliar with creating your own website, you might want to read a little on the sub:ect. 2n introductory boo" such as &ams Teach Boursel to +reate %eb 7ages in 44 #ours, would be a great way to learn the basics and help you discover a ew ways to apply gray-bo/ testing techni)ues.

#T!0 and web pages can be simply tested as a gray bo/ because #T!0 isn't a programming language that's been compiled and inaccessible to the testerit's a mar"up language. In the early days o word processors, you couldn't :ust select te/t and ma"e it bold or italic. Bou had to embed mar"ups, sometimes called ield tags, in the te/t. ?or e/ample, to create the bolded phrase This is bold te/t. you would enter something such as this into your word processor9 Qbegin boldRThis is bold te/t.Qend boldR #T!0 wor"s the same way. To create the line in #T!0 you would enter This is bol" te(t> #T!0 has evolved to where it now has hundreds o di erent ield tags and options, as evidenced by the #T!0 in 0isting 14.1. (ut, in the end, #T!0 is nothing but a ancy old-wordprocessor-li"e mar"up language. The di erence between #T!0 and a program is that #T!0 doesn't e/ecute or run, it :ust determines how te/t and graphics appear onscreen. TI7

168

To see the #T!0 or a web page, right-clic" a blan" area o the page =not on a graphic> and select 6iew &ource =I8> or 6iew 7age &ource =?ire o/> rom the menu.

&ince #T!0 isn't a programming language and is so easy or you, as the tester, to view, you might as well ta"e advantage o it and supplement your testing. I you're a blac"-bo/ tester, it's the per ect opportunity to start moving toward white-bo/ testing. &tart by learning to create your own simple web pages. 0earn the basic and common #T!0 tags. 0oo" at the #T!0 or many di erent pages on the %eb, see what techni)ues are used and how those techni)ues ma"e things wor" on the page. .nce you become amiliar with #T!0, you'll be able to loo" at the web pages you're testing in a whole new way and be a more e ective tester. #hite-Bo( Testing In ?igure 14.1, you saw an e/ample o a web page with much static content in the orm o te/t and images. This static content was most li"ely created with straight #T!0. That same web page also has customizable and dynamic changing content. -emember, #T!0 isn't a programming languageit's merely a tagging system or te/t and graphics. To create these e/tra dynamic eatures re)uires the #T!0 to be supplemented with programming code that can e/ecute and ollow decision paths. Bou've li"ely heard o the popular web programming languages and technologies that can be used to create these types o eatures9 ,#T!0, Hava, Hava&cript, 2ctiveO, 6(&cript, 7erl, +FI, 2&7, and O!0. 2s e/plained in +hapters 5, 38/amining the +ode,3 and 7, 3Testing the &o tware with O--ay Flasses,3 to apply white-bo/ testing, you don't necessarily need to become an e/pert in these languages, :ust amiliar enough to be able to read and understand them and to devise test cases based on what you see in the code. This chapter can't possibly go into all the details o white-bo/ testing a website, but several eatures could be more e ectively tested with a white-bo/ approach. . course, they could also be tested as a blac"-bo/, but the potential comple/ity is such that to really ma"e sure you ind the important bugs that you have some "nowledge o the website's system structure and programming9 ,ynamic +ontent. ,ynamic content is graphics and te/t that changes based on certain conditions or e/ample, the time o day, the user's pre erences, or speci ic user actions. It's possible that the programming or the content is done in a simple scripting language such as Hava&cript and is embedded within the #T!0. This is "nown as client-side programming. I it is, you can apply gray-bo/ testing techni)ues when you e/amine the script and view the #T!0. ?or e iciency, most dynamic content programming is located on the website's serverE it's called server-side programming and would re)uire you to have access to the web server to view the code. ,atabase-,riven %eb 7ages. !any e-commerce web pages that show catalogs or inventories are database driven. The #T!0 provides a simple layout or the web content and then pictures, te/t descriptions, pricing in ormation, and so on are pulled rom a database on the website's server and plugged into the pages. 7rogrammatically +reated %eb 7ages. !any web pages, especially ones with dynamic content, are programmatically generatedthat is, the #T!0 and possibly even the programming is created by so tware. 2 web page designer may type entries in a database and drag and drop elements in a layout program, press a button, and out comes the #T!0 that displays a web page. I this sounds scary, it's really no di erent

169

than a computer language compiler creating machine code. I you're testing such a system, you have to chec" that the #T!0 it creates is what the designer e/pects. &erver 7er ormance and 0oading. 7opular websites might receive millions o individual hits a day. 8ach one re)uires a download o data rom the website's server to the browser's computer. I you wanted to test a system or per ormance and loading, you'd have to ind a way to simulate the millions o connections and downloads. +hapter 1G, 32utomated Testing and Test Tools,3 introduces the techni)ues and tools you can use to do this. &ecurity. 2s you learned in the previous chapter, website security issues are always in the news as hac"ers try new and di erent ways to gain access to a website's internal data. ?inancial, medical, and other websites that contain personal data are especially at ris" and re)uire intimate "nowledge o server technology to test them or proper security.

Configuration an" Compatibility Testing It's time to get bac" to what you can do, today, to test a web page. -ecall rom +hapters A, 3+on iguration Testing,3 and 9, 3+ompatibility Testing,3 what con iguration and compatibility testing are. +on iguration testing is the process o chec"ing the operation o your so tware with various types o hardware and so tware plat orms and their di erent settings. +ompatibility testing is chec"ing your so tware's operation with other so tware. %eb pages are per ect e/amples o where you can apply this type o testing. 2ssume that you have a website to test. Bou need to thin" about what the possible hardware and so tware con igurations might be that could a ect the operation or appearance o the site. #ere's a list to consider9 #ardware 7lat orm. Is it a !ac, 7+, 7,2, !&*T6, or a %i?i wristwatchI 8ach hardware device has its own operating system, screen layout, communications so tware, and so on. 8ach can a ect how the website appears onscreen. (rowser &o tware and 6ersion. There are many di erent web browsers and browser versions. &ome run on only one type o hardware plat orm, others run on multiple plat orms. &ome e/amples are 2.0 9.<, ?ire o/ 1.<, Internet 8/plorer G.< and 5.<, 7oc"et I8, *etscape 7.4, and .pera 7.G4. 8ach browser and version supports a di erent set o eatures. 2 website may loo" great under one browser and not display at all under another. %eb designers can choose to design a site using the least common denominator o eatures so that it loo"s the same on all o them, or write specialized code to ma"e the site wor" best on each one. They may even choose to only support a ew o the most popular browsers. #ow would this impact your testingI (rowser 7lug-Ins. !any browsers can accept plug-ins or e/tensions to gain additional unctionality. 2n e/ample o this would be to play speci ic types o audio or video iles. (rowser .ptions. !ost web browsers allow or a great deal o customization. ?igure 14.7 shows an e/ample o this. Bou can select security options, choose how 20T te/t is handled, decide what plug-ins to enable, and so on. 8ach option has potential impact on how your website operatesand, hence, is a test scenario to consider. 2igure 10>9> This e(ample shows how 'onfigurable the !nternet E(plorer web browser is>

170

6ideo -esolution and +olor ,epth. !any plat orms can display in various screen resolutions and colors. 2 7+ running %indows, or e/ample, can have screen dimensions o 54</4A<, A<</5<<, 1,<44/75A, 14A</1<44, and up. !obile devices have tiny screens with very di erent resolutions. Bour website may loo" di erent, or even wrong, in one resolution, but not in another. Te/t and graphics can wrap di erently, be cut o , or not appear at all. The number o colors that the plat orm supports can also impact the loo" o your site. 7+s typically support as ew as 4G5 colors and as many as 4 ;4. +ould your website be used on a mobile system with only 15 colorsI

Te/t &ize. ,id you "now that a user can change the size o the te/t used in the browserI +ould your site be used with very small or very large te/tI %hat i it was being run on a small screen, in a low resolution, with large te/tI !odem &peeds. 8nough can't be said about per ormance. &omeday everyone will have high-speed connections with website data delivered as ast as you can view it. $ntil then, you need to test that your website wor"s well at a wide range o modem speeds.

I you consider all the possibilities outlined here, testing even the simplest website can become a huge tas". It's not enough that the website loo"s good on your 7+i you want to ensure that it wor"s well or its intended audience, you need to research the possible con igurations they might have. %ith that in ormation, you can create e)uivalence partitions o the con igurations you eel are most important to test. 2 good place to start your research is www.websidestory.com and www.upsdell.comS(rowser*ewsSstat.htm. These sites have re)uently updated surveys related to technology, browsers, video resolutions, etc. These are a great irst step in deciding what con igurations are popular and which direction they are trending.

171

172