Chapter 5 - Testing

SE - Vasile Stoicu-Tivadar 1 / 36
Objectives
for the 7-th Chapter
Testing the Software

•Who does it ?
•The process of testing
•The five phases of testing
•Techniques to make testing easier
•Object-oriented testing
•The look-and-see method
•Blueprint-based testing
•Constructing behavior samples
•Testing constructors for correct initialization
•State transition –based test methodology
What is
Who does
Testing – the process of uncovering errors (“bugs”) in code
What
(at the last stage - to determine if it works as defined in the
functional specification)
About errors, faults and failures see [A4], page 278-
Debugging – the process of removing them
It’s often been said that the worst person to test an application is the programmer
Who
who wrote it. This isn't actually true - the worst person is that programmer's mother,
because they never want to find fault with their son or daughter (but something
similar is with the programmer as well ! )
Why ? does
•the programmer knows how computers work, and thus wouldn’t try
something illogical or out of sequence.
Every programmer has had the experience of listening to a user describe
a set of steps they followed, and shaking their head, thinking, "Why in
the world would you do that?"
Who does the testing ?
•the programmer’s natural mind-set is to make something work – not to try to break it.
And when it’s their own baby they’re testing, they’re definitely predisposed toward being
faint of heart when it comes to intentional demolition.
BUT despite their best intentions, the programmer has a vested interest (if only
subconsciously) in not finding any bugs in the application.
Regardless of which is true, it’s clear that someone other than the programmer should
test the application. Who?
There are basically two sources of testing personnel:

•A dedicated testing staff, or
•A select group of end-users.
Who does the testing – continued 1
Advantages of using a dedicated testing
staff :
1. the control over getting the testing done. If they’re responsible for the testing and also control the resources
needed to do the testing, they can be assured that the testing will get done. If the customer is responsible for
testing, it may or may not get done, depending on other demands on the customers’ time.
2. a dedicated testing staff will more likely know how to perform testing.A technically oriented person has to lay
out the proper procedures.
A corollary to this is that a dedicated staff will also become increasingly skilled at testing. They’ll learn the
tips and tricks. They’ll know the ins and outs. A dedicated testing staff will become more skilled at it than
people who are thrown into the task for a one-time shot. And furthermore, a dedicated testing staff, used
to working in tandem with the same programming team, develops specialized knowledge of the way the
programming team works – their styles and attendant flaws.
3. a dedicated testing team will also acquire special tools and resources, in addition to the growing level of skill.
A customer isn’t ordinarily going to acquire specialized software tools for a one-time use, while a dedicated
team may find them a valuable investment, both for their own productivity as well as turning out higher quality
work.
And as they learn how to use those tools, subsequent uses will be less expensive than if the one-time
user were to ambitiously undertake the same effort. Other resources, such as a range of test beds of
data, specialized machines which allow testing in a variety of environments, and so on, are also more
likely to be available to a dedicated testing staff than to the customer’s user.
4. the programming team can be assured that the testing was done properly and completely. When the testing
team works "for" the programming team, the programmers can confirm and audit that the testing was
performed as it was supposed to be.
Disadvantages of a full-time testing staff

1. they're subject to doing substandard work, leaving in the middle of a project, and a
hundred other things that you've undoubtedly seen yourself (but these things aren't reasons in
and of themselves to avoid a full-time testing staff)
2. they are often expensive. It's not as easy to invoice a customer for "testing" - the perception
is that testing is part of the project, and thus should be "included" or, worse, "free." If you've gone
the fixed price route, then testing needs to be factored in. This additional factor adds complexity,
and thus, uncertainty, to the mix.
3. it's simply difficult to find and retain talented testing personnel. By its nature, testing is
repetitive work and can become boring quickly. At the same time, it's still "processor intensive" -
meaning that testing personnel have to be bright and motivated - qualities which are at odds with
repetitive, boring work.
One solution - to use testing as a training ground for potential programmers or other
technical staff - sounds ideal, but often those people can find full-time development jobs at
other firms without having to go through "an apprenticeship."
The advantages of having the users do the final
acceptance testing
There are variations to consider. Let’s look at the advantages of the customer becoming involved as
the primary testing resource at the end, based on the assumption that the customer should, and is
going to, provide daily and ongoing support for the application.
Why make that assumption? Just as a manufacturing company would hire a contractor to build
an addition, but use their own maintenance people for tasks like moving office partitions, a user
department would likely contract out for a large custom software development project. But they
wouldn’t want to rely on that staff for daily support of the application such as adding users,
changing vendor code numbers, or instructions on how to run the monthly posting transactions
in two parts. As a result, it’s important for the users to fully understand how to use the application
and what’s behind it. Then, when business conditions change, they can take full advantage of all
of the flexibility in the application, instead of having to assume nothing works the minute that the
environment changes.
-> via training, but it’s a long and expensive process, and is generally not very effective. Due to a
variety of constraints - such as scheduling people together at the same time - training on a large,
complex system is often done in a few full-day sessions. This is often not enough time to cover a
system thoroughly enough.
->The alternative, then, is to have the users who will be working with the system become
involved in the testing. By doing so, they’ll be able to spend more time with the system – perhaps by
spending a couple of hours a day over several weeks or months, instead of a week-long crash course.
Thus, they learn exactly what the system was intended to do, and they become expert in running the
system through all its paces.
Another advantage to the users doing the final testing is a lower cost. Meanwhile, their existing staff
is already paid for in the daily operations budget, so it "appears" to be free.
The disadvantages of having the users do the final

acceptance testing
By now, the disadvantages of having the users perform the bulk of the
testing should be fairly obvious.
Are they going to do it?
Can they do it well?
Will they do it on time?
How can the developers depend on the testing being done
properly?
Users have other jobs, and because of this, are prone to being less than
reliable in terms of getting the job done correctly and on time, if at all.
WHAT is the solution ?
The optimal solution is a combination The best application of your limited resources - and
technical talent is just that - is toward the design and programming of systems. Bring in high
school and college students to do some of the functional testing, and then partner with the
appropriate customers who can and are willing to join in at the final stage to do the functional
testing.
And have someone on staff whose primary responsibility is testing. You could even think of
that person as being the QA department – for the most part, testing, but with other
responsibilities as well. These other responsibilities could involve the monitoring and handling of
your processes. Once your development staff stretches past a few people, the number of forms
and mechanisms to track grows. Verbal communication simply does not work. This QA person
makes sure that this process is running smoothly – in effect, performing QA on a macro scale as
well as with the details.
Including the customer in the functional testing doesn’t mean that the programming team
just disappears upon shipment of a late-beta CD. Their role during this phase of final testing
becomes that of "training mentor" and "technology transfer guide."
As training mentors, the programming team is still responsible for developing test plans
and instructing the customer’s testing staff on how to test properly.
In the technology transfer role, they’re responsible for training the testing staff as to the
functionality designed and implemented. As detailed as a functional specification is, it can
never cover every imaginable scenario. The programming team is also responsible for
training the customer’s staff about the internal technical details of the application.
The process of testing
5 steps:
Unit testing - the functionality of an individual task or module is compared
against the specification
Integration testing - is defined as "do all the pieces work together" and the
level of detail and amount of effort will be proportional to "how many pieces" you've
got. Just as with Unit Testing, you have to have a definition of what the app is
supposed to do and how it's going to work. In this case - what are the pieces in the
game, and what are the rules and expectations of how the pieces are supposed to
interact?
System testing - does the app work in the production environment?

Acceptance testing - this is where the customer checks over what you've
delivered, and pronounces it fit
Regression testing - refers to the practice of testing an application after a

change was made to the application to make sure that the change (or fix) didn’t
break anything else.
Module testing
There are several testing techniques: (see [A4], page 290)
•Code Review
•Code Walk-Throughts– less formal -collegial, in which the source
code is analyzed by colleagues from the test team; the programmers
explain what they have done (bu using documentation as well); the
listeners make the effort to understand what has been done and in
this way can help to discover mistakes; the discussion is led by
designers and the focus is on the code, not on the people involved,
finding errors, not eliminating them; managers may be present.
•Code Inspection –more formal, supposes preparing actions by
assembling a team, the initial studying the code, establish a list of
objectives, organizing an analysis meeting. Just now are checked the
algorithms corectness, the performances, the interfaces a.s.o. The
study is individual. During this meeting ech team member presents
their own conclusions that will be further discussed.
11 / 36
•Formal Demonstration Techniques (might be automated) - suppose the

analysis of the correctness of a program through the perspective of the analysis
of the correctness (validity) of its representation (model) from a formal,
mathematical point of view, according to rigorous principles, analogous to any
other mathematical modeling.
These techniques ignore the structure and syntax of the code and therefore only
demonstrate the validity of a program structure (in terms of design) and not its
implementation.
[A4], page 293
Other techniques also take into account the characteristics of the language:
•Symbolic execution - involves the simulated execution of a code, using
symbols instead of data; the test program is considered to have an initial state
determined by inputs and conditions; the program is executed, which goes
through successive states; all these states are retained and are analyzed
whether the inputs generated the transitions in the correct states.
[A4], page 296
12 / 36
Selection of test cases is difficult because full testing cannot be done
(all input combinations should be generated) and therefore a small set of test
stimuli must be chosen. There are two approaches:
- closed-box - the test system is viewed only from the perspective of inputs and
outputs
- open-box - the internal structure is also taken into account when testing
To know if the test was performed correctly (actually acceptable), the

completeness of the test cases must be analyzed:
- states: test cases are considered so that the system goes through all the
states
- branches: similarly, go through all the branches in the program at least once
- paths: similar, all paths
and so on
The following are steps for module testing in that the modules are components
of a complex software system and module testing means testing component
applications: 13 / 36
Step one: push all the buttons
First, QA creates a list of every menu and submenu option, and verifies that each one either
operates as expected, or has been, at the current time, stubbed out because that task hasn't yet
been finished.
Then QA creates a list of every object on a screen, and checks off each object as it’s tested. All the
buttons are pushed, all the checkboxes are checked, and data is entered into each of the fields. Tab
order is another item that is checked in this phase - when you tab through the screen, does the
focus shift appropriately - and, do all the keyboard shortcuts work?
Step two: test the functionality
->To verify that the basic functionality of each object is in place.
When they press the Add button, enter data into every field, and then press Save, is the
record actually added? Does all the data get added? Did the entire entry in each field get
saved, or did the last five characters in the invoice number get chopped off?
QA need be able to verify what is happening in the application down to the raw table level.
In a practical sense, steps one and two are usually performed together, but these make up two
separate columns in the checklist.
The process of testing.
Unit testing-continued 1
Step three: test the rules
There are three general types of rules in an application - those attached to a field, those attached to
a form, and application-wide rules. In all three cases, these rules have to be identified and then
those rules have to be tested.
Identifying those rules is easy; they're in the functional specification! However, simply having a list
of rules isn't enough. The real work here falls on the shoulders of
the QA staff has to write up a detailed test plan that will verify those rules.
Once QA writes the test plan, they'll pass the test plan by the developer to make sure it's robust
and fairly covers the functionality. QA and the developer may work together to put together a case
chart to handle all of the possible options within a program and provide test data or require a test
plan to handle these cases. Finally, QA will execute it. As QA tests each option, they should be able
to check off which options were tested and what the result was.
Unit testing-continued 2
Step four: break it!
As developers, we know our application works. Right? We don’t need anybody to prove to us that it
works - we’ve run the application after writing the code and we’ve seen that it works. Of course,
we’ve only seen that it works one time - why would we waste our time trying it more than once?
Up to this point, QA has simply been verifying functionality - much like an inspector would verify that
all the items on the packing list are indeed in the box. But the ultimate mission of QA is not to prove
that the application will work - their mission is to break the application.
So the final step in the testing process is to repeat the first three steps, but this time, pulling out all
the stops to be devious, conniving, and downright evil in trying to find ways to break the application.
The greatest fear of a testing person is that there’s a bug in the system that they didn’t find.
Testing Process
Integration testing
(see A4 , pag. 303 )
Diverse approaches of introducing successively the software

modules into the whole architecture and appropriately testing
them :
• Bottom up: one introduces the software modules
successively, starting from the bottom of the modules
hierarchy
• Top-down: … starting from the Top of the modules
hierarchy
• big-bang: putting together all the modules (does not
work for complex software)
• sandwich: simultaneousy, two teams begin, one from the
bottom, the other from the top of the hierarchys (for large or
very large - huge applications)
IP - Vasile Stoicu-Tivadar 17 / 36
System testing
Does the app work in the production environment? I know of shops who
actually set up test environments that mimic the production environment of the
customer, and do a separate round of system testing in that environment. As
you can imagine, I think this is quite a luxury, both in terms of having the
resources to do so, and the time (and money) to do yet another round of
testing. After all, someone is paying for the testing, and, ultimately, it's the
customer.
However, no matter how much you try, it's going to be near on impossible to
precisely duplicate the customer's environment. As an example, witness the
fragility of Windows (before 2000) even after a decade of use all over the
world. Optimally, your application will have to have some bodacious exception
handling for unexpected environmental conditions and situations
Acceptance testing
This maps directly to the inspection department in a factory, where boxes from the
receiving dock are opened and the parts are examined to make sure they meet the
standards of the customer.
It's a rare customer who will actually perform full blown acceptance testing; even
rarer is the customer who, after doing a poor job of acceptance testing, will not
complain about finding defects in the application.
You can help this situation to an extent by providing test scripts - even your own test
plans, if you like - for your customer to use for their acceptance testing. If they feel
that a lot of their work has been done for them, they may be more inclined to pitch in
and do their part.
Be aware that if your user still doesn’t use the materials you’ve given them, you’ll
have to run through them yourselves – you’ll suffer in the end, regardless of "whose
fault" it was that testing wasn’t done.
If you don’t have a model office as described in the Integration Testing section,
another option you might consider for user acceptance testing is a test environment
at a customer’s site. You have to be careful to isolate it enough so it can’t impact
their actual production but not too much so you don’t get a reasonable test.
Regression testing
There are some automated test tools that perform some of the mundane
testing - and they can prove very handy precisely in this situation. However, as
we’ve already discussed, there are limits to these tools, and so the burden is
still on the QA department to ensure that nothing has been broken.
Adherence to a couple of techniques can help to some extent:

• Modularizing your code so that there are distinct boundaries
between functions;
• using common routines and libraries as often as possible.
Doug Hennig uses a detailed change log that documents every change
made to an application in order to aid in hunting down the problems causes by
changes to code. While this doesn’t solve the problem up front, it makes a
great deal of difference in tracking the problem down and determining where
else the bug might manifest itself.
TEST PLANNING (A4 pag. 313)
Each step of the testing process must be planned (as each
other engineering activity):
• Establishing test objectives
• Designing test cases
• Writing test cases
• Testing test cases
• Executing tests
• Evaluating test results
Automated Testing (A4, page 315)
The testing team uses automated testing tools, as well, besides
manual testing and automated theorem provers and symbolic execution tools
(that include specialised equipment, and specific software):
• Code Analysis Tools (analyse the source code from the
consistency and correctness of the code and algorithms,
data, correct sequence etc.)
• Test Execution Tools
• Capture and Replay or capture and playback
(especially for User Interface Testing )
• Stubs (simplified modules, meant to replace the original
ones, used only in testing process), special drivers
• Complex Testing Environments
• Test Case Generators
Regarding diminishing returns
in testing
Testing is the most difficult of activities, because you're never done.
Unlike a programmer, who can, at some point, identify when they have
finished writing code that fulfills all of the requirements spelled out in the
functional specification, a tester can never be sure that they have found all
of the defects. As a result, the testing process becomes an exercise in
dealing with diminishing returns: If you find 100 defects during the first week
of testing, 50 defects during the second week, 10 during the third week, and
then 2 during each of the next three weeks, how many more weeks do you
want to continue to test?
This is the designers call, and the decision has to do with the quality
requirements of the application. Quality is a relative term, depending on
the use of the application. A usual desktop application probably doesn't
have to be tested very thoroughly. On the other hand, an application that
tracks retrofit schedules of heart monitoring equipment needs to be
considerably better, since errors have a potentially more catastrophic effect
(safety-critical systems – many real-time systems).
Fault “Seeding” (see A4 page 318)
Is quite impossible to measure the number of faults in a software.
There is a technique for a rough estimate of the number of faults still
unidentified, the Fault “seeding”
This technique involves the implantation (seeding) of a number of faults in the source code,
the testing being done with limited objectives (for example by the time limit or the number of
faults found) by the test team, and estimating the number of remaining faulyts in the total
faults detected, dividing them into those found from the implanted ones and those found from
the non-implanted ones, based on the simplifying hypothesis that non-seeded errors are in the
same ratio, whether seeded or not:
Thus, the estimated number of faults is determined by the rule of three :
Number of Detected Seeded Faults Number of Detected Non-Seeded Faults
____________________________ = _________________________________
Total Number of Seeded Faults Total Number of Non-Seeded Faults
The method is easy to apply (implantation, of course, must be done by someone outside of the
testing team) but involves an additional load on the workload of the test team, plus an estimate
based on empirical evidence (there is no guarantee that the result corresponds to truth - "we
just make an idea").
Example
One seeded 8 errors, other, after testing, identified 12
errors, 3 of them being seeded ones.
Then
Number of Detected NSF = 12 – 3 =9
3/8=9/TNrNSF
Thus Total Number of non-seeded Faults = 8*9/3=24

(there still are, besides the 9 already detected Faults, 15
other, non-discovered yet)
Confidence in the Software (see A4 page 320)
It is the probability that the software is error-free:

1 if n > N
C=
S / (S-N+1) if <= N
Where S – no. of seeded faults; n – no. of identified faults during testing;
N – no. of faults.
But this relation does not allow the calculus of the confidence until all seeded faults are
detected, thus another relation based on the previously presented estimation technique:
1 if n > N
S S+N+1
C= ______ / ( ______)
s - 1 N+s
Where s – nr of identified seeded faults during tests (even we don’t identify all)
26 / 36
Criteria for Ending Testing (A4, pag. 322)
BASED ON
•obtaining a desired Confidence in the software
•determination of the number of states, paths or branches
still pending for testing
•Determination of fault-prone code based on previous
experience of testing similar modules
Techniques to make testing easier
Two very frustrating situations that occur during the testing process:
 testing something that just worked a minute ago but doesn’t any longer
 having to restore the environment because the application blew up and took every special
setting and switch with it
Some techniques that can help out with these problems:

Original data
One of the most common reasons that a routine works at 11:46 but not at 11:47 is that the data that
was entered in the last test is causing problems (either the data was bad, it got corrupted, it was
incomplete, or it wiped out something else that shouldn’t have been wiped out)
Fundamental rules we’ve had for all of our application development :
 to create a clean set of original test data so that you have an accurate and measured baseline
from which to start. Each time you are ready to run a test, you can copy all the data from the original
test data directory into the current test data directory and run off that clean set of test data.
 If you are testing a long process, you may want to create several data sets that represent
different points in the process or for different parts of the system.
The cardinal rule for original test data: make it clean, robust and keep it in a location
that can be accessed easily so that the test data directory can be replenished with this
clean data on demand.
Techniques to make testing easier –
continued
Original environment
You also have to be able to restore the environment back to its original state.
For example, suppose your application always runs with the flag for
exact comparisons set OFF. However, during one specific point in a
routine, you need to have the flag set to ON, so at the beginning of the
routine, you do so. At the conclusion of the routine, you set the flag
back OFF. But, somewhere in the middle of that routine, the system
crashes. If you don’t reset the flag, and simply run the application again,
any comparison that was relying on the exact comparison to be off will
now fail - and possibly send you down a blind alley for hours until you
realize what happened.
Create a quick general-purpose tool that will automatically reset your
development environment back to where it came from so that you’re always
starting with the same baseline, just as you did with a clean set of test data.
Testing techniques
Track users
Even if you are creating a single user system, include a routine that automatically tracks the
user who is on the system. Often times, there will be more than one person using the system,
and providing them each with their separate login allows you to determine who was using the
system when problems occurred.
Track activities
Create a function at the start of every routine that logs the name of the user and the date and
time that they called the routine. The table that this information is placed in is relatively small
compared to the rest of the application, and you can clean it out at any time.
Knowing who was in a routine (and when) provides valuable information when trying to track
down particularly thorny or infrequent bugs.
Track errors
You should keep a very detailed error log that captures everything you consider valuable.
For instance, you should capture the error message, the line of code that causes the
error, the call stack of programs that led to the error, any parameters that were passed,
the name of the user and the time stamp, and a host of environment information, including
files open, libraries loaded, all memory variables, the current state of the environment,
and so on.
Some developers grab a screen shot at the time an error occurs. They take a snap
shot of the screen at a particular instance and store it to a file, and so can see what the user is
seeing at the time of the error. The only difficulty is that these screen shots are pretty big.
Testing techniques - continued
Audit trail
Track the user and the date/time that a record was last changed, and, if you have the space, log this same
information for when the record was added. Yes, this information takes another 30-40 bytes per record, but it is
invaluable to being able to save to do detective work on what is happening later on.
A more involved version of an audit trail feature tracks every change to any field. While this log can get awfully
large awfully quick, it can be useful to be able to turn this on and off on demand in order to isolate a specific
problem.
Error log access

You could place this on your Developer menu so that you can access it at a customer site if you've logged into the
app using your "super duper secret developer password."
You might not want to call this file an "Error Log" because it tends to unnecessarily worry some users. "What?
You’ve got errors? What kind of hacks are you???"
Instead, use a name like "Activity Log" and explain to the customer that this file is used to store any kind of activity
that you're not expecting. These could be errors, unexplained problems, data problems that shouldn’t have
happened - just anything out of the ordinary. They are free to send the activity log to us at any time. You should
follow the same technique as with the Activities Log and allow the user to delete it completely - the next time an
error occurs, the file is created if it didn’t already exist.
Debugging tools
If you haven’t already, set up a series of developer tools on a menu pad that is only available to the developer.
These tools should allow you to suspend program execution, or cancel the program outright so that you can halt
and investigate the environment at any time. Make sure all of your debugging facilities are available and ready to
be enabled whenever you need them.
Object-oriented testing (only as
lecture, if one needs, not for
Exam)
The many books on OOP (C++ and JAVA) tend not to deal with
testing/debugging.
This section will provide the beginnings of testing/debugging in OOA based on

behavior specification.
Features:
•This methodology should attempt to produce a test suite that is

thorough as time and money allow.
•A failed test should output the behavioral aspect it was aimed at,
thus providing a useful starting point for the debugging process
•The test suite should be able to evolve as the sofware system
evolves, making sure each update or release maintains the levels of
performance achieved inSE the pastStoicu-Tivadar
- Vasile 32 / 36
The look-and-see method
In this method:
•after writing the code fo a class, you create an instance of it and apply a method to it ->
the look part
•Then see if the code runs and that the results are what you expect them to be -> the see
part
•If not, you try to find out what’s wrong and correct it.
•After you’ve finished debugging this method, you apply a second method to the instance.
•You continue in this manner until you satisfied that the code is working as expected.
3 major problems:
•Thoroughness – in all this looking and seeing there is no guarantee that you are looking
in then right place for errors (other programmers or yourself will be seeing another
behavior later)
•Repeatability – there is no guarantee that tests that were passed earlier will still be
satisfied (because debugging)
•Self-disclosure – the test results should be in the form of a binary decision not requiring
the tester to stare at the output for a while before deciding whether or not it is correct and
which part of the behavior it was addressing
Testing rudiments
In contrast, a systematic approach would attempt to produce a test suite

that is thorough as possible, that could be reapplied as needed and
produce crisp pass/fail results.
Every class should have an associated test file. Tests should provide crisp/fail results
and document the underlying purpose.
Example:
cout << “Testing if name is properly set bu entity constructor “ << endl;
entity *e = new entity (“trial”);
If (strcmp(e->get_name(), “trial”)==0) {
cout <<“test satisfied”;}
else {
cout << “test not satisfied”;}
Such tests can be entered into a file whose name associates to the name of the class
under test (example: testentity.cpp)
Whenever entity.h or entity.cpp is modified, testentity.cpp is also modified, if
necessary, and recompiled and executed to make sure the new code has not been
inadvertently corrputed.
Testing rudiments – continued 1
Tools can be developed to minimize the routine drudgery.
Example: a class test can be defined that supports testing C++ classes (for a
container class)
//within above series, focus on empty()
main() {
Test * ConTest = new Test; //make a new instance of Test for this class
ConTest->Is_Equal(“con1->empty ?”, con1->empty(), TRUE);
// output “test passed” if container is empty
ConTest->ShowInfo(“Container Tests”,'’container.h container.C”,
“container"); // otherwise “test NOT passed”
// print out relevant Information ConTest->Focus(”container::size()');

entity book = new entity(“book'); ConTest->Is_Equal(”con1 size is 0 ?”, con1->size(),0);
ConTest->Print(1, 'Creating new entity called book…”);
Con1Test->Focus(”container::add(”entity”) & container::is_in”);
ConTest->Exists(book); //was book successfully created Con1->add(book);
ConTest->is_Equal(”book is added, is con1 holding book now ?”,
container *con1= new container: con1->is_in(book), TRUE);
ConTest->Print(1, “Creating new container con1 with no nane ..');
ConTest->Exists(con1); // more focal tests in this series
// additional series
ConTest->Series(“container::add() & container::is_ in()”);
ConTest->Report(); // report the test result statistiocs
//start a series of tests concerning add and is_in // how many series were done
ConTest->Focus(“container::empty()'); // how many tests were done, how many were in error
}
Testing rudiments – continued 2
Test supports documenting tests and their purposes, comparing query outputs
with expected values, and automatically tallying test results.
Example: some output of the test; we can see at a glance at the end of the
printout that there were tests that failed and then by scanning the file we can
easily identify which ones they were.
Test Focus: contalner::empty()
In the early stages, however, a
---------------------
con1 empty?: True full printout is most likely not to
ShowInfo:
Test passed. be had:
----------------------
Title : Container Tests •many crashes occur that
Test Focus: container::size()
Test on File : container.h container.C halt the execution
con1’s size is 0? : True
Test Class : container
-----------------------
Test passed. •The run-time system may
give little or no indication of
Creating new entity book …
Test Focus: container::add(“entity ”) the cause of the crash
& container::is_in(“entity ")
Object entity exists.
book is added, and is in con 1? : False
In such cases, the point at
Test NOT passed
Creating new container con1 with no name … which printing stopped
…..
Object container exists. provides useful information
Report:
----------------------- since the point of failure must
-----------------------
contalner::add() & contajner::is_in() tests
series Total Series : 7
have occurred after it. In
----------------------- Total Test : 25
contrast, look-and-see testing
Error Detected :5 may result in a crash and offer
----------------------- no clue as to what caused it.
Blueprint-based testing
Rudimentary test-supporting tools raise new questions:
•How do we determine what series of tests to run ?
•How do we come up with the purpose of each test and then implement it ?
•How many tests are enough ?
If both are based on the
Solution : the implementor supplies a class definition blueprint and are
the tester provides a test suite developed independently
and in parallel
Why ? All the behavior is Blueprint = abstract specification –

specified in the blueprint specifies exactly the bahavior expected
from the objects in the class
The behavior of an object is the set of all the query-terminated sequences of commands that
can be applied together with their responses.
A way for testing -> to list each query-terminated sequence of commands in a file and check
whether the actual response agrees with the value dictated by the behavior specification (like
in the example with the container).
BUT we don’t know about how to choose a sample (a finite set of pairs in the behavior
relation). There are two approaches: the first one focuses on the intended
functionality of the class -> SE - Vasile Stoicu-Tivadar 37 / 36
Constructing behavior samples
An object behavior specification (class alarm) :
constructor
alarm make-alarm(key)
queries open'(alarm) = F & key?(alarm) = key =>armed?(arm(alarm,key))=T

armed?(arm(alarrn,key)) = F
boolean armed?(alarm) //is the alarm set to work? // only applies when above condition fail
boolean open?(alarm) // is the door (or window) open ? key?(alarm) = key => armed?(disarm(alarm,key)) = F
boolean sound?(alarm) // is the alarm screeching? armed?(disarm(alarm, key))= armed?(alarrn)
armed?(open(alarm)))= F
hidden armed?(close(alarm)) = armed?(alarm)
key key?(alarm) // what is the required key ? open?(arm(alarm,key)) = open?(alarm)

open?(disarm(alarm,key)) = open?(alarm)
commands open?(open(alarm)) = T
open?(close(alarm)) = F
alarm’ arm(alarm,key) //set the alarm to work using the proper key
sound?(arm(alarm,key))) = sound?(alarm)
alarm' disarm(alarm,key) //turn off the alarm using the proper key
key = key?(alarm)=>sound?(disarm(alarrn,key)) = F
alarm’ open(alarm) // open the door
sound?(disarm(alarm, key)) = sound?(alarm)
alarm’ close(alarm) // close the door
// only applies when above condition fails
armed?(alarm)=T => sound?(open(alarm)) = T
Equivalences sound?(Open(alarm))= sound?(alarm)
// only applies when above condition fails
armed?(make-alarm(key)) =F sound?(close(alarm)) = sound?(alarm)
open?(make-alarm(key)) =F
sound?(make-alarm(key)) =F
key?(make-alarm(key)) = key
Constructing behavior samples – continued 1
Scenarios:
The main function of the alarm : to sound when it is armed and a break-in occurs.
Thus, if the alarm is armed with the right key and the door is subsequently opened, we want the alarm to sound.
=> The following query-terminated sequence and response
alarm = make-alarm(key1)
alarm' = arm(alarm,key1)
alarm" = open(alarm)
sound?(alarm")= T
Somebody knowing the right key should be able to get in by disarming the alarm:
alarm" = disarm(alarm,key1)
sound?(alarm") = F
But not knowing the proper key, a burglar will not be able to disarm the alarm and it will still warn of a break-in:
alarm" = disarm(alarm',key2)
alarm'" = open(alarm")
sound?(alarm'")=T
These query-terminated sequences represent normal user scenarios for the alarm, and its proper response is critical to its
existence. However, there are many other possible sequences that might arise in practice that must be tested.
Arming the alarm with the wrong key and subsequently opening the door; does the alarm sound?
alarm = make-alarm(keyl)
alarm" = open(alarm')
sound?(alarm") = ?
Arming the alarm with the right key,subsequently opening, then closing the door; can the alarm be armed?
alarm"' = close(alarm")
alarm iv= arm(alarm'",key1)
armed?(alarm iv ) = ?
Arming the alarm with the right key, subsequently opening the door, then forgetting to close the door: can the alarm be
armed?
alarm"' = arm(alarm",key1)
armed?(alarm''')= ?
Since the behavior specification is complete and consistent, each of the above responses can be worked out by
simulation. Clearly, we should test for them and as many other scenarios that might arise as possible.
But is there any way of knowing if all the scenarios have been considered ?
No, but our second approach can guarantee that all "aspects" of the behavior have been considered (query /
command pairs, or equivalently, state transitions).
Testing constructors for correct initialization
Before proceeding with the query/command pairs, let's note that the query/constructor pairs in the
object behavior specification provide the information for defining constructors and testing that they work
properly. For example, the query/constructor pairs for class alarm are:
armed?(make-alarm(key)) = F
open?(make-alarm(key)) = F
sound?(make-alarm(key)) = F
key?(make-alarm(key)) = key
They can be tested in a series of the form:
alarm * a = new alarm(“key1 ");

AlarmTest->lsEqual(“a armed?”, a->armed_q() FALSE);
AlarmTest->lsEqual(“a open? “, a->open_q(), FALSE);
AlarmTest->lsEqual(“a sound?”, a->sound_q(), FALSE);
AlarmTest->ls_Equal( “a key?”, a->key_q(), “key1");
A State transition –based testing
Objects are state machines (although no necessarily finite state machines). To fully test such a
system obviously requires that we test the transitions and outputs of the object in each of its states.
The state equations in an object behavior specification provide the information we need to do such
testing.
A complete specification contains all combinations of state representing queries and commands.
Example: the alarm class has three state-representing queries: armed?, open? and sounding?
Each returns a boolean value, so there are at most 23 = 8 states. Assuming the implementation
uses instance variables armed, open, and sounding, let's make eigh instances, each in a
different state:
alarm *alarm1= new alarm(/*armed*/F, /*open*/F, /*sounding*/F, /*key*/ “key1”);

alarm *alarm2= new alarm(/*armed*/T, /*open*/F, /*sounding*/F, /*key*/ “key1”);
alarm *alarm3= new alarm(/*armed*/F, /*open*/T, /*sounding*/F, /*key*/ “key1”);
alarm *alarm4= new alarm(/*armed*/T, /*open*/T, /*sounding*/F, /*key*/ “key1”);
alarm *alarm5= new alarm(/*armed*/F, /*open*/F, /*sounding*/T, /*key*/ “key1”);
alarm *alarm6= new alarm(/*armed*/T, /*open*/F, /*sounding*/T, /*key*/ “key1”);
alarm *alarm7= new alarm(/*armed*/F, /*open*/T, /*sounding*/T, /*key*/ “key1”);
alarm *alarm8= new alarm(/*armed*/T, /*open*/T, /*sounding*/T, /*key*/ “key1”);
A State transition –based testing –
continued
We have to write 16 query/command pairs (see the behavior specification). Example for one equation:
armed ?(open(alarm))=F
can be transformed to the method:
alarm::armed_q_on_open(){
open();
return armed_q()==F;
}
Each of them we apply to the 8 test instances:
test(alarm1->armed_q_on_open);
…
We have 16 query/command pairs for each of the 8 instances, so we will have 128 pass/fail tests.
A State transition –based test
methodology
A framework for supporting
this methodology:
The main steps of the methodology:
1. Write test methods for each
query/command equation in the Test
object behavior description,
purpose
2. Form a container of these test
precondition
methods,
postcondition
3. Form a container of instance
makers in different states, virtual make_instances
4. Apply each test method to each test_instances
instance (freshly reconstituted as
test_results
necessary).
print
5. Form a container of test results
execute
showing the pairs of tests and
instances they failed on. apply_test
test_of_X
test1
…
testn
methodology
Example
To test a class X, we define a subclass test_of_X:
Class test_of_alarm: public Test{};
The methods will implement the tests developed from the object behavior specification:
Bool test_of_X::query_on_command(X * testInstance){
............ //local constants to be used for arguments and tests
purpose = ............ ; //description of test objective
precondition = ........; // condition that must be satisfied by test instance
testInstance-> ........; // command sent to test instance
postcondition = ....... ; // condition that should be true after commands
}
For example, consider the equations

for query/command pair, armed?/arm.
test_of_alarm::armed_q_on_arm1(){
For the first rule:
char* key= "key1";
open?(alarm) = F & key?(alarm) = key purpose = "test if alarm can be armed under proper conditions";
precondition = !test-instance>open_q() & &
=> armed?(arm(alarm,key)) = T
test_instance->key_q() = key;
we write test_instance->arm(key);
postcondition = test-instance->armed_q();
)
methodology
Example – continued
The precondition corresponds 1 of the rule, while the postcondition
to the "if" part
represents the "then" part.
A rule in the behavior specification has the form precondition => postcondition
though in many rules the precondition is only implicitly there. For example, the second rule:
armed?(arm(alarm,key)) = F
only applies when the previous rule does not. Thus, its explicit form is:
not(open?(alarm) = F & key?(alarm) = key) =>armed?(arm(a?arm,key)) = F
Thus, this rule is translated as

Bool test_of_alarm::armed_q_on_arm2(){
char * key= "key1”;
Bool complement = test_instance->open_q() && test_instance->key_q() ==key;
purpose = " test if alarm rejects arming under right conditions”;
precondition = !complement;
test_instance->arm(key);
postcondition = !test-instance->armed_q();
} SE - Vasile Stoicu-Tivadar 47 / 36
methodology
Example – continued 2
Many rules are unconditional, which means their precondition is always true. For example,
armed?(open(a!arm)) = F
is translated as
Bool test_of_alarm::armed_q_on_open() {
purpose = 'test if alarm always is not armed after open”;
precondition = T;
test_instance->open();
postcondition = !test-instance->armed_q();
}
methodology
Example – continued
Step 3 in the methodology 3 "instance makers
calls for providing that can be called
to generate fresh copies of instances”:
class test_of_alarm:public Test{
public:
test_of_alarm():Test(){}
void make_instances() {
test_instances = new set;
alarm * alarm 1 = new alarm (/* armed */F, /* open */F, /* sounding */F, /* key*/"key” );
test_instances->add(alarm1);
…
alarm * alarm8 = new alarm (/* armed */T, /" open */T, /*sounding*/T, /* key*/”key1”);
test_instances->add(alarm8);
}
For each test developed as above, we write a corresponding method for class test_of_alarm that uses a
macro to apply the test to each of the test instances. For example, for the test
test_of_alarm::armed_q_on_arm1
we write the method:

void test_of_alarm::armed_q_on_arm1_all() {
apply_test(armed_q_on_arm1) ;
}
methodology
We then write an executable file that invokes each of the tests:
int main() {
test_of_alarm *t = new test_of_alarm();
t-> armed_q_on_arm1_all();
t-> armed_q_on_arm2_all();
t-> armed_q_on_open_all():
t-> armed_q_on_close_alI();
t->print();
}
The results appear in the form of an instance of test_result, which is an instance of class
function, holding for each failure-detecting method the container of failed instances.
For example, the print-out:
((armed_q_on_arm1(a2 a5)) (open_q_on_arm (a 1)))
indicates that armed_q_on_arm1 failed on instances a2 and a5 while open_q_on_arm failed

on a1. All other tests passed or were not applicable.
methodology
How this is done ?: see method execute which is automatically called after
applying a test method:
Bool Test: :execute(){

cout<< purpose;
if (!precondition) {
cout <<": test not applicable"<< endl;
return TRUE; }
else if (postcondition) {
cout <<": test satisfied" <<endl;
return TRUE; }
else {
cout<< ": test NOT satisfied“<< endl;
return FALSE;
}
methodology
Final considerations
In a relatively small finite state system such as alarms, it is feasible to completely cover the
state space with test instances. However, this may not be feasible for a large state space or if the state
space is infinite, as in the case of container classes.
The guiding principle must be to cover the tests rather than the states - use enough test
instances so that every' test is applicable at least once (within the limits of time and money,
the more test instances, the better)
Minimal test coverage : for each test, we make a test instance that will apply to that test.
But if we have the same test with different preconditions, we need to split this test in two different ones.
Example: a test that will apply to any instance with an open door but with different key settings (input
key is correct or incorrect).
Combining the approaches

In general, a test suite should contain:
1. Tests for constructors
2. Samples for query-terminated sequences that characterize normal and abnormal scenarios
3. Tests based on query/command pairs in the object behavior specifcation and test instances to cover
them.
What this chapter means for You
You can use the concepts from this chapter in the following ways:
as individual
- to learn about the tests, especially about acceptance testing
- to guide your practical actions in object-oriented testing
as team
- the project manager can use the knowlidge about testing for an
appropriate approach in the management of the project and in the
relationship whith the customer
- allows to co-ordinate the activity of the design team and of the
testing team, especially for object-oriented testing
Thank You for Your
attention !
SE - Vasile Stoicu-Tivadar

Chapter 5 - Testing

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Chapter 5 - Testing

Загружено:

Авторское право:

Доступные форматы

SE - Vasile Stoicu-Tivadar 1 / 36

Testing the Software

There are basically two sources of testing personnel:

Disadvantages of a full-time testing staff

The disadvantages of having the users do the final

System testing - does the app work in the production environment?

Regression testing - refers to the practice of testing an application after a

•Formal Demonstration Techniques (might be automated) - suppose the

To know if the test was performed correctly (actually acceptable), the

Diverse approaches of introducing successively the software

Adherence to a couple of techniques can help to some extent:

Thus Total Number of non-seeded Faults = 8*9/3=24

It is the probability that the software is error-free:

Some techniques that can help out with these problems:

Error log access

This section will provide the beginnings of testing/debugging in OOA based on

•This methodology should attempt to produce a test suite that is

In contrast, a systematic approach would attempt to produce a test suite

// print out relevant Information ConTest->Focus(”container::size()');

Why ? All the behavior is Blueprint = abstract specification –

queries open'(alarm) = F & key?(alarm) = key =>armed?(arm(alarm,key))=T

key key?(alarm) // what is the required key ? open?(arm(alarm,key)) = open?(alarm)

armed?(make-alarm(key)) =F sound?(close(alarm)) = sound?(alarm)

They can be tested in a series of the form:

alarm * a = new alarm(“key1 ");

alarm *alarm1= new alarm(/*armed*/F, /*open*/F, /*sounding*/F, /*key*/ “key1”);

Each of them we apply to the 8 test instances:

For example, consider the equations

Thus, this rule is translated as

we write the method:

((armed_q_on_arm1(a2 a5)) (open_q_on_arm (a 1)))

indicates that armed_q_on_arm1 failed on instances a2 and a5 while open_q_on_arm failed

Bool Test: :execute(){

Combining the approaches

Вам также может понравиться

alarm alarm1= new alarm(/armed/F, /open/F, /sounding/F, /key*/ “key1”);