Software Testing

UNIT-I: INTRODUCTION AND FUNDAMENTALS OF TESTING

1. INTRODUCTION:
1.1 Power of Software:
It is not wrong if we say that, the world will come to a standstill if all the computers on the
earth stop working. Nowadays, computers are used in everywhere -- education,
entertainment, transportation, banking, healthcare, and of course in all business activities.
And, in all these applications, the software is very complex. Millions of lines of code written
by hundreds of programmers are integrated into one package. If there is a single mistake
committed by one programmer, it may lead a problem. Therefore it is the software that has
now become the lifeline of systems.
As automation has brought in revolutionary changes in the quality of life, the demand for
more software-driven services and products is increasing. The ubiquitous mobile phone is an
example. Two decades ago, the mobile phone was only for the elite of the society. Today, it
is a necessary gadget in everyone's hand. Today, mobile phone is a complex technology
gadget with nearly two millions of code residing in it. As the users demand more and more
services through their mobile phones, it is expected that the software will be about 10 million
lines of code in the next few years. This indicates how complexity of software is increasing
day by day, as users demand more and more facilities in their existing services.

1.2 Challenges in Software Projects:

The complexity of software projects is increasing when users demand more facilities for
their existing services. For instance, bank account holders no longer want to go to a bank
during the working hours to do any transactions-they need ATMs, Internet banking, mobile
banking, etc making the bank software more and more complex.

As crime continues to increase, software now needs to provide good security features. This
is true for mobile phone software, banking software, e-commerce portals, etc. And to provide
foolproof security is a very challenging task, as evident from the increase in virus attacks,
credit card information being stolen from the Internet and the cyber crime.

Developing mission-critical software for a number of applications, such as bio-medical

equipment, nuclear plants, transportation control systems, which demand zero-defect
software, is highly challenging.

As software is gaining more importance, the demand for software professionals is

increasing, but there a shortage of quality professionals. As a result employees jump from
one organization to another, that too in the middle of the project leading to many projects
getting into trouble halfway. Manpower attrition is one of the main challenges to be tackled
in software organizations.

As the demand for new software products and services increases, there is an immense
pressure on the development organizations to deliver fast. The time pressure is so much that
the organizations and people are unable to cope up with it. Unfortunately, in many cases, this
is leading to the development of software full of bug.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 1

 Software Testing

1.3 Software Fiascos:

Software development is considered more of an art and less of science or engineering. To
bring software development into the area of engineering, a lot of research has been carried
out during the last four decades. In spite of the research in finding out ways and means of
developing quality software, the history shows that a large number of projects are executed
very badly. We keep reading in the newspapers about the mobile phones that have been
withdraw from the market because of a software problem, about the credit card fraud, and so
on and all these incidents are software fiascos. There are many stories in the history of
software development that teach us very important lessons.

1.3.1 CT Scanner
Getting a CT scan (Figure) done in a hospital is very simple and painless process. Really?
Think again!

In a Canadian hospital, suddenly a patient undergoing the scanning died. The machine was
rechecked thoroughly and then again the scanning started. After scanning of a few patients
was completed, another patient died. Randomly, patients were dieing and after the death of
more than 20 patients, the defect was identified. A glitch in the hardware was resetting the
software. During the short time of that resetting, the patient was exposed to large amounts of
radiation because of lack of control on the radiation. As a result, the patient used to die. Next
time, when you get a CT scan done, pray to God that the software should have been
developed and tested by those engineers who read this book, the people with quality
consciousness.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 2

 Software Testing
1.4 Reasons for Software Failure:
The Reasons for software failure can be attributed to the following categories.

i. People: Ultimately, every problem is attributed to the people. Lack of quality

awareness, lack of proper communication, lack of inter personal skills, personal
problems and lack of professional ethics are the main reasons.
ii. Technology: More interest towards introducing new technologies leads to lot of
problems while Implementation.
iii. Time: Due to market pressures, organizations try to reduce the development time
so much that the targets become unrealistic, leading to overworked engineers,
which results in low quality software.
iv. Money: At the time of initially estimating the project cost, we commit mistakes,
particularly when we are working on new technologies. It is simply, because at
the beginning we do not know enough the details to estimate correctly. Once, a
budget is given, the project must be completed within that stipulated budgets else,
it is declared as a failure.

1.5 What is the Solution?

Statistics reveal that nearly 74% of the software projects could not be completed within the
time frame or within the budget or both. Nearly 20% of the projects are abandoned halfway.
The problem can be due to the following.
i. Wrong estimation before starting the project,
ii. The project could not be managed and completed within the estimated time and
budget.

The software engineers and project managers have two choices --- to live with such a painful
process or to develop a process whereby the software development will be more of fun and
challenge. To make software development a fun-filled challenge; a creative activity is needed
that contributes to better quality of life. There is only one solution for this, a systematic and
disciplined approach to software development.

Managing software projects, in a highly competitive environment with fast changing

technologies and shortage of skilled workforce, has become the biggest challenge for all
software organizations. The project team has to make every project a success. But then, what
are the secrets of success?

 Every employee in the organization should be committed to software quality

assurance.
 The product must be tested thoroughly.
 Every individual must follow a code of ethics and professional practice.

NOTE: To make software projects successful, the three important mantras are

a) software quality assurance

b) Software testing
c) Code of ethics and professional practice.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 3

 Software Testing
1.5.1 Software Quality Assurance
If we have to deliver quality software, as per the customer requirements, we have to aware of
quality throughout-from the beginning of the project to its completion. We divide the
software development activity into a number of phases such as obtaining user requirements,
design, coding, testing and maintenance. We can deliver a quality product to the customer
only when we are aware of quality in every phase of development. And this quality
awareness must be there in each and every person associated with the project -- from the
CEO to the programmers. Even one or a few persons take it easy, quality will be
compromised.

Software Quality Assurance (SQA) addresses the various mechanisms to be implemented to

prevent defects in the software. SQA encompasses all phases of development, it is applicable
to all types of projects and it addresses all the people associated with the project including the
management. Generally, in software development organizations, there will be a separate
SQA department. The persons working in this department will check whether the
development teams are following the quality norms defined in the organization. In small
organizations, there may not be a separate department, but at least one or few people are
assigned the SQA responsibilities.

1.5.2 Software Testing

Even though we ensure quality at every stage of development, we need to give importance to
the testing phase. In spite of the best efforts of SQA engineers and development engineers, it
is likely that there may be few defects left in the software. In the testing phase, we need to
thoroughly test the software and find out as many defects as possible, so that, we can
confidently deliver a good quality product to the customer.

Because testing is the most important phase in development, organizations are setting up
separate testing groups whose job is only to test the software developed by different groups.
The test engineers specialize in testing. The testing engineers need to have special skills and
they need to learn many tricks to do testing.

Every organization is now giving a lot of thrust to the testing team. How many people are
involved in testing, of course, differs depending on the type of project. In Microsoft
Corporation the ratio of development engineers to test engineers is 1:1. In NASA Goddard
Space Flight Center, the ratio is 1:7, ie. For every development engineer, there will be seven
test engineers. For mission-critical applications involving safety and security of people (eg,
space, medical, nuclear, transportation etc.), the testing teams should be pretty big.

1.5.3 Code of Ethics

Some software engineers, out of ignorance, do many illegal things. For example, you may
come across an engineer who sends the code written by him to a friend working in another
organization. This person is not aware that the code written by an employee is the intellectual
property of the company and it cannot be shared with others. This ignorance may cost him
his job.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 4

 Software Testing

Some software engineers, out of fear towards their employer, do unethical things. Suppose,
you are testing software for medical equipment that will be used in intensive care unit of a
hospital. The software still has bugs, but the manager asks you to declare that the software is
OK. What will you do?

If you say the software is OK, you may kill a patient, and if you say the software is not OK,
you may lose your job. In such situations the code of ethics provides you the guidelines
based on which you can take a decision based on what you think is morally correct.

Every software engineer has to realize that he/she significantly contributes for betterment of
society and for improving the quality of life. So, he/she has a responsibility towards society
and the software developed by him/her should be of high quality otherwise, it may result in
loss of money or loss of time or even loss of lives. Keeping in view the important role played
by software engineers as professionals, ACM and IEEE developed a code of ethics and
professional practice. Every software engineers must follow this code. This code helps you in
understanding what is ethical and what is unethical in your profession.

1.6 Software Testing Professionals

Software testing professionals are now becoming attractive people. Because they do a
professional job in a specialized area, they are in great demand. They command lot of respect
(even by the development engineers) and make a lot of money However, to achieve this
status; you need to really work hard. As shown in Figure 1.4, the test engineers have to
interact with many other groups to carry out their work so, the fundamental; requirement of a
test engineer is to have good inter-personal skills.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 5

 Software Testing
1.6.1 Skill Sets for Testing Professionals
The various skills you need to acquire to become software professional are:
 A good understanding of software quality assurance and quality management
standards such as Capability Maturity Model Integration (CMMI).
 A thorough knowledge of testing process, testing types and methods, designing and
executing tests.
 Hands-on expertise in using popular testing tools.
 Knowledge in an application area (such as financial accounting, business
applications, telecommunications, networking, security, etc
 Excellent communication and inter-personal skills with ability to effectively inform
bad news (that the software has bugs).
 Good documentation skills to document test results and test reports.
 Good listening skills, to listen to the members of other groups.
 The art of learning fast new techniques and technologies.
 A good understanding of the "code of ethics and professional practice.

1.6.2 Tasks Handled by Testing Professionals

Many young software engineers think that testing is not a creative job. But testing is a very
creative and challenging Job. The test engineers need to understand the application well,
study the software functionality in detail to find out where the defects are likely to occur,
study the code to ensure that each and every line of code is tested, create test cases in such a
way that testing is done carefully to uncover the hidden defects and also ensure that the
software is usable and reliable. All this has to be done very fast.

To take up testing of a software product, the test engineers need to do the following:

Understand the application domain: The software may be a Client/Server based, database
management system, or a telecom software, or data communication software, or embedded
software, or process control software, a web site, etc. As a test engineer, you may have to test
software of various application domains. Before starting the testing, you need to understand
the application from user point of view.

Learn about the development environment: Based on the application and the end user
needs, the development environment varies -- the operating system may be windows, UNIX,
Linux, or-Solaris; the programming language may be C, C++ or Java; some special tools
would have been used to develop the software such as database engines, cross-compilers, etc.
You need to have to work on different development environments.

Learn how to use the application: The test engineer has to assume that he/she is the end
user and run the application; to get a feel of how the software works. Only then, he will be in
a position to generate the test cases and also create automated test procedures

Study the source code: If the test engineer is asked to validate the source code also he needs
to study the source code based on which the test case generation has to be done.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 6

 Software Testing
Study the requirements specifications document: The requirements document gives the
functional, performance and reliability requirements of the software. The software has to be
tested for all these requirements. You need to see for detail while studying the specifications
document and check whether each requirement can be tested or not.
Study the acceptance test procedure: For every project, an acceptance procedure document
has to be developed. This document gives the test plan based on which the customer will
validate the software. The test engineers need to carry out the testing as per this document.
This document has to be prepared by the development team and the testing team along with
the customer representative.
Test innovatively: Test the software in minimum possible time to uncover maximum
possible bugs. This can be done by designing test cases innovatively and using testing tools
efficiently.
Report the test results: Report the test results to the development team. This has to be done
very carefully, because no developer would like to hear that there are bugs in his/her
software.

Contribute to the organization’s growth: The bottom line is to help your organization to
grow (Your contribution as a test engineer to the quality of the software would help in
customer satisfaction, which in turn leads to more business for the organization.

Be proud of your work: Many test engineers feel that they are doing only testing and not
coding. Testing is as respectful a job as coding. In fact, organizations are giving equal
importance and value to the test engineers as development engineers; only when you feel
proud that you are involved in an important work, you can do a good job.

2. FUNDAMENTALS OF TESTING:
2.1 What is testing?
Testing is a process to check whether the software meets the user requirements or not. During
software development, the developers make many mistakes at different stages and testing
helps in correcting those mistakes.

So, in a broad sense, testing encompasses all phases of development-in every phase, the work
products of that phase are tested. Hence, there is a testing activity corresponding to every
phase of development. For example, in the requirements engineering stage, the SRS
document is written and tested to check whether it captures all the user requirements or not.
Testing is generally considered as a separate phase after the implementation/ construction
phase. Hence, once the code is converted into a working system, the testing phase starts.

The objective of testing is to find the defects. When the testing is done, the reasons for
failures are identified and the defects are found. Then the defects are corrected. This is an
iterative process-you need to test the software, find defects, correct the code, and test the
software again.

When you do the testing your objective is to find defects, i.e. you look at the software with
negative feelings and then do your job. This is called negative testing.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 7

 Software Testing
2.1.1 Testing versus Debugging
 Testing is different from debugging, Testing is done to find the defects, whereas
debugging is done to find the cause of the defect.
 Debugging is the job of a developer. On the other hand, the testing is the job of test
engineers.
 Testing involves checking the software for defects and generates the defect report.
Whereas Debugging involves finding out which portion of the code is causing the
defect and removing the defect.

2.1.2 Testing and Bebugging

Testing is a process of finding the defect in the software. Whereas Bebugging is a process of
seeding the defects intentionally.

To check the effectiveness of the testing process, bugs are sometimes introduced
intentionally in the software. This is known as defect seeding or bebugging.

Suppose you have tested a software package and declared that it is OK. Now, the manager
can intentionally introduce 10 defects in the code and asked to test it again. If you are able to
find only 9 defects, it indicates that you are not able to find 10% of the defects that are
introduced. This shows that you are not able to find 10% of the total defects from the original
software. This technique of intentionally introducing defects is very useful to find out when
the testing is complete.

2.1.3 Verification and Validation

Validation: Are we building the right product?
Verification: Are we building the product right?
Validation is to check whether the software meets the customer expectations. This is done by
testing the software for its functionality and other requirements as given in the requirements
document. Generally, validation is carried out with the involvement of the customer.

Verification is to check whether the software conforms to specifications. Verification is done

by the development team at various phases of development. The SRS document, the design
document, and the code are reviewed to ensure that the product is being developed using a
process-oriented approach. Verification is an in-house activity of the development
organization.

2.1.4 Root Cause Analysis

While doing testing and debugging, it is important to identify the bug and remove it. More
important is to find out why the bug has occurred, i.e. to find out the root of the cause of the
problem. This is called root cause analysis.

Consider the following example; where a program is written for controlling the temperature
of an air conditioner. In the code, a variable temp' is assigned a value of 25 using the
statement.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 8

 Software Testing
int temp = 25;

When you carry out the testing, you realize that you cannot reduce the temperature of the air
conditioner below 25. You find that the error is due to the above line of code; when you
change the value to 16, the program works well. Certainly, you have debugged well. But
what is the root cause? Is it the mistake that you made while coding? Or, is it a problem in
the specifications? You can refer to the SRS document and check whether any value is
mentioned for the minimum temperature that can be set. If it is not mentioned, or if it
mentioned erroneously as 25, then the root cause is due to poor specifications and you need
to improve the specifications process.

In this case, the root cause is poor specification and the effect is writing a wrong code.

2.2 Significance of Testing

As human beings we all make mistakes and these mistakes are reflected as defects in the
software product. These defects make the software fail which are due to our limitations as
human beings-ignorance, arrogance, lack of communication, politics within the team,
jealousy, and too much greed to save more money and so on. In spite of all our mistakes, we
still need to deliver quality software. Hence, we need to test the software. Here are some
reasons why testing the software thoroughly is a must:

 As a result of communication gap between users and requirements analysts, the

software would not have been developed as per the user requirements.
 Due to the communication gap amongst developers, the complete functionality
required would not have been implemented.
 Individual programmers' inability to do efficient coding would have resulted
defective code.
 Software had been tested on a simulated environment as the actual environment is not
readily available at the developer's premises. Testing is necessary in the actual user
environment.
 Software that goes into the embedded systems is generally developed on a PC or at
workstation. It must be tested on the actual hardware before it is delivered to the
customer.
 Before the software is delivered to the customer, it must be ensured that privacy,
safety and security of the users and user data are taken care by a rigorous testing.
 Due to constraints of time imposed by the client, the software was developed in a
hurry, and hence quality would have been compromised.
 Constraints of budget imposed by the client and then the management would have
resulted in compromising on quality.

2.2.1 Cost of Quality

The ultimate objective is to develop quality software. But quality does not come free --- you
have to pay a price for it. You have to put in the required effort to produce quality product.
This effort is the 'cost of quality.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 9

 Software Testing
This cost has three components:
 Failure cost
 Appraisal cost
 Prevention cost
Failure cost is the cost of fixing the product defects.
Appraisal cost is the cost of assessing the product to check whether it has defects or not (the
cost of removing the defects is not included). We can look at this cost as the insurance paid
to assure that the product is defect-free.

Prevention cost is the cost of modifying the process to avoid defects in the software For
example; you may like to develop a prototype to reduce the defects due to improper
specifications.

2.3 Psychology of Testing

Consider the following definition for testing: Testing is the process to prove that the software
works correctly.

If the person who has developed the software carries out the testing, he/she will try to use this
definition. This is the typical psychology of testing. The person who developed the software
will only try to show that the software works correctly. So, he will give only those inputs for
which correct results are obtained, or press only those keys by which the software does not
crash.

However, in a commercial environment, using this definition for testing is dangerous, you
cannot deliver software, full of bugs, to the customer. But, developers always try to follow
this definition of testing.

NOTE: From a developer's point of view, the definition of testing is: "testing is the process
to prove that the software works correctly".

Now, consider the following definition: Testing is the process to prove that the software does
not work.

When software is given to the QA department, they follow this definition. Here the aim of
the test engineer is to prove that the software does not work. If persons other than developers
test the software, then they follow this definition. This type of psychology towards testing
would bring out most of the defects.

NOTE: From a quality assurance/test engineer's point of view, the definition of testing is:
Testing is the process to prove that the software does not work.
Finally, a realistic definition of testing is: Testing is the process to detect the defects and
minimize the risk associated with the residual defects.

NOTE: From a manager's point of view, the definition of testing is: "testing is the process to
detect the defects and minimize the risk associated with the residual defects.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 10

 Software Testing
2.4 Testing Choices:
Many IT user organizations are giving thrust to this third party testing. Even software
development organizations are getting the software tested by independent agencies to ensure
quality products.
The managers of IT user organizations as well as software development organizations have
two choices: to do the testing in-house or to outsource the testing activity.

2.4.1 In-House Testing: In-house testing is certainly a good choice, if the necessary
infrastructure and manpower are available. The advantages are:

 The intellectual property will not be given out to a third party.

 The management will have total control over time and budget.
 The in-house testing team has access to the developers and hence, the test
engineers can interact with the developers on need basis.

On the down side, in-house testing has some difficulties too:

 Maintaining the infrastructure is very costly.

 The necessary expertise may not be always available (for example, testing the
security aspects of banking software.
 Getting the manpower, training them and keeping them for the duration of the
project is a challenge.

2.4.2 Outsourcing:
As testing has become very specialized, now a number of organizations are outsourcing
their testing activity. Thus have many advantages:
 Software testing needs lot of infrastructure- high end servers, network with high
bandwidth, testing tools, and specialized test instruments. The prime contractor
need not maintain this infrastructure.
 The subcontractor will have the necessary human resources readily available--
professionals trained in specialized areas, such as security testing, conformance
testing to check whether the software meets international/national standards, etc.
The prime contractor need not bother about recruiting and training the manpower.
 As testing professionals are experts in their area, their productivity will be very
high and hence testing is done faster and better at the subcontractor's premises.
 The problems associated with psychology of testing are eliminated as the test
engineers belong to another organization.
 If the testing activity is outsourced to an organization in a developing country,
then it results in lot of cost saving.
On the downside, the management will encounter a number of challenges:
 The intellectual property (such as: the SRS document, the design, even the source
code) may be transferred to a third party. Ensuring that the third party will honor
the intellectual property rights is a big issue to be tackled.
 The time frame is not under strict control of the prime contractor.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 11

 Software Testing

 Evaluating the vendor, particularly for the first time is very difficult. Many prime
contractors feel that outsourcing creates problems-yes, when the choice of the
vendor is incorrect, and then outsourcing will result in a bad experience.
 If the outsourcing organization is located in a different country, the laws related to
intellectual property, copyrights and cyber-laws may be different.
 Keeping in view, the pros and cons of in-house testing and outsourcing, the
management has to decide very carefully where the testing has to be carried out.

2.5 Who does the testing?

2.5.1 Developers as Testers: If developers carry out the testing, there are some
advantages:
 The developers are familiar with the user requirements from the beginning of the
coding activity. Hence they are in a better position to generate good test-cases.
 The developers are familiar with the structure of the program---both code as well as
the structure of modules, sub-systems, etc. So, they can do the testing better.

The developers have to carry out the following types of testing:

 Unit/component testing: Unit or component is the smallest piece of software that can
be tested independently. Only the developer can do the testing of the unit/component
because he/she knows the details of the unit/component.
 Integration testing: While carrying out the integration of the different modules, the
interface details between modules need to be studied. This again involves the code-
level details. As test engineers are not familiar with the code, they cannot handle this
activity. Hence, integration testing has to be done by the developers.

2.5.2 Independent Testing Team: If an independent testing team does the testing, they
can do testing completely and thoroughly. However, as test engineers are not the developers,
there are a few issues to be tackled:
 The test engineers may not be in the project since the beginning of the project and
hence, they may not be familiar with the user requirements. So, the test engineers
need to have an excellent knowledge of the application domain and also study all the
work products such as SRS document, design document, etc.
 If the test engineers have to do the code reviews, then they need to become familiar
with the programming language and the source code.
 The test engineers need to constantly interact with the developers to come out with a
good testing strategy.
 The test engineers need to realize that the developers are emotionally attached to their
work (the code) and hence the test engineers need to be very careful when pointing
out software bugs. You need to tell them politely!

Test engineers can do the following testing with minimal interaction with the developers:
 System testing: System testing involves testing the software against the
specifications. If the SRS document is available, the test engineers can study it and
then the system testing can be done.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 12

 Software Testing
 Acceptance testing: Acceptance testing is done by the client in association with the
development organization. Even before the software is offered for testing to the
customer, the testing team can carry out the testing and ensure that the software
offered to the customer for testing is of high quality.

2.5.3 Buddy Testing:

When two people sit together and do the testing, it is called buddy testing. If a developer and
a tester together work at unit level testing, and the test code is written while the actual code is
being developed, then it leads to better software. However, it is difficult to implement this
method because the tester also gets familiar with the code, and the resource (manpower)
requirement is rather high.

2.6 Testing Phases:

If you start testing only after the software is developed completely, it is called reactive
testing. On the other hand, if you start designing the tests in the early phases of development
it is called preventive testing. And, obviously, prevention is better than cure.

Testing needs to be done on every work product that is produce during the development.
Work products, such as SRS document and design document, are tested through reviews and
inspections. Testing the software is done through dynamic testing by executing the code.
During the course of construction/implementation, the testing is done at different levels such
as unit level, module level, sub-system level and system level.

The V-Model depicts the testing to be done at various phases of development.

2.6.1 V Model:
To bring out a quality product, we need to test the output of every phase of software
development. This is fundamental to produce quality software. This is reflected in the V
Model. This model shows the various development phases and the corresponding testing
phases.

Business Testing in actual

environment
Needs

User Acceptance
Requirement Testing

System
SRS Testing

Integration
Design
Testing

Component/Unit
Testing
B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 13
 Software Testing

In the software development process, the end user needs are converted into user
requirements. From these user requirements, software requirements specifications document
is prepared. Based on the SRS document, the design is carried out and the design document is
made. Based on the design, the various modules are designed and each module is further
divided into units/components. These activities are shown in the left side of the V Model.

Each unit/component is coded and tested. The components are integrated into modules and
these modules are tested. Module testing is done to check whether the design is correct. After
the modules are integrated, system testing is done. System testing is to ensure that the
software is as per the SRS document. Subsequently, the software is tested by the customer
and this testing is called acceptance testing. Acceptance testing is done to check whether the
software meets the user requirements. After that, the software is installed in the operational
environment and tested. This is called operational testing. This testing is done to check
whether the software meets all the needs of the user. The various levels of testing are shown
on the right side of the V Model.

2.6.2 Testing and Life Cycle Models:

For commercial projects, the waterfall model is used. The projects in which the user
requirements are not very clear, prototyping model is used. For research projects, the
Evolutionary development model in used for projects involving lot of risk, Spiral model is
used. In all these models, testing is certainly an important phase. But when to test differs
slightly.

 In Waterfall model, testing phase starts after the module integration is done and hence
testing is done on a reasonably mature product. The testing is done using the SRS
document as the reference.
 In Prototyping model, the prototype is tested based on the preliminary user
requirements obtained. Hence, at this level, testing is very simple. However, after the
user feedback on the prototype is obtained and the SRS document is finalized, the
software is developed using the Waterfall model and testing is done on a mature
product.
 In Evolutionary development model, the software is developed incrementally and
hence testing is also done incrementally. This involves lot of testing effort because
every time the system is incremented, the testing is done differently.
 Projects that use Spiral model are high-risk projects. Hence, in every spiral while
studying the risk factors, the testability of the software has to be given lot of thrust.

2.7 Testing the System:

Once the implementation stage is over, the software has to be tested. During the development
itself, the unit/component testing and integration testing are done. Then, the system has to be
tested. The system has to be tested with SRS document as the reference.

The SRS document contains mainly two sections: One section contains the functional
requirements and the other section contains the non functional requirements.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 14

 Software Testing

Functional requirements specify what the software does. This functionality obviously
differs from application to application. For most of the application software packages
functional testing involves testing the Graphical User Interface (GUI). The results obtained
through the software have to be compared with the existing manual system. If the results
match, then we can confidently say that, the software is indeed meeting the functional
requirement.

Non-functional requirements are generally divided into three categories:

 Product requirements
 Process requirements
 External requirement

Figure 3.2 shows the various requirements under these three categories.

If portability is a requirement, the various platforms on which the software has to work also
will be specified (for example, Windows XP and Red Hat Linux operating systems running
on Pentium 4 processor). The test engineers need to test the software on both these operating
systems.

The test engineers need to study the various non-functional requirements that need to be
tested and write test cases for each of these requirements as well. Note that testing each non-
functional requirement is a very involved task.

2.8 Testing Strategies:

There are two types of testing:
1. Static testing
2. Dynamic testing

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 15

 Software Testing
2.8.1 Static Testing:
In static testing, the software is tested without executing it. This is generally done by manual
code reviews and code inspections. The programmer, along with other group members, goes
through the code and checks for any syntax errors and logical errors. The code is checked to
ensure that it implements all the requirements given in the SRS document. It is also checked
whether the code has enough comments so that other engineers can easily understand the
code, for easy maintenance. These code reviews can be done informally or formally. This is
called static testing or static analysis.

Static testing is generally done manually. Tools are also available for static analysis. For
example, whether the code written in С programming language is portable or not can be
easily tested using tool “lint” available on the UNIX systems.

2.8.2 Dynamic Testing:

In dynamic testing, the code is executed and tested. The code is compiled, executed with test
inputs, and if the actual results match with the expected results, then the software can be
considered as OK.

Consider a program to find the square root of a given number. You first write the code and
then execute it. You need to give some numbers as input and check whether you are getting
the correct result or not. The inputs can be:
12, 34, 16, 32, -23, -46 . . . .
For positive numbers, it should give the correct value (4 for 16. It should give 0 if the input is
0 and if the input is a negative number, it should display a message: "Please give only
positive numbers".
Now, can we say that the software is perfectly OK? Suppose, you give a number 45678990.
What will be the output? If your program takes integers as input, and if the system allocates
only 2 bytes to an integer, then the above value is not acceptable as it results in an overflow
and you will get a run time error.

So, there are lots of issues that need to be resolved in dynamic testing you need to check the
software for valid inputs and invalid inputs, and also ensure that the software gives
appropriate error messages.

2.8.3 Why Testing is Difficult?

In dynamic testing, the code is executed and the test inputs are given then the actual results
are compared with expected results. How many types of test inputs have to be given is the
major issue. Ideally, we need to give all possible test inputs.

Testing is difficult because:

 We need to test the software for both right and wrong inputs and cheek the
functionality as well as the performance parameters.
 We need to give the inputs in such a way that each and every line of the code is
tested.
 We need to give inputs randomly and check that the software never fails.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 16

 Software Testing
 We need to test the software as though a normal user is using it and check whether
the necessary error messages, help, etc. are provided.
 We need to test the software in different hardware and software environments
depending on the requirement.
 We need to test the functional and non-functional requirements.

Exhaustive testing:
Let us consider another example of testing a GUI of a standard calculator (available on your
Windows systems (Start-Programs-Accessories- Calculator). The GUI is shown in figure 3.3.
In this GUI, the meaning of various symbols is as follows:
 +: To perform addition.
 - : To perform subtraction.
 *: To perform Multiplication.
 / : To perform division
 . : Decimal point
 Sqrt : To find square root of a number.
 % : To find percent.
 1/x : To find inverse of a number
 MC : To clear the memory
 MR : To recall for memory
 MS : To save in memory
 M+ : To add the memory
 C : To clear the current calculation
 CE: To clear the displayed number.
 Backspaces: To remove left most digit.
 +/- : To give sign to a number (positive or negative)

Your job is now to test whether the calculator is working correctly or not.

You have to test the addition ( + ) by adding integer, floating point numbers—Small
numbers, large numbers—and check whether the results are OK or not. You need to repeat
this process or all other arithmetic operations. Then you need to check whether correct
percentages are being obtained. You need to test whether the inverse number functionality is
correct. You need to check all memory operations. You need to do all this for positive
numbers, negative numbers, 0, very small numbers, very large numbers and so on.
For one small GUI like this, if you have to give all possible combinations of inputs and test,
then you need infinite time and energy to do the testing.

2.8.4: Test Case:

When you want to test particular software, you have to give certain inputs to the software
during execution. For these test inputs, you expect some output (the expected result). But
actually, the software will produce some output (the actual test result). Note that the expected
result and the actual test result will be same if the software is working correctly. Otherwise,
they will differ.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 17

 Software Testing
Just before giving the test inputs, the software will be in a particular state. This initial state is
described by entry criteria.
After the testing, the software will go to another state. This state is described by exit criteria.
As shown in Figure 3.4,

A test case consists of:

 Entry criteria (the pre-conditions before the test inputs are given)
 The test inputs
 The exit criteria (the post-conditions after the test inputs are executed)
 The expected results

Entry Criteria Exit Criteria

Executable

Code
Test Input Test Result

Suppose you need to test a program that takes a filename as input, opens the file, writes
today’s date into the file and close the file. You can define a test case as follows:

 Test input: filename “abc.dat”

 Entry criteria: The file abc.dat should not exist in the current directory.
 Expected output: The file abc.dat should be in the current directory, with the content
in today’s date.
 Exit criteria: The program should terminate without errors. The file abc.dat should
be available in the current directory.

Pesticide Paradox
In 1990, Boris Beizer, used the phrase pesticide paradox to describe an interesting
phenomenon in testing. When you use the same set of test cases to test software, you cannot
uncover the defects. This is similar to using the same pesticide again and again the insects set
immune to it! Similarly, the software gets immune to your test cases. Hence, it is not a good
strategy to repeatedly use the same set of test cases. You need to keep on changing the test
cases.

Defect Clustering
A software package consisting of a number of modules, it is likely that defects cluster in a
module or function. So, when you encounter defects in a module, keep testing that module
more rigorously by choosing a large number of test cases that can test that module.
This can happen due to various reasons-the developers who worked on that module would
not have got the necessary expertise, the module would have been more complex, etc.
Whatever be the reason, if you find a good number of defects in a module, thorough testing
of the module is a must. More test cases need to be selected to test the functionality of the
module.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 18

 Software Testing
2.8.5 Test Oracle:
Test oracles are the people (or machines) that check the correctness of the program for a
given test cases as shown in Figure 3.6. A common way is to give the program and the input
test data to a friend and ask him to derive the output from the program. Your friend acts like
a computer, goes through the code step by step as per the input data, and arrives at the output.
If this output is same as the expected, then the program can be said to be correct, otherwise
there is a defect in the software.

Software
under Test Comparator
Test Cases

Results of
Test Oracle Testing

2.8.6 Test Software:

Test engineers need to appreciate that the environment in which the software is developed
may be different from the environment in which the software will be actually used by the
customer. In such cases, you need to simulate the actual working environment in your
laboratory. This can be done by writing test software called simulator. Here are some
examples:

 Suppose you need to test the software that has to work on a satellite-based computer
network. Since you will not have access to a satellite-based network, you can
simulate the environment on a LAN. You need to write some additional software to
simulate the long delays in a satellite network. If the software works in the simulated
environment, you get some confidence on the software. Finally, you need to conduct
a field trial by installing the software on the satellite-based computer network and
testing it.
 Suppose you have to test embedded software that has to be ported on an Washing
machine hardware module. If the module is not available to you, you need to test the
software on a PC using the cross platform tools and emulators.
 Suppose you have to integrate your software package with another software package
being developed by another organization. When your software is ready, you can write
special software that gives the necessary inputs from the other software package. In
other words, you are simulating an environment in which a subsystem can be tested.
This is called test harness. Writing test harnesses is a part of test engineer's job.
 Testing unit/components is also very difficult for developers. Additional code needs
to be written to test the units/component. This test software is called stubs and
drivers.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 19

 Software Testing
2.8.7 Manual vs. Automated Testing:
For many years, software testing was done manually. In manual testing, the test engineers'
job will be as follows:
 From the specifications, design the tests
 Design the test cases
 For each test case, ensure that the necessary test setup is available and the entry
criteria are met.
 Give the test inputs
 Check whether the test results are matching with the expected results
 If there is deviation, it is a defect. Enter the defect details in a database
 Repeat the entire process for all the test cases

This manual testing has a lot of problems, Every time the testing is repeated, manually
entering the data becomes very boring and it is likely that mistakes are made in data entry.
Comparing the actual results with expected results may also turn out to be boring for many
types of testing. Entering all the details of the defect, each time, is again a very tedious task.

Lot of repetitive, unproductive work can be easily automated using software testing tools.
Nowadays, a lot of commercial and open source tools are available that can be used for
carrying out functional testing and performance testing. Tools are also available for
managing the entire testing process. Specialized testing tools are used for testing embedded
systems, networks, etc.

2.8.8 Testing Software of Different Technologies:

In your testing career, it is likely that you may need to test software for different technologies
and hence it is necessary to gain sufficient knowledge in those technologies and also learn
the details of testing software of these technologies.
 Client/Server software: Client/Server software is the most widely used software
nowadays. In Client/Server systems, the resources available on the server are
accessed by multiple clients simultaneously. Even when multiple clients access the
same resource (say, a database) simultaneously, the response should be very fast. If
the system becomes slow, it is not acceptable. The communication link between the
client and the server also affects the performance. Security is another important
feature to be tested in client/server systems.
 Web-based applications: Whether it is testing a simple web site or a complicated web
service or an e-commerce application, the following are the issues: the web
application has to work with different browsers-most applications are tested only for
one browser and for other browsers, the display just looks bad. Each and every link
on every page needs to be tested. Each page download should not take more than 10
seconds. The audio/video need to be handled with additional care. Security is the
most important issue for e-commerce applications.
 Object-oriented software: Testing software written in an object-oriented programming
language poses special challenges. Re-use is the most attractive feature of object-
oriented programming languages. So, it is likely that some portions of the present

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 20

 Software Testing
code have been developed elsewhere, and used in other projects. Adapting the code
written for some other project has to be done very carefully. Similarly, the code that
is being given to other groups for adaptation has to be tested thoroughly and given.
 Mobile applications:. Mobile phones and smart phones are being used not just for
voice communication but also for email and Internet access. Developing mobile
applications is different from developing desktop applications-small display size, low
bandwidth of mobile network, high delays and different mobile communication
protocols are the issues to be tackled. Testing the mobile applications also involves
testing in a simulated environment using mobile phone emulators and then deploying
the application over the network for field trial and testing the application. Micro-
browsers available on the mobile phones, small display size with different resolution,
and easy navigation of the menu items and so on, are the focus of testing mobile
applications.

2.9 Metrics in Testing Phase:

Some important metrics in testing phase are given below:
 Estimated time for the testing phase and the actual time taken for the testing phase.
During the planning stage, the time required for the testing phase is estimated based
on past experience or using historical data. While executing the project, the actual
time taken for the testing is measured. If the two values are close to each other, then it
can be concluded that the estimation process is good enough. Otherwise, the
estimation process needs to be improved.
 Estimated budget for the testing phase and the actual money spent on the testing
phase. These values also help in improving the estimation process.
 Estimated effort for the testing phase and the actual effort. The effort is measured in
person months.
 Defects density. The number of defects found per 1000 lines of code is called defect
density. Defect density gives an indication of the quality of the software.
 Number of defects found per unit time. During the testing phase, the testing team
has to keep track of the number of defects found per week or per day. Tracking this
data helps in finding out when testing can be stopped.
 Cost of quality. The three components of cost of quality are failure cost, appraisal
cost and prevention cost.

2.10 When is Testing Complete?

It is very difficult to say when the testing phase is completed and when the product can be
released.

2.10.1 Criteria for Completion of Testing:

It is very difficult to say when testing is complete. Three criteria used in practice for
completion of testing are:
 When you run out of time.
 When you run out of money.
 Based on statistical criteria.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 21

 Software Testing
In practice, the first two criteria are followed. During the planning stage, certain time and
money (or effort) is allocated for the testing process. The test team keeps testing the
software, and when they run out of time (money), the product is delivered. Most managers do
not realize that this is a very dangerous practice because if the software fails at customer's
site, the reputation of the organization is a stake.

A more practical approach for declaring that the testing is complete is to use the third
criterion. After the coding is completed and testing begins, initially many defects are
detected. Slowly, the number of defects found is keeps on reducing. The graph showing the
number of defects found every week would be as shown in Figure 3.7. If the number of
defects found per weeks less than a predefined threshold consecutively for three weeks, then
the software can be considered a mature product and released to the client.

5
4.5
4
3.5
3
2.5
2
1.5
Threshold
1
0.5
0
week1 week2 week3 week4 week5 week6

Defect Seeding:
Defect seeding another method used to decide when to finish the testing. In Defect seeding,
intentionally some defects are introduced in the code and the testing team is asked to detect
the defects. Suppose, 100 defects are intentionally introduced in the code and the testing team
is able to detect 90 defects. We can conclude that 10% of the defects are undetected, out of
the intentionally introduced defects; so it is likely that 10% of the actual defects are also not
detected by the testing team. Based on this conclusion, we can estimate the residual defects
(or latent defects) in the software. In other words, we can conclude that:

Latent Defects = Total defects found * (total defects intentionally seeded / seeded defects
found -1)

Suppose, during the testing you have found 300 defects in the software. When you
intentionally introduce 100 defects, you are able to find only 90 defects. So, you can
estimate the latent defects to be

Latent defects = 300 * (100/90-1) = 300 * (10/90) = 34

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 22

 Software Testing

2.11 Risk-based Testing:

In management, one of the important lessons we learn is profit is the reward for risk-taking.
Even every software project will have risks associated with it. But then what is risk?

Risk can be defined as probability of occurrence of a problem in future. Occurrence of that

problem may cause damage. Suppose you developed an e-commerce portal for online
purchase of books. If the portal is made up without enough testing on the security aspects,
then the probability of occurrence of security violations is very high and hence the damage
will also be very high. So, while testing the software, we need to reduce the risk. This testing
is called 'risk-based testing.

Mathematically, we can define risk as follows:

Risk = Probability of occurrence of problem (P) * Damage that can be caused by the problem
(D).

2.11.1 Types of Risks:

Broadly we can divide risks into two types
 Product risks
 Project risks
The product risks are due to leaving bugs in the software and delivering the software to the
customer. Some product risks are:
 Will the product work on different flavors of Windows? windows
95/NT/98/2000/XP)
 Will this software interface with the legacy database?

The project risks are due to the problems in executing a project. Failure to complete the
project within the time frame (time overrun) or within the budget (budget overrun) are
project risks. If the project is not completed within the time frame, there are two types of
damage:
1. You need to spend additional money due to the extended time on manpower complete
the project.
2. The customer may impose a late delivery clause and cut some money from your total
project cost. Such damages are very costly and project risks need to be minimized to a
great extent.

Some of the project risks are:

 Shortage of skills of team members
 Internal politics
 Lack of management support for testing/ QA
 Third party supply failure
 Persons leaving in the middle of the project
 Accepting the project without studying the requirements well.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 23

 Software Testing
Risk-based tastings involves the following, steps:
 Identify the various risks for the project- product risks and project risks.
 Analyze the constraints
 Based on the constraints, prioritize the tests to be carried out
 Identify the types of testing to be done
 Identify whether use of testing tools will be of help in reducing the risks.

2.12 The Myths and Realities of Testing:

There are some myths prevalent amongst the prospective test engineers. Here are some of
them:
 Testing is easy: Many young engineers fresh out of school think that testing is a very
easy job and they prefer a more challenging job 'development. It needs to be
appreciated that testing is equally challenging job and a creative bent of mind is
required to be a test engineer.
 Test engineers have no career growth: Some test engineers, when they begin their
career, look at their jobs as a stopgap arrangement. They do not look at it as starting a
new career. Test engineers can have a career as lucrative as a development engineer.
If you keep upgrading your skills continuously, career growth is no issue. Test
engineers become better developers as they can bring out quality software. Test
engineers can take up software quality assurance as their career option, which
involves development of processes for the entire organization.
 If I learn a testing tool, I will get a job: Learning how to use testing tools is an
important skill for prospective test engineers. However, that alone is not enough You
need to have a good understanding of the software quality assurance, learn the testing
process thoroughly, gain expertise in at least one application domain and gain
experience by working hands-on on a few testing projects.
 Testing does not involve coding: With the availability of testing tools, carrying out
testing repeatedly using the same test cases has now become easy, as no coding is
required. However, to do advance testing, a number of test scripts have to be written.
Writing test scripts is similar to writing code in any programming language. And, you
also need to think how long you keep coding in your life, you got to grow up.
 Testing is a phase that comes at the end of development life cycle: If testing is
taken up very late (when you are left with 15 days to deliver the software to the
customer), then it is like taking a cancer patient to a doctor when the disease is out of
control. Testing must be started at very early stages. If a module is ready, test it
without waiting for others to complete their modules and then integrate them.
 Testing is completed: When you start testing the software, you find many defects
and slowly with rigorous testing, the number of defects found keeps reducing. Even if
you do not find a single defect even after say, one week of testing, it does not mean
that the software has no defects. Perhaps you have not designed the correct test cases!
 Developers know that their software will have bugs: Unfortunately, no software
developer thinks that his/her code has any defect. If you tell him/her that the software
is not working properly (and also show it), even then, they will not agree. They will
tell you 100 reasons why it did not work at that time: maybe it is because of the

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 24

 Software Testing
keyboard or because of the problem with the operating system or something else! It is
because the developer gets emotionally attached to their work! It is sad, but it is a
fact.
 Test engineers are the most-disliked people in the organization: In most
organizations, test engineers and quality assurance people are indeed disliked. This is
because developers do not like their mistakes being pointed out by someone else. If
you point out a problem, you become the bad guy. So, development group and the
testing/QA group always fight with each other. But the test/QA engineers have to
strive to be liked. This can happen if the test engineers communicate effectively. It is
an art to point out a problem without hurting the person. If you cultivate that habit,
you will be a great test engineer (and it also helps you in becoming a great
husband/wife).

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R. INAMDAR) Page 25