SOFTWARE TESTING

FUNDAMENTALS

Dr.P.Keerthika/Associate Professor/Department of
CSE/ Kongu Engineering College
Software Testing
Why Testing is important?
BASICS OF SOFTWARE TESTING
Overview
• Introduction
• Testing – Definition
• Approaches to Testing
• Essentials of Software Testing
• Principles of Software Testing
• Salient Features of Software Testing
• Test Policy & Challenges
• Test Strategy
• Test Planning
• Cost of Testing
• Categories of defect
• Developing Test Methodologies
• Testing Process
• Testing MEthodologies
Software Testing - Definition
• Execution of a work product with intent to find a
defect
• Detecting the difference between existing and
required conditions and to evaluate software
features
• As a process, designed to
– Prove that the program is error/defect free
– Establish that the software that software performs
functions correctly and fit for use
– Establish that all functional expectations are available
 Software Verification & Validation

Comparing a work product with Checking the outcome of developed

processes, standards & product with respect to expectation
guidelines of customers
Cost of Software Verification &
Validation
• If done first time – comes under appraisal cost
• If repeated times – comes under failure cost – for
organizations undergoing testing
• Software Testing – finding the difference between actual
behaviour and expected behaviour of an application
• Stages of ST as per SDLC
– Feasibility testing Begins at start of project
– Contract Testing
– Requirements Testing
– Design Testing
– Coding Testing
– Acceptance Testing Continues till the end of project
Historical Perspective of Testing
• Debugging Oriented Testing
– Developers tests – not documented
• Demonstration Oriented Testing
– Testers - through demonstration
• Destruction Oriented Testing
– Negative way of testing – change in tester‟s responsibility
• “proving that software doesnot fail at some abnormal instances” instead of
“proving that software works under normal conditions”
• Evaluation Oriented Testing
– Evaluation based on metrics/parameters
• Prevention Oriented Testing
– Followed by highly matured organizations
– To make it defect-free
Why Testing Necessary
• Understanding of customer requirements differ from person
to person
– So, to be checked at each stage of SDLC
• Developers think what they developed is always right &
satisfies user requirements
– but not correct - it should be assessed.
• Different entities – involved in different phases
– may be gaps between requirements, design, coding
• May have integration issues
– unit, integration testing
• Blind fold thoughts of developers
– So, testers should challenge them
Approaches to Testing
 Approaches
 Big
Bang Approach
 TQM Approach

 TQM as against Big Bang Approach

 Characteristics of Big Bang Approach
 TQM in Cost Perspective
1. Big Bang Approach
 Testing after development completed
 System testing/final testing – done before releasing
the software
 Last part of SDLC as per waterfall model
 Final product tested- before release
 Has main thrust on black box testing
 1. Big Bang Approach
• Phase-wise defect distribution
Development Phase Percentage of Defects
Requirements 58
Design 35
Coding 5
Other 2

• Characteristics
– All defects found in last phase
– Cost-huge
– Huge rework, retesting, scrap, sorting of software components
– No corrective & preventive actions – arise from these defects
– Regression testing – correcting defects leads to creation of new defects –
dependent components affected
1. Big Bang Approach
• Organizations following Big Bang Approach
– Are less matured
– May spend huge cost of failure
– Needs several retesting & Regression Testing
– Observables:
• Verification activities (during SDLC) – can find 2/3 of total
defects
• Validation activities (in unit testing) – can find ¾ of remaining
defects
• Verification activities (during System testing) – 10% of total
defects
• Remaining 5% to 10% defects – identified only when the product
reaches customer – unless the organization takes preventive
measures
doesn’t
Big Bang Preventive Measures
go for
 2. TQM Approach
• TQM Cost Triangle
– Stages of maturity
• 0 – highly matured
– No need of verification/validation
– Quality is free
• 1 – Verification done
– Will have appraisal cost
• 2 – validation
– filters the defects escaped from verification
– High validation cost
• 3 – defects found by customer
– Results in 1000 times much higher cost
2. TQM Approach
 Less defects produced
 Very few defects undetected
 Defect removal costs
 After coding (approx. 10 times >) before coding
 During Production (approx. 100 times>) before coding

 TQM – aims at reducing cost of development, cost

of quality by continual improvement
TQM Cost Perspective
• Green Money / Cost of Prevention
– Money spent on defining process, training people, developing quality
– Investment in doing quality work
– Improves profitability
• Blue Money / Cost of Appraisal
– Cost incurred in development, 1st time review/testing
– Doesn‟t return profit
– Includes all appraisal techniques used during SDLC such as 1st time
verification/validation
• Red Money / Cost of Failure
– Pure loss for organization
– Money lost in rework, sorting, scrap
– Directly reduces profit
– Customer doesn‟t pay for it
– Includes cost incurred in re-inspection, re-testing, regression testing
Views of Testing
 Manager‟s View:
 Product must be safe, reliable, when used by users
 Satisfy user requirements

 Testing process- should uncover defects & give

confidence to customer
 Tester‟s View:
 To discover defects, faults, weakness of product
 Customer‟s View:
 Testingmust give confidence
 All defects removed

 s/w must be defect-free by testing

Objectives of Testing
 Must find
 What should be done & What should not be done
 Deviation from requirement specification

 Risks- what should be done

 Software Testing
A SQA activity – in many places
 But in Reality - it is a Quality Control (QC) activity
Basic Principles of Testing
 Define the expected output or result for each test
case
 Defects may be in the product or test case or test
plan
 Developers or the development teams must not test
their own products
 Inspect the results of each test completely and
carefully
 Used to identify the weak processes, build right
processes and improve the process capability
Basic Principles of Testing
 Include test cases for invalid and unexpected
conditions
 Test the program to see if it does what it is not
supposed to do and what it is supposed to do
 Do not plan tests with an assumption that no error
will be found
 The probability of locating more errors in any
module is directly proportional to the no. of errors
already found in that module
Successful Testers & Test Cases
 Successful Testers – must conduct SWOT analysis of
software and the processes used to make it
 Successful Test Case – must consider all cases - +ve,
-ve, all scenarios
Testing during SDLC
 Requirement Testing
 Mock running of future application using requirement statements
 Completeness of Requirement statements
 User Expectation clarity
 Measurability of expected results
 Testability, Traceability of requirements at the end

 Design Testing
 completeness

 Code Testing
 Readability, maintainability in future, ability to integration testing

 Test Scenario & Test Case Testing

 Should be feasible
 Should coverall requirements
 Test Case – should cover all scenarios
Requirement Traceability Matrix
 Tracing from requirements to coding, design….
 A blueprint of software under development
 Matrix Structure
Requirement Traceability Matrix
 Advantages:
 Used to understand the software in a better way
 Helps in tracing if any software requirement is not
implemented (OR) any gaps between requirements
and design & code &…
 Entire software can be tracked completely

 Test case failure can be tracked completely

 Problems:
 Theoretically – all software must have
 In reality – most software doesnot have

WHY???????
Requirement Traceability Matrix
 Why???
 If number of requirements is high,
 Preparing RTM is difficult
 One-one, Many – one, one- many, many - many
relationships exist
 Needs more efforts to connect columns and rows
 Requirementschanges frequently
 Customer doesn‟t find any use of it – hence, will not pay
for it.
Requirement Traceability Matrix
 Horizontal Traceability
 Traced from requirement ---through----test results
 Vertical Traceability
 A requirement may have child requirements – Tracing from parent to
child requirements
 Bidirectional Traceability
 Horizontal in both directions – requirements to results, results to
requirements RISK TRACEABILITY MATRIX
 Risk Traceability
 Tracing the possibility of risk of failure
Essentials of Software Testing
 SWOT Analysis
 Strengths
 Identifying strong areas
 Weakness
 Failure possibilities
 Opportunity
 Identifying space for improvement
 Threats
 Identifying risks that results in failure
Workbench
 Tester‟s Workbench
Workbench
 Tester‟s Workbench
 For creating test strategy
Input
 Test plan
Do Process
 Writing test scenario

 Writing test cases Check Process

 Test execution Output
 Defect Management
Standards & Tools
 Retesting

 Regression Testing Rework

Important Features of Testing
Process
 It is a destructive process, but a constructive destruction
 Testing needs a sadistic approach with a consideration that
there is a defect
 Testers should identify the risks to the users and test the product
accordingly
 If the test does not detect the defect, then it is an unsuccessful
test
 A test that detects the defects is a valuable investment for
development and the customer
 Helps in product improvement
 Identifies the weaker areas of the processes used for S/W development,
improves the processes and results in customer satisfaction
Important Features of Testing
Process
 It is risky to develop a software and leave it untested
before delivery
 Reducing the coverage of testing is a risk associated with
the S/W
 Testing must give the desired level of confidence to the users
that the product will not fail

 With high pressure to deliver a software as quickly as

possible, Test process must provide maximum value in
shortest time frame
Important Features of Testing
Process
 Highest payback comes from detecting defect early in SDLC
and preventing defect leakage/defect migration from one
phase to another
 It is always economical to fix the defects as and when they appear and
conduct an analysis to find its root cause
 Phase contamination – uncorrected defects in the phase where it is
detected leads to leakage of the defects to the next stage and creates
problem
 Organizations‟ aim must be “defect prevention rather than
finding and fixing a defect”
 Consider testing as a defect-prevention mechanism
 Requires analysis of root cause and defect prevention mechanism to
prevent recurrence and removal of potential problems
Misconceptions about Testing
 Anyone can do testing, & no special skills are
required for testing
 Testers can test quality of product at the end of
development process
 Defects found in testing are blamed on developers
 Defects found by customers are blamed on testers
Principles of Testing
 Programmers/Team must avoid testing their own
work products
 Thoroughly inspect results of each test to find
potential improvements
 Initiate actions for correction, corrective action and
preventive action
Salient Features of Good Testing
 Capturing user requirements
 Testers need to analyze and document the requirements so that they can write test scenarios and test
cases
 Capturing user needs
 Present and future requirements, process requirements, and implicit requirements
 Design objectives
 They state the reason for choosing a particular approach to build a S/W
 Functional , UI, and performance requirements and other such requirements need to be tested
 User interfaces
 Users ability to interact with the system, receive error messages, and act according to the instructions
 Displays and reports generated by the system
 Internal structures
 Guided by software designs and guidelines used in design and development
 Ex: commenting standards to be used
 Execution of code
 Prove that application, module and program work correctly as per the requirements
Test Policy
 How testing should be done
 Can be defined by Senior Management – covering
all aspects of testing
 Decides the framework of testing and its status in
the mission of achieving customer satisfaction
Test Strategy / Test Approach
 Defines the action part of the test policy
 Differ from
 product to product
 customer to customer

 time to time

 Examples
 Definition
of functional coverage/requirement
coverage/feature coverage of a particular product
 How much testing would be done manually

 What can be automated?

 Number of Testers / developers

 Test planning
 1st activity of the test team
 Should talk about limitations and constraints of testing
 Should talk about risks assumptions during testing
 Plan testing efforts adequately with an assumptions
that defects are there
 If defects are not found, it is failure of testing activity
 Successful tester is not one who appreciates
development but one finds defect in the product
 Testing is not a formality to be completed at the end
of the development cycle
 Testing process & no of defects
found
 We assume that
“If more number of defects found then chances of
customer finding the defects is reduced”
 Actually
“If more number of defects found then there is a
probability of finding some more defects”
Test team efficiency
 dependent on organization culture.
 Test manager should be aware of the efficiency of
the test team.
 Often, Assess the test team efficiency.
 Efficient Test team : less iteration of testing is
required
 Less Efficient Development team : more iterations
needed in fixing defects.
Test team efficiency
 If there are 100 defects in a product then
if testing is 90% efficient - > It can find 90
errors
 If there are 200 defects in a product
if testing is 90% efficient - > It can find 180
errors.
Test team efficiency
Defects introduced during development= x
Total defects found by test team =y
( includes both defects introduced during development & others)

Defects found that does not belongs to defects

introduced during development =z

Test team Efficiency = (y-z) / x

 Mutation testing
 is a type of white box testing where we mutate (change)
certain statements in the source code and check if
the test cases are able to find the errors
 Test Cases:
 Designed & executed to find defects
 If test cases – not capable of finding defects – loss for an
Organization
 Test case efficiency
 Defects deliberately introduced by development = X
 Defects found by test cases in original program =Y

 Defects found by test cases in mutant = Z

 Test Case Efficiency = (Z-Y)/X

 Test Team Approach
 The test team is defined by the type of
organization and the type of product being
developed
 Four different approaches
 Location of the test team in an organization
 Independent Test Team
 Test team reporting to Development manager
 Matrix Organization

 Developersbecoming testers
 Independent testing team – outside organization

 Domain experts doing software testing

Location of Test Team - Independent Test Team

 Independent of development activities and

does not report to the development group
 Reports only to the senior management

 Test manager is essential to lead the test

team

Senior
Management

Development Team Test Team

 Location of Test Team - Independent Test Team

Advantages
 Test team is not under delivery pressure

 Test team is not under a pressure of „not finding‟ a defect

 Independent view about a product is obtained

 Expert guidance and mentoring is given by the test manager

Disadvantages
 Always „us‟ Vs „them‟ mentality between the development

and test team

 Testers may not have a good understanding of the

development process
 Management may be excessively inclined towards either of
the team
 Location of Test Team - Test Team Reporting to
Development Manager

 Test team reports to the development

manager and hence involved from start of
the project.
Development
Manager

Development Test Team

Team
Location of Test Team - Test Team
Reporting to Development Manager
 Advantages
 Better cooperation between the development and test
team

 Test team can be involved in development and

verification/validation activities.

 This gives them a good understanding of the

requirements and the development process
Location of Test Team - Test Team
Reporting to Development Manager
 Disadvantages

 Expert guidance by the test manager may not be

available

 Sometimes the development managers are more

inclined towards development team
 Location of Test Team – Matrix
Organization
 Effort is made to achieve the advantages of both
the approaches and get rid of the drawbacks of
them
 Test team reports functionally to the development
manager and administratively to the test
manager
 Developers Becoming Testers
 Developers in the initial stages of SDLC take
the role of testers in the later stages
 Suitable when the application is technologically

heavy or simple
 Developers Becoming Testers -
Advantages
 Developers are not in need of knowledge
transfer.
 Developers have better understanding of

design and coding and does the testing easily

 Developers have the capability to adapt to

automation testing
 Less costly as there is no separate test team

 Psychological acceptance is not an issue as

they themselves find the defect
 Developers Becoming Testers-
Disadvantages
▪ Developers may not find value in performing
testing.
▪ They may be blindfolded in understanding the
requirements or selection of approach and may
not be willing to find defects.
▪ They may concentrate more on development
activities.
▪ Development needs more f creation skill while
testing needs destruction skill.
 Independent Testing Team
 Separate testing team with independent
responsibility of testing
 People in the team have sufficient knowledge

and skill to perform testing

 Independent Testing Team-
Advantages
 The team concentrate more on test planning,
test strategies and approach, test artifacts, etc.
 Have independent view about the work

products
 Special skill required to do special tests may

be available with the team

 Testers work for customers
 Independent Testing Team-
Disadvantages
 Additional cost for the organization

 Testing team needs ramping up and

knowledge transfer

 Organizations need to check for jealousy

between development and test team
 Domain Experts Doing Software
Testing
 Organizations employ domain experts to do
testing
 Useful in system testing and acceptance testing

where domain specific testing is required

 Domain Experts Doing Software
Testing- Advantages
 “Fitness for use” can be tested; Experts test the
software from user‟s perspective
 Domain experts assist the developers about he

defects and customer expectations.

 Also interpret requirements in the correct

context
 They understand the scenario faced by the

actual users
 Domain Experts Doing Software
Testing - Disadvantages
 Domain experts may not understand what a
customer is looking for

 It is very difficult to get domain experts in

diverse areas for a project in diverse domains

 Huge cost as these experts cost much more

than the developers and testers.
Process problems faced by Testing
 „Q‟ Organizations consider that defects are due to
incorrect process
 Incorrect process leads to 90% of working problems
 If software testing process is faulty, it introduce
defects which are not found during testing but found
by a customer
 Those defects may be „Not a defect‟, „duplicate‟,
„cannot be reproduced‟, „out of scope‟.
 Constituents of Processes
 People
 Material
 Machines
 Methods
People
 Customer / user – Specifying requirements
 Business analyst / system analyst – Documenting
requirements
 Test managers/ test leads – Defines test plans and
test artifacts
 Testers – defines test scenarios, cases, test data.
 Personal attributes and capabilities may create
problems in development and testing.
 Proper skill sets may not be available.
Material
 Requirement doc, Development standards, Test
standards, Guidelines and other material are not
available.
 Those documents are not clear and incomplete.
 Test plan, project plan, organization process
document may be faulty
 Test tools and defect tracking tools not available.
Machines
 Test cases-uses various machines, simulators and
environmental factors for representing real life
scenarios.
 Problems may be generated for wrong
environmental configurations.
 Usage of wrong tools.
Methods
 Methods for test planning and risk analysis are not
clear.
 Defining test cases, test scenarios, test data may not
be proper.
Economics of testing
 Cost of T. Curve is
guided by,
 Testteam efficiency
 Defect fixing efficiency

 Defect introduction index of development team

 Cost of customer dissatisfaction is guided by,

 Customer - supplier relationship
 If more projects successful, customer will be confident
 Cost Aspect of Testing
Cost of quality = Cost of prevention + Cost of
appraisal + Cost of failure

Cost of quality is 50% of the cost of the product

(Reduce the cost of the testing to the maximum extent)

Cost of product = cost for development + cost for testing

 Cost Aspect of Testing –
Resources for development
 Cost Aspect of Testing –
Cost of development

Cost of development =
[no of resources for development project]
[ cost of capturing requirements, conducting analysis,
asking queries, elicitation of requirements + design
cost (both high level and low level design) + coding
cost (integrating & creating final product)]
 Cost Aspect of Testing
 Assessment of Cost of Testing
 Cost of Prevention in testing
 Cost of Appraisal in Testing
 Cost of failure in Testing
 Cost Aspect of Testing
 Assessment of Cost of Testing
 Based on
 Type of application
 Development and Testing methodology
 Domain and Technological aspects
 Maturity of Development and testing processes
 Cost of testing – based on
 Size
 Importance of an application to a user
 K f(x)
 K- constant

 F(x) – function of size of an application

 Cost Aspect of Testing
 Cost of Prevention in testing
 Cost spent in creation of verification & validation
artifacts
 Cost spent in training

 Cost spent in preventing defects

 Planning for Verification & validation
 Writing test cases, scenarios,…
 Test team competency, training….

 Cost of Appraisal in Testing

 First time verification & validation testig
 Cost Aspect of Testing
 Cost of failure in Testing
 Calculated on the basis of
 Historicaldata, pre-exit without testing
 No. of test cases per iteration
 No. of iteration
 Regression testing
 Retesting
Establishing Testing Policy
 Testing driven by
 Test policy
 What should/should not be done
 Test strategy/approach
 Stepsinvolved
 Can be discussed with user

 Test planning
 Should contain test objectives, methods applied for defining
test scenario, test cases and test data
 Test objectives
 What is the target
 Methods - applied for testing efforts are defined at organizational levels
Structured Approach to Testing
 Testing – done at all phases of SDLC
 Wastes in Testing – due to Wrong approach
 Waste in wrong development
 Waste in testing to detect defects

 Wastage as wrong specifications, designs, codes,

documents
 Mustbe replaced by correct spec, designs, codes &
documents
 Wastage as system must be retested to ensure that the
corrections are correct
Categories of Defects
 Must be defined in test plan
 Differ from organization to organization, project to project,
customer to customer
 Categories:
 On the basis of requirement specification
 Variance from product specifications, requirement specification -
Responsible for „producer‟s gap‟
 Variance from user expectations – Responsible for „user‟s gap‟

 Types of defects
 Misinterpretation of specifications
 Missing specifications
 Features not supported
 Root causes of defects
Defect, Error, Mistake in software
Developing Test Strategy
 Select and rank test factors for the given
application
 Identify the system development phases and
related test factors
 Identify associated risks with each selected test
factor in case if it is not achieved
 Identify phase in which risks of not meeting a test
factor need to be addressed
Developing Test Methodologies
(Test Plan)
 Acquire and study test strategy as defined earlier
 Determine the type of development project being
executed
 Based on different organization
 Based on criticality
 Life
affecting software
 Huge money affecting software

 Determine the type of software system being made

 Identify tactical risks related to development
 Structural risks
 Technical risks
Developing Test Methodologies
(Test Plan)
 Determine when testing must occur during life cycle
 Steps to develop customized test strategy
 Select and rank test factors as expected by customer
 Identify system development phase where these factors
must be controlled Test Strategy Matrix

 Identify business risks for software under development

 Place risks in matrix in- order to analyze

Developing Test Methodologies
(Test Plan)
 Types of development methodology impact test plan
decisions
 Testing Process
 Defining test policy
 Defining test strategy
 Preparing test plan
 Establishing test objectives to be achieved
 Design test scenarios and test cases
 Writing/reviewing test cases
 Defining test data
 Creation of test bed
 Executing test cases
 Test result
 Test result analysis
 Performing retesting/regression testing when defects are resolved by
development team
 Root cause analysis & corrective/preventive actions
Test Methodologies/Approaches
 Black box testing
 Product tested as per s/w specifications, requirement
spec.
 By business analysts/system analyst/customer

 White box testing

 Software is tested for structures, architecture, design,
coding, standards….
 Gray box testing
 Combination of both
 Combined wherever necessary – depending on the
requirements of the product
 Black box testing
 Considers general functionalities as in requirement
spec
 Advantages:
 Proves functionality of software as what it should do/
should not do
 Disadvantages:
 Logical
error in coding-missed
 Same code tested many times
 Black box testing
 Test case designing methodologies
 Designing test cases for Black box testing – difficult
 Test scenarios can be used for defining test cases

 Test data definition

 Techniques
 Equivalence partitioning
 Boundary value analysis
 Cause and effect graph
 State transition testing
 Use case based testing
 Error guessing
White Box Testing
 Done on the basis of Internal structure of software
 Reviews of requirements, design, codes
 Advantages:
 For verification
 Ensures whether procedures, methods followed properly

 Gives early warnings-if something wrong

 Disadv:
 Does not check – user requirements-met correctly
 Does not report whether decisions made are based on
requirements
 Use of checklists
 White box testing
 Test case designing methodologies
 Techniques
 Statement coverage
 Decision coverage
 Condition coverage
 Path coverage
 Logic coverage
Gray Box Testing
 Both verification & Validation
 Advantages:
 Tested - Both functionally and structurally
 Disadvantage:
 Usually done by automation tools
 Hence, understanding is tough
Skills required by Tester
 Testing Tips
 Testing must occur throughout SDLC

 Testing must cover functional/structural parts

 Skills – General, Testing Skills

 General Skills
 Written & verbal presentation skills
 Effective listening skill
 Facilitation skill
 Software development, operations & maintenance
 Continuous education
Skills required by Tester
 Testing Skills  Execution of test plan
 Continuous improvement of
 Concepts of testing
testing process
 Levels of testing
 Reduce software development
 Techniques for verification & risk
validation
 Perform testing effectively
 Selection & use o testing tools
 Uncover maximum number of
 Knowledge of testing defects
standards
 Use business logic
 Risk assessment and
management
 Developing test plan
 Defining acceptance criteria
 Checking of testing process
Thank you