Testing Principles And Concepts

CBOK Knowledge Domain 5

Testing Principles And Concepts
- Outline
• Testing Techniques

• Testing Methods

• Independent Testing
Testing Techniques
• Various approaches used in testing
– Static or “human” (desk checking)
– Dynamic (testing that executes system code)
Dynamic Testing
• Techniques used are determined by the type of
testing that must be conducted
– Functional
– Structural
Functional Testing
• Addresses the overall behavior of the program by
testing the following:
– Transaction flows
– Input validation
– Functional completeness
Structural Testing
• Includes the following:
– Path testing
– Code coverage testing
– Analysis testing
– Logic testing
– Nested loop testing
– Performance testing
– Stress testing
– Others...
White Box Testing
Consists of testing paths, branch by branch, to
produce predictable results
White Box Testing Techniques
• Statement coverage

• Decision coverage

• Condition coverage

• Decision/Condition coverage

• Multiple Condition coverage

Statement Coverage
Execute all statements at least once
Decision Coverage
Execute each decision direction at least once
Condition Coverage
Execute each decision with all possible outcomes at
least once
Decision/Condition Coverage
• Execute all possible combinations of condition
outcomes in each decision

• Treat all iterations as two-way conditions exercising

the loop zero times and once
Multiple Condition Coverage
Invokes each point of entry at least once
White Box Testing Techniques
continued...
• Choose a combination of techniques appropriate for
the application

• Too exhaustive a combination of these techniques

leads to an unmanageable number of test cases
Black Box Testing
• Focuses on testing the function of the program or
application against its specifications

• Determines whether combinations of inputs and

operations produce expected results
Black Box Testing continued...
• When creating black box test cases, the input data
used is critical

• Three successful techniques for managing the

amount of input data required:
– Equivalence Partitioning
– Boundary Analysis
– Error Guessing
Equivalence Partitioning
• A subset of data that is representative of a larger
class

• A technique for testing equivalence classes rather

than undertaking exhaustive testing of each value of
the larger class
Equivalence Partitioning
continued...
• For example, a program which edits credit limits
within a given range ($10,000 - $15,000 would have 3
equivalence classes:
– Less than $10,000 (invalid)
– Between $10,000 and $15,000 (valid)
– Greater than $15,000 (invalid)
Boundary Analysis
A technique that consists of developing test cases
and data that focus on the input and output
boundaries of a given function
Boundary Analysis continued...
• In the same credit limit example, boundary analysis
would test:
– Low boundary plus or minus one ($9,999 and
$10,001)
– On the boundary ($10,000 and $15,000)
– Upper boundary plus or minus one ($14,999 and
$15,001)
Error Guessing
• Based on the theory that test cases can be
developed based upon the intuition and experience of
the Test Engineer

• For example, in an example where one of the inputs

is the date, a test engineer might try February
29,2000 or 9/9/99
Incremental Testing
• A disciplined method of testing the interfaces
between unit-tested programs as well as between
system components

• Involves adding unit-tested programs to a given

module or component one by one, and testing each
resultant combination
Incremental Testing Types
• Top-down

• Bottom-up
Top-Down
• Begins testing from the top of the module hierarchy
and works down to the bottom using interim stubs to
simulate lower interfacing modules or programs

• Modules are added in descending hierarchical order

Bottom-Up
• Begins testing from the bottom of the hierarchy and
works up to the top

• Modules are added in ascending hierarchical order

• Bottom-up testing requires the development of driver

modules which provide the test input, call the module
or program being testing, and display test output
Thread Testing
• A technique, often used during early integration
testing

• Demonstrates key functional capabilities by testing a

string of units that accomplish a specific function in
the application
Thread & Incremental Combined
• Thread and Incremental testing usually utilized
together
– Units can undergo incremental testing until
enough units are integrated and a single business
function can be performed, threading through the
integrated components
Effective Strategies
For a simple two-tier client/server application:
1. Unit and bottom-up incremental testing of the
application server components
2. Unit and incremental testing of the GUI or client
components
3. Testing of the network
4. Thread testing of a valid business transaction
through the integrated client, server, and network
Illustrates various techniques
used throughout the test stages
Stages/ White Black Incremental Thread
Techniques Box Box

Unit Testing X

String/ X X X X
Integration
Testing
System X X X
Testing

Acceptance X
Testing
Establishing A Test Policy
• The establishment of a testing policy is an IS
management responsibility

• Three methods can be used to establish a testing

policy:
– Management directive
– Information technology consensus
– User’s meeting
Management Directive
• One or more senior IT managers write the policy

• They determine what they want from testing,

document that into a policy, and issue it to the
department

• This is an economical and effective method for writing

a testing policy; the potential disadvantage is that it is
not an organizational policy, but rather the policy of
IS management
Information Technology
Consensus
• IT management has a group of more senior and respected
individuals in the department to develop a policy
• Senior management must have the responsibility for accepting
and issuing the policy
• Development of the policy is representative of the thinking of all
the IT department
• It is advantageous, as it involves key members of the IS
department, therefore, staff is encouraged to follow the policy
• The disadvantage is that it is an IT policy and not and
organizational policy
User’s Meeting
• Key members of user management, representing all areas of the
organization, work with IS to jointly develop a testing policy
• IT management has final responsibility for the policy
• The advantage is that it is a true organizational policy and
involves all areas with an interest in testing
• Disadvantages:
– The time required to follow this approach
– A policy might be developed that IT is obligated to accept
because it is a consensus policy and yet is not the type of
policy that IT itself would have written
Test Policy continued...
Testing is an organizational responsibility. It is the
recommendation of the author that a user committee be
convened to develop a testing policy. This meeting
serves the following purposes:
– It permits all involved parties to participate in the
development of a testing policy
– It is an educational process where users understand
the options and costs associated with testing
– It clearly establishes for all involved departments that
testing is an organizational responsibility and not just
an IS responsibility
Test Policy continued...
• Processes, procedures, and standards are designed
and implemented to support the fulfillment of
organizational policies.
• If there is no organizational testing policy established
or supported by executive management, it will be
difficult to gain support for the implementation of test
processes.
• Test processes that aren’t tied to a policy are
considered guidelines and do not have to be
followed.
Independent Testing
A fundamental principle of effective testing is that a
program or application should not be tested by the
developer or development team that created it.
– The developer’s goal is to make the application
work
– The application tester’s goal is to make the
application fail
Independent Testing continued...
• By nature, it is very difficult for an individual to adopt
both mindsets.

• Based on this principle, it is recommended that most

testing be performed by independent testers or test
teams.
Independent Testing continued...
• The primary responsibility of individuals accountable for testing
activities is to ensure that quality is measured accurately
• Roles and reporting structure of test resources differ across
and within organizations
– Business or Systems Analysts
– Test Engineers
• Ideally, the test resources will have a reporting structure
independent from the group designing or developing the
application, this helps assure the quality of the application is
given as much consideration as the project budget and timeline
Independent Testing continued...
• Misconceptions regarding the skill set required to
perform testing:
– Testing is easy
– Anyone can perform testing
– No training or prior experience is necessary
Independent Testing continued...
• In truth, to test effectively, an individual must:
– Thoroughly understand the system
– Thoroughly understand the technology the system
is being deployed upon (e.g. client/server or
Internet technology introduce their own
challenges)
– Possess creativity, insight, and business
knowledge
– Understand the development methodology being
used and the resulting artifacts
Independent Testing continued...
• An independent test team is usually responsible for
system testing, the overseeing of acceptance testing,
and providing an unbiased assessment of the quality
of an application

• Team may also support or participate in other phases

of testing as well as executing special test types,
such as performance and load testing
Independent Testing continued...
• Usual Test Team Is Comprised Of:
– Test Manager or Team Lead
– Test Engineers
– Additional Testers
Independent Testing continued...
• Roles Of Players On The Test Team:
– Test Manager should join the team no later than the
start of the requirements definition stage
– Key Test Engineers may also join the team at this stage
on large projects to assist with test planning activities
– Other Test Engineers can join later to assist with the
creation of test cases and scripts
– Additional Testers, including users who will participate
in testing, usually join the test team right before system
testing is scheduled to begin
Independent Testing continued...
• The Test Manager ensures that testing is performed, that it
is documented, and that testing techniques are established
and developed. They are responsible for ensuring that tests
are designed and executed in a timely and productive
manner, as well as:
– Test planning and estimation
– Designing the test strategy
– Reviewing analysis and design artifacts
– Chairing the test readiness review
– Managing the test effort
– Overseeing acceptance tests
Independent Testing continued...
• Test Engineers are usually responsible for:
– Developing test cases and procedures
– Test data planning, capture, and conditioning
– Reviewing analysis and design artifacts
– Test execution
– Utilizing automated test tools for regression testing
– Preparing test documentation
– Defect tracking and reporting
Independent Testing continued...
• Other Testers:
– Primarily focus on test execution
– Defect reporting
– Regression testing

• These Testers may be junior members of the test

team, users, marketing or product representatives,
etc...
Independent Testing continued...
• The test team should be represented in all key
requirements and design meetings, including: JAD or
requirements definition sessions, risk analysis
sessions, prototype review sessions, etc…

• They should also participate in all inspections or

walk-throughs for requirements and design artifacts
Testing Principles And Concepts
- Summary
• Establish a Test Policy

• Develop a test strategy that is relevant to the situation

– Include the types of testing to be performed

• Ensure testing is performed by an independent test

team
Bibliography
• CSTE Study Guide, QAI, 2000
• Perry, William E. Effective Methods for
Software Testing, Wiley Computer
Publishing, 1995