Functional System Testing

Written by Aishwarya Bisht

Outline
Goal of testing Test cases, test suites and test data What is functional system testing? Coverage Functional testing techniques:
Functional analysis Equivalence partitioning Boundary value analysis
2 Black Box Testing

The goal of software testing

The process of uncovering evidence of defects in software systems
Does not include efforts associated with tracking down bugs and fixing them

No amount of testing will improve the quality of a computer program

The more testing we do of a system, the more convinced we might be of its correctness Testing cannot in general prove a system works 100% correctly
3 Black Box Testing

Test cases
The basic component of testing is a Test Case In its most general form: (inputs, expected-result)
inputs include system state, user commands and data values to be processed expected result includes visible/audible interface changes or changes in the system state

Test cases are organized into Test Suites

functionality, security, performance,

Black Box Testing

Test case execution

A running of the software (under test) that provides the inputs specified in the test case and observing the results and comparing them to those specified by the test case
If the actual result varies from the expected result, then a failure has been detected

Black Box Testing

Test data
An effective test strategy requires careful acquisition and preparation of test data prior to testing
Testing can suffer if test data is poor

Test data concerns:

Depth: quantity and size of data Breadth: variance of data values and data types Scope: completeness, relevance and accuracy of data Result of a query should be valid for the specific purpose of the query, and not due to a missing or inappropriate value Conditions: data should reflect specific conditions in the domain Data that would otherwise arrive after performing specific operations over time

Test data and test results are expensive to construct

6 Black Box Testing

Example: Test data for TVRS

Name: test1.db Description: Violation ID 243567 237812 264683 255245 000345
7

Violation records designed for validating violation lookup

Offenders first name Rachel Dan Dan Dina longFirstN Offenders last name Josef Levi Porat Josef longLastN
Black Box Testing

Issuing policeman ID 8700342 6386541 1346329 8245731 8700342

Specification Vs. Implementation

The basic approaches to testing software are based on its specification and implementation White box testing test cases and data are constructed based on the code that implements the software
Quality and correctness of computations is validated Will not be further discussed in this tutorial

Black box testing test cases and data are constructed based solely on the softwares specification

Black Box Testing

Functional System Testing

Testing of a completed application to determine that it provides all of the behaviors required of it
Testing of completed increments that provide some degree of end-user functionality

Search for defects that are variances between the actual operation of the system and the requirements for the system System is treated as a black box
9 Black Box Testing

How much testing is adequate?

Completely validating IEEE 754 floatingpoint division requires 264 test-cases!
float divide(float x, float y)

From practical and economic perspectives, exhaustive testing is usually not possible
Which software pieces should we test? Which test cases should we choose?
10 Black Box Testing

Coverage
Coverage is a measure of how completely a test suite exercises the capabilities of a piece of software
Each line of code should be executed at least once One test case should be constructed from each specified requirement

It is necessary to use testing techniques that narrow down the number of test cases allowing the broadest testing coverage with the least effort
11 Black Box Testing

Technique: Functional Analysis

Analyze the expected behavior of the system according to its functional specification Generate a test procedure for each of the possible usage scenarios
Corresponds to use case scenarios Analyze how a change in one part of the system affects other parts Grand tour test cases: the result of one test case produces the data that is the input to the next test case
12 Black Box Testing

Example: Functional Analysis I

Use Case: Remove Traffic Violation
1. 2. 3. 4. 5. 6. Supervisor calls for deletion of the chosen Traffic Violation TVRS prompts Supervisor for confirmation Supervisor confirms TVRS requests OffendersDB to delete the Traffic Violation from the offenders record OffendersDB approves that the Traffic Violation has been deleted TVRS allows Supervisor to look up a new Traffic Violation as described in the Lookup Traffic Violation UC
Black Box Testing

13

Example: Functional Analysis II

Test case ID: 134543 Pre-conditions: 1. TVRS initialized with test1.db database 2. Violation 243567 displayed in the Lookup Violation dialog

Related use cases: Lookup Traffic Violation, Remove Traffic Violation

Action Expected result

Press Delete button

Press the Yes button Enter 243567 at Violation ID text field and press the Search button

Confirmation dialog is displayed

Lookup Violation dialog is displayed A message dialog stating that violation 243567 is not stored in TVRS

Test results

Passed Failed

Actual results:

Defect diagnosis:
14 Black Box Testing

Example: Functional Analysis III

How do we know that violation 243567 is stored in the system?
In addition, a query could be run on the Offenders database

Verify effects of change Filled when the test case is executed

15

Can a tester diagnose the cause of a defect?

Black Box Testing

Technique: Equivalence Partitioning

Identifies ranges of input and initial conditions that are expected to produce the same result A group of test cases form an equivalence class if:
They test the same feature/scenario If one test reveals a fault, the other ones (probably) will too If a test does not reveal a fault, the other ones (probably) will not either

It is adequate to use only a single representative of the equivalence class

16 Black Box Testing

Example: Equivalence Partitioning I

Input value specification for Lookup Violation form:
Field name
Violation ID Offenders first name Offenders last name

Valid values
[0-9]{0, 9} [a-zA-Z]{0, 10} [a-zA-Z]{0, 10}

17

Black Box Testing

Example: Equivalence Partitioning II

Field Valid equivalent classes
Known violation Unknown violation Empty Offenders first name Unknown violation Single known violation Many known violations Empty 18

Valid representative values

00243567 32456720 David Rachel Dan Black Box Testing

Invalid equivalent classes

ID < 0 or ID > 999999999 Non numeric ID

Invalid representative values

-1, 1234567890 23ab@

Violation ID

Character# > 10 Invalid character

Hasalongname ad0@am

Example: Equivalence Partitioning III

The number of test cases to choose from is reduced to (3 + 2) (4 + 2) (4 + 2) = 180 The actual number can be further limited

19

Single invalid field per test case (3 4 4 + 6 = 54) Importance of use case Resources available Most frequent input Life-critical software Infeasible test cases Randomly ...
Black Box Testing

Technique: Boundary Value Analysis

Based on experience / heuristics
Testing boundary conditions of equivalence classes is more effective

Choose input boundary values as equivalence classes representatives Choose inputs that invoke output boundary values Examples:
(0, 10] validate using 0, 1, 2, 9, 10, 11 Read up to 5 elements validate reading 0, 1, 4, 5, 6 elements
20 Black Box Testing

BVA as an equivalence partitioning extension

Choose one (or more) arbitrary value(s) in each equivalence class Choose valid values exactly on lower and upper boundaries of equivalence class Choose invalid values immediately below and above each boundary (if applicable)

21

Black Box Testing

General purpose test-suite construction technique

May be used to obtain reasonable coverage with little effort
Use cautiously! Unsuitable when values of different fields are related
1. While test cases can be added Each new test case should include as many un-included valid non-boundary equivalence class representatives as possible While test cases can be added Each new test case should include as many un-included valid boundary equivalence class representatives as possible While test cases can be added Each new test case should include a single invalid equivalence class representative that has not been included before Manually replace/remove redundant or infeasible test-cases
Black Box Testing

2.

3.

4.
22

Example: Country Club I

Specification
Day
Guest status Age (years) [0, 16) [16, 60)

Sunday - Thursday
Visitor Member Student

Friday - Saturday
Visitor Member Student

Admission fee 25 50 10 25 15 20 45 30
Black Box Testing

35 70 50

10 25 15

30 65 45

[60, 120] 35
23

Example: Country Club II

Field Valid equivalent classes Sun - Thu
Fri - Sat

Valid representative values Mon, Sun, Thu

Fri, Sat

Invalid equivalent classes

Invalid representative values

Day

Guest status

Visitor
Member Student

Visitor
Member Student 2, 0, 15

A combo box is used for choosing the day and guest status

Age

[0, 16)

Non-numeric value

14@a

[16, 60)
24

34, 16, 59
100, 60, 120
Black Box Testing

[60, 120]

Age < 0 or Age > 120

-1, 121

Example: Country Club III

Test case ID 1 2 3 4 5 6 7 8 9 10 11 Day Mon Fri Mon Sun Sat Thu Sun Sat Thu Mon Fri Guest status Visitor Member Student Visitor Member Student Member Student Visitor Member Student Age 2 34 100 0 16 60 15 120 59 14@a -1 Result 25 25 30 25 10 30 10 45 50 Invalid age Invalid age

valid

valid (boundary)

invalid

12
25

Fri

Visitor
Black Box Testing

121

Invalid age