Software Testing

COMP3400 Software Engineering

Software Testing Strategies

The software team should conduct effective formal technical reviews. By doing this, many errors will be eliminated before testing commences. Testing begins at the component level and works outward toward the integration of the entire computerbased system.

Software Testing Strategies

Different testing techniques are appropriate at different points in time. Testing is conducted by the developer of the software and (for large projects) an independent test group.

Software Testing Strategies

Testing and debugging are different activities, but debugging must be accomodated in any testing strategy.

Verification and validation

Verification refers to the set of activities that ensure that software correctly implements a specific function. Validation refers to a different set of activities that ensure that the software that has been built is traceable to customer requirements.

Software Testing for Conventional Software Architectures

Unit testing testing of each component of the software as implemented in source code. Integration testing testing on the design and the construction of the software architecture.

Validation testing requirements established as part of software requirements analysis are validated against the software that has been constructed. System testing software and other system elements are tested as a whole.

Unit Testing

Focuses on the internal processing logic and data structures within boundaries of a component. The testing can be conducted in parallel for multiple components.

Unit testing considerations

The module interface is tested to ensure that information properly flows into and out of the program unit under test. Local data structures are examined to ensure that the data stored temporarily maintains its integrity during execution.

All indepedent paths through a control structure are exercised to ensure that all statements in the module are executed at least once. Boundary conditions are tested to ensure that the module operates properly at boundaries established to restrict or limit processing. All error handling paths are tested.

Common errors in computation

Misunderstood or incorrect arithmetic precedence. Mixed mode operations. Incorrect initialization. Precision inaccuracy. Incorrect symbolic representation of an expression.

Comparison and control flow errors

Comparison of different data types. Incorrect logical operators or precedence. Incorrect comparison of variables. Improper or nonexistent loop termination. Improperly modified loop variables.

Error handling errors

Error description is unintelligible. Error noted does not correspond to error encountered. Error condition causes operating system intervention prior to error handling. Error description does not provide enough information to assist in the location of the cause of the error.

Unit test procedures

Design of unit tests can be performed before coding begins or after source code has been generated. Driver and/or stub software must be developed for each unit test.

Driver is a main program. Stub is a dummy program and uses the subordinate modules interface, may do minimal data manipulation, provides verification of entry and returns control to the module undergoing testing.

Integration Testing

Integration testing is a systematic tecnique for constructing software architecture while at the same time conducting tests to uncover errors associated with interfacing. Common error of integration is to combine all components and test the entire program as a whole. (Big-bang approach) Incremental integration is the correct aproach for testing.

Top-down integration

Modules are integrated by moving downward through the control hierarchy.

Begins with the main control module. Subordinated modules are incorporated in either a depth-first or breadth-first manner.

Top-down integration
M1 is the main module. M1

M2

M3

M4

M5

M6

M7

M8

In a depth-first manner the modules are integrated as branches. First, M2 and M5 are integrated. Then, M8 or M6 is integrated. The selection of the branch is aribitrary.

In a breadth-first manner the modules are integrated by levels. First, M2, M3 and M4 are integrated. Then, M5, M6 and M7 and finally, M8 is integrated.

Bottom-up integration

Begins construction and testing with atomic modules. Stubs are eliminated. Bottom-up integration strategy:

Combine low-level components into clusters. Construct a driver to test the cluster. Removes driver and combine clusters moving upward in the program structure.

Bottom-up integration
Mc Ma Mb

D1

D2

D3

Cluster 3 Cluster 2 Cluster 1

Validation testing

Begins at the culmination of integration testing. Validation succeeds when software functions in a manner that can be reasonably expected by the user.

Reasonable expectations are defined in the document Software Requirements Specification.

Validation testing

After a validation test case has been conducted one of two possible conditions exist:
The performance conforms to specification and is accepted. A deviation from specification is uncovered and a deficiency list is created.

The deviation can be rarely corrected prior system scheduled delivery. Negotiation with the customer is needed to resolve deficiencies.

Alpha and Beta Testing

Alpha test
Test conducted at the developers site by end-users. Developer is looking over the shoulder of typical users and records errors and usage problems. Tests are conducted in a controlled environment.

Alpha and Beta Testing

Beta Test
Test conducted at end-user sites. The developer is generally not present. The test is a live application of the software in an environment that cannot be controlled by the developer. The end-user records all problems and reports these to the developer.

System testing

System testing is a series of different tests whose primary purpose is to fully exercise the computer-based system.
Recovery testing Security testing Stress testing Performance testing

Recovery testing

A system must be fault tolerant, that is, processing faults must not cause overall system to cease. The system must recover from faults and resume processing within a prespecified time. Recovery testing is a test that forces the software to fail.

Recovery testing

The test verifies that recovery is properly performed. If recovery is automatic (performed by the system itself), reinitialization, checkpoint mechanisms, data recovery, and restart are evaluated for correctness.

Recovery testing

If recovery requires human intervention, the mean-time-torepair (MTTR) is evaluated to determine whether it is within acceptable limits.

Security testing

Verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration. The tester plays the role(s) of the individual who desires to penetrate the system. The role of the system designer is to make penetration cost more than the value of the information that will be obtain.

Security Testing

The tester may attempt to:

acquire passwords, attack the system with custom software, overwhelm the system, thereby denying services to others, purposely cause system errors, browse through insecure data.

Stress testing

Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume. The tester attempts to overwhelm the program.

Performance testing

Performance tests are often coupled with stress testing and usually require both hardware and software instrumentation. External instrumentation can monitor execution intervals, log events as they occur, and sample machine states on a regular basis.

Performance testing

By instrumenting a system, the tester can uncover situations that lead to degradation and possible system failure.