Testing COTS-based Applications

STC 2003, Salt Lake City, UT

Randall W. Rice, CSQA, CSTE Rice Consulting Solutions, LLC Oklahoma City, OK www.riceconsulting.com
1
2003, Rice Consulting Solutions, LLC

RCS

What is COTS?
COTS stands for Commercial Off-theshelf applications. Available from commercial vendors Sometimes called vendor software

RCS

Variations of COTS COTSbased, GOTS

COTS-based applications may be delivered by integrating two or more products. GOTS - Government Off-the-shelf applications
typically developed by the technical staff of the government agency for which it is created. It is sometimes developed by an external entity, but with funding and specification from the agency.
3

RCS

MOTS
Modified or modifiable off-the-shelf, or military off-the-shelf Typically a COTS product whose source code can be modified. The product may be customized by the purchaser, by the vendor, or by another party to meet the requirements of the customer.
4

RCS

MOTS
In the military context, MOTS refers to an off-theshelf product that is developed or customized by a commercial vendor to respond to specific military requirements. Because a MOTS product is adapted for a specific purpose, it can be purchased and used immediately. Since MOTS software specifications are written by external sources, government agencies are sometimes leery of these products, because they fear that future changes to the product will not be in their control.
5

RCS

What is the Purpose of COTS?

To reduce risks of internal software development To reduce the costs of internal software development To increase the speed and reliability of delivering applications

RCS

What are the Challenges of COTS Applications?

Selecting the right package Dealing with vendor licensing issues Integrating with existing and future applications Testing from the external perspective
7

RCS

Key Point #1 Buying the Right Product is Foundational.

No matter how much customization and testing is performed, if the COTS product fails to meet your needs at the outset, the project will fail.
8

RCS

What are the Challenges of Testing COTS Applications?

Unknown structure Unknown functional requirements Requires an external, black-box approach Release schedules Technology issues Test tool issues
9

RCS

What are the Risks of Implementing COTS Applications?

10

Functional problems Security issues Compatibility issues Integration and interoperability issues Vendor issues Procurement and licensing issues Testing issues
RCS

A COTS Testing Framework

Marketplace Verify VerifyNeeds Needs& & Research Research Products Products
Statement of Reqs. Evaluation Criteria Eval Scorecards Acceptance Criteria Verify: User needs Technical needs Mission needs Research: Candidate products and vendors

Qualified

Adapted

Assembled

Evolving

Evaluate Evaluate Products Products& & Plan PlanTests Tests

Graded Scorecards Test objectives Test Plans Evaluate: Products Functional fit Technical fit Mission fit Quality levels Performance levels

Verify Verify& &Validate ValidateFunctionality Functionality and andInterfaces Interfaces

Verified Plans Test Results Verify: Integration plans Test plans Validate: Implementation Integration Compatibility Verified Plans Test Results Verify: Test plans Verified Plans Regression Cases Test Results Verify: Test plans Release and upgrade plans Validate: Integration Interoperability Compatibility Regression

11

Validate: Integration Interoperability Middleware

RCS

Roles and Responsibilities for COTS Testing

Vendors to provide and support a quality product Customers to select the right product and validate Senior Management to guide the acquisition and testing efforts Users to provide quality requirements and to participate in acceptance testing
12

RCS

Roles and Responsibilities for COTS Testing

In-house Developers to assist in product integration Test Team Leadership to plan tests and lead the testing process Testers to understand tests and perform them correctly QA Analysts and Leaders to gather data from the acquisition, integration and testing processes for process improvement

13

RCS

Test Terminology
Verification
All QC activities throughout the life cycle that ensure interim deliverables meet specific specifications.

Validation
The test phase of the life cycle which ensures that the end product (e.g..., software or system) meets user needs.
14

RCS

Two Other Views of Quality

Producer View
Did we build the system or application according to specifications? Does it work correctly?

Customer View

15

Does the product work correctly for me? Does the product meet my requirements? Does it work in my environment? Does it work with other things I need to use? Is it easy to use? Is it a good value? RCS

Its possible for a product to meet producer specifications and miss customer expectations.

16

RCS

Key Point #2 Testing COTS is a Black-box Process.

Requirements and specifications are great targets for testing, but for COTS products, testing becomes more focused toward validating that the products meet operational or mission needs.
17

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)

Implement (Verification)

Evaluate (Verification)

Acquire (Validate)
18

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)
Addresses: User needs Mission needs Evaluate Technology fit (Verification) - Integration - Interoperability - Compatibility Constraints - Environment Acquire - User (Validate) - Project

Implement (Verification)

19

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)
- Needs - Constraints - Acceptance Criteria

Implement (Verification)

Evaluate (Verification)

Acquire (Validate)
20

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)
Addresses: Product Features Project Risks Implement Product Risks (Verification) Product Trade-Offs Vendor Information Test Planning

- Needs - Constraints - Acceptance Criteria

Evaluate (Verification)

Acquire (Validate)
21

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)
- Needs - Constraints - Acceptance Criteria

Implement (Verification)

Evaluate (Verification)
- COTS Product - Test Strategy - Test Plan - Acceptance Criteria

Acquire (Validate)
22

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)
- Needs - Constraints - Acceptance Criteria

Implement Addresses: Evaluate Product Selection (Verification) Vendor Contracting (Verification)

Integration Testing (Functional) Testing (Behavioral)

Acquire (Validate)
23

- COTS Product - Test Strategy - Test Plan - Acceptance Criteria

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
Specify (Verification)
- Needs - Constraints - Acceptance Criteria

Implement (Verification)

Evaluate (Verification)
- COTS Product - Test Strategy - Test Plan - Acceptance Criteria

- Tested COTS Product - Baseline - Test Evaluation

24

Acquire (Validate)

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
- Post-Implementation Evaluation - Functional Baseline

Specify (Verification)

- Needs - Constraints - Acceptance Criteria

Implement (Verification)

Addresses: Deployment Training Support Post-Implementation Reviews

Evaluate (Verification)
- COTS Product - Test Strategy - Test Plan - Acceptance Criteria

- Tested COTS Product - Baseline - Test Evaluation

25

Acquire (Validate)

RCS

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
- Post-Implementation Evaluation - Functional Baseline

Specify (Verification)

- Needs - Constraints - Acceptance Criteria

Implement (Verification)
- Tested COTS Product - Functional Baseline - Test Evaluation
26

Evaluate (Verification)
- COTS Product - Test Strategy - Test Plan - Acceptance Criteria

Acquire (Validate)

RCS

The Evolutionary Nature of the COTS Lifecycle and How it Impacts Testing
The framework is:
Cyclical and repeatable
New releases will have new or removed features Test plans will change

Evolving
Technical and user requirements will evolve.

Integrated
Products must work with other products. Test plans and environments must include multiple applications and interfaces.

Dependent on multiple sources of involvement for success

COTS testing is not just one teams responsibility.
27

RCS

Key Point #3 Integration and Interoperability Testing Validates the Glue that Holds Related Products Together.
The vendor cant test integration and operability for you. These kinds of tests are the customers responsibility. Failure to test integration and interoperability is to place the project at great risk of failure.

28

RCS

Summary
Testing COTS-based applications is a challenge, but with careful planning and effective testing strategies tailored for COTS, the project risks can be reduced.
29

RCS

Bio - Randall W. Rice

Over 25 years experience in building and testing information systems in a variety of industries and technical environments Certified Quality Analyst Certified Software Test Engineer Conference chair 1995 - 2002, QAIs annual software testing conference Co-author with William E.Perry, Surviving the Top Ten Challenges of Software Testing and forthcoming book, Testing Dirty Systems (2003).
30

RCS

Contact Information
Randall W. Rice, CSQA, CSTE Rice Consulting Solutions, LLC P.O. Box 891284 Oklahoma City, OK 73189 Ph: 405-793-7449 Fax: 405-793-7454 Web site: www.riceconsulting.com e-mail: rrice@riceconsulting.com
31

RCS