8th Chapter Testing Software Developed by Contractors

COTS - Commercial off the shelf

Two differences between in-house and COTS:

1. S/W is developed by non employees of the organization

2. Loss of Control over reallocation of resources (resources cannot be moved from
one project to another project)

Challenges or Difficulties in testing COTS:-

 Task or items missing

 S/W fails to perform
 Extra Features
 Does not meet business needs
 Does not meet people needs
 Does not meet operational needs

Differences in testing In-house s/w and Cots:-

Quality factors not specified – Factors like reliability and ease to use not include in
contractual criteria

Non-Testable requirements and criteria

Customer standards may not be met if the contract does not specify the operational
standards and document standards.

Missing Requirement

Overlooked changes in standards, technology – introduction of new technology into

contract, then cost to modify the s/w will be high

Training and development may be difficult

IF Contractors are in another country (offshore)

 Cultural differences

 Languages may be different

 Employee moral and support (of the organization giving contract)

 Root cause of the contractor IT organization not addressed(

Page 1 of 6 sunilanathan@yahoo.com
8th Chapter Testing Software Developed by Contractors

S/W Testers Responsibility for s/w Developed by a contractor

 Contracting organization is responsible for Quality of Organization.

 Process needed to monitor the development and validate correct functioning of
s/w developed by contractor.

Test Process for COTS:

1) Completeness of needs specification: - Adequately define your needs; define in

terms of two categories of o/p

1. Output products and reports: - Output products and reports are specific
documents

2. Management decision information: - Define the information needed for

decision-making purposes. All you need is information.

2) Define critical success factor

CSFs are those criteria or factors that must be present in the acquired s/w
The needs define what we are what we are looking for and CSF factors tells us how
we will evaluate that product after we get it.
Use the example of Automobile

3) Compatibility with your computer environment

Simple matching between your processing capabilities (s/w and hardware) and the
requirements needs to run the s/w

Hardware compatibility:- (Hardware vendor, main storage,Disk storage capacity and

identifier, type of printer, Maximum display size, keyboard)

Software compatibility (O/S):Name(unix, windows), Version of O/S

Compatibility with other software package and

Compatibility with available data: Like special files etc, verify if data can be read and
used by other s/w.

4) S/W can be integrated into your business system work flow

This step is to determine whether you can plug the COTs into your existing manual
system without disturbing operation.
Manual system is based on certain set of assumptions

Page 2 of 6 sunilanathan@yahoo.com
8th Chapter Testing Software Developed by Contractors

5) Demonstrate the s/w in operation

Observing the functioning of s/w is like taking an automobile for a test drive.
Two ways to perform it:-
 Computer-store controlled demonstration
 Customer site demonstration

Aspects that should be looked during Demonstration

 Understandability – Ease at which the operating process can be learned

 Clarity of Communication – Communication between man and machine
 Ease of Instruction manual – Steps to use the process should be clear in the
manual to follow
 Functionality of S/w – Are these functions that sales person described, fun.
that we expected and described are in Manual
 Knowledge to execute – Can a lesser-skilled person operate with minimal
training?
 Effectiveness of Help-Online help etc
 Evaluate program compatibility- Can it interact with other software and
check how data is passed
 Data compatibility – Determine if it is easy to use existing business data with
new s/w
 Smell test – Attempt to get a feel for what is happening and how that might
impact business

6) Evaluate the people fit

Objective of this step is determine whether employees can use the s/w and train the
employees if necessary
Results of this test are
 S/w can be used as it is
 Additional training & support required
 S/w is not usable with the skill set of proposed users.

Tester responsibility (Best practices test process for COTS)

1) Assure the process for contracting s/w is adequate:

Processes are subject to variation unless a process for contracting is followed. Contracts
are legal documents.
Following information should be included in all contracts:

 What is done – Deliverables specified

 Who does it – Responsibilities of both parties spelled out?
 When is it done- Dates
 How is it done- Specify the methods (deliverables)

Page 3 of 6 sunilanathan@yahoo.com
8th Chapter Testing Software Developed by Contractors

 Where is it done- Location

 Penalties for nonperformance

Concerns needed to be addressed:-

 Warranty – Guarantees provided by contractor

 Deliverables –
 Delivery date
 Commencement date
 Installation
 Updates
 Contractor Support – Location, quantity and type of support.
 Costs- Amount to be paid and payment terms
 Foreign relationships – systems interconnected with other s/w from different
countries
 Penalties
 Life of Contract
 Modification capability
 Service discontinuance- customers rights on discontinue of service
 Mannual/training discontinuance
 Acceptance test criteria – specify the criteria which determine the product is
acceptable (includes system, documentation and training material)
 Purchase versus lease
 Fairness of contract
 Performance of maintenance- Timing of the performance and location desired of
maintenance should be specified in the contract
 Transportability – Rights of the customer to move the application form location to
location
 Termination – Methods of terminating the contract to be specified
 Contractor employee rights – To visit the premises of the customer to perform
service on the application system
 Governing law
 Contractor inspection
 Access to developers and development documentation
 Requirements and contract criteria are testable – Performance based contracting

2) Assure Requirements and contract criteria are testable

The s/w requirements in contract, be testable if it is not testable the contractor has
much discretion on implemented requirement.
It is Easy to demonstrate a testable requirement when it fails to meet the contract.

Page 4 of 6 sunilanathan@yahoo.com
8th Chapter Testing Software Developed by Contractors

3) Review the Adequacy of the Contractors Test Plan:

Evaluate the contractor’s ability to deliver the s/w that meets the requirements
(example if the test plan states defect removal efficiency at 95% at requirement stage
then it is understood that 5% will move to design). Also evaluate the adequacy and
completeness of testing to be performed.

4) Assure Development is Effective and efficient

Verification during development needs access to developers documentation and
ability to interact with developer to identify defects.

5) Perform Acceptance Testing

S/W should not be placed into production without some acceptance testing. Extend
of acceptance testing will depend on the risk associated with use of S/W.

Minimum Acceptance testing should validate:

 Documentation is consistent with s/w execution

 Documentation is understandable
 User will be adequately trained prior use of s/w
 Is operable with the operational constrains of the organization

6) Issue a Report on the adequacy of the s/w to meet the needs of organization
Report should two main issues 1) Contractual requirements 2) Needs of organization
The report should focus on two definitions
 Meets requirements
 Fit for use
Report is normally prepared after acceptance testing.

7) Ensure knowledge transfer occurs and Intellectual property rights are protected
 There should be adequate knowledge transfer from developer to the acquirer.
 Intellectual property rights of both parties protected.

The contracting organization does not have the right to reproduce or distribute the
software without additional compensation to the contractor.

8) Incorporate Copyrighted material into the contractor’s manuals:

Contracting and contractor (example developer materials prior to acquiring) may
have intellectual property that they want protected. These should be covered in a
nondisclosure agreement.

9) Assure the ongoing Operation and Maintenance of the contracted s/w

Major concerns during the Operation and Maintenance are

 Adequacy of Control – Data processed by application is accurate, complete
and authorized. And provide processing integrity.

Page 5 of 6 sunilanathan@yahoo.com
8th Chapter Testing Software Developed by Contractors

 Adequacy of documentation -
 Speed of Service –
 Nearness of Service
 Competency of Service
 Cost of Service – Specify the terms of cost
 Adequacy of hardware
 Skilled personnel – Customer personnel to be trained to operate the
application
 Multi-contractor problem resolution – If Application provided by more than
one contractor then procedure to resolve a problem primarily.
 Cost of Operations – Cost associated with usage measurable
 Error diagnosis – Responsibility to diagnose error
 Error documentation – Procedure to specify the type of documentation
collected at times of error

10) Assure the Effectiveness of Contractual Relations

Is an ongoing relationship, therefore special needs and interests of customer are
obliged.

Concerns that arise in maintaining relationship

 Contractor Obligations met
 Customer Obligations met
 Needs met
 Limits on Cost increases
 Customizing options or added features
 Renegotiation
 Compensation for Error
 Returns on termination

Page 6 of 6 sunilanathan@yahoo.com