Standard Test Approach

STANDARD TEST APPROACH
Version: 3.0
XXXX Software Development Projects
Prepared for
XXXX (XXXX)
Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

Standard Test Approach January 2011
Version: DRAFT 3 Page 1 of 41
Revision History
Date Version Description Author

11/02/2010 1.0 Standard Approach Document Andre Jackson
1/25/2011 3.0 Test Plan/Approach Doc Merge Andre Jackson
Document Sign-Off
Title Signature
IT Director
Sr. App Engineer
Sr. Systems Analyst
App Engineer
QA Engineer

Table of Contents
1 INTRODUCTION................................................................................................ 6
2 QUALITY OBJECTIVES.....................................................................................7
2.1 TEST APPROACH OBJECTIVES....................................................................................................................7

2.2 HIGH-LEVEL PRODUCT DEVELOPMENT OBJECTIVE....................................................................................8
2.2.1 PRIMARY OBJECTIVE..............................................................................................................................8
2.2.2 SECONDARY OBJECTIVE..........................................................................................................................9
2.3 SCOPE OF APPLICATIONS UNDER TEST.....................................................................................................9
3 TEST METHODOLOGY......................................................................................9
3.1 TEST STAGES.........................................................................................................................................9

Overview.................................................................................................................................................................10
Milestone 1 - Planning Phase.................................................................................................................................10
Milestone 2 - Design Phase.....................................................................................................................................10
Milestone 2a - Usability Testing.............................................................................................................................11
Milestone 3 - Developing Phase.............................................................................................................................11
Milestone 3a - Unit Testing (Multiple)....................................................................................................................11
Milestone 3b - Acceptance into Internal Release Testing......................................................................................12
Milestone 3c - Internal Release Testing..................................................................................................................12
Milestone 3d - Acceptance into Alpha Testing.......................................................................................................12
Milestone 3e - Alpha Testing..................................................................................................................................13
Milestone 4 - Stabilization Phase...........................................................................................................................13
Milestone 4a - Acceptance into Beta Testing.........................................................................................................13
Milestone 4b - Beta Testing/Business Acceptance Testing (BAT)..........................................................................13
Milestone 4c - Release to Production.....................................................................................................................14
Milestone 4d - Post Release....................................................................................................................................14
3.2 TEST LEVELS..........................................................................................................................................14
3.2.1 Build Tests......................................................................................................................................................15
3.2.2 Milestone Tests..............................................................................................................................................15
3.2.3 Release Tests.................................................................................................................................................15
3.3 BUG REGRESSION.................................................................................................................................16
3.4 BUG TRIAGE........................................................................................................................................16
3.5 SUSPENSION CRITERIA AND RESUMPTION REQUIREMENTS.......................................................................16
3.6 TEST COMPLETENESS...........................................................................................................................17
3.6.1 Standard Conditions......................................................................................................................................17
3.6.2 Bug Reporting & Triage Conditions...............................................................................................................17
4 SOFTWARE RISK ISSUES...............................................................................18
4.1 SCHEDULE..............................................................................................................................................18
4.2 TECHNICAL..........................................................................................................................................18
4.3 MANAGEMENT.....................................................................................................................................18
4.4 PERSONNEL.........................................................................................................................................18
4.5 REQUIREMENTS...................................................................................................................................18
5 TEST APPROACH............................................................................................ 19

5.1 SPECIAL TESTING TOOLS.........................................................................................................................19
5.2 SPECIAL TRAINING ON TESTING TOOLS..................................................................................................19
5.3 TEST METRICS.....................................................................................................................................20
5.4 CONFIGURATION MANAGEMENT............................................................................................................20
5.5 REGRESSION TESTING..........................................................................................................................21
5.6 REQUIREMENTS MANAGEMENT.............................................................................................................21
6 TEST STRATEGY............................................................................................ 22
6.1 SYSTEM TEST......................................................................................................................................22

6.2 PERFORMANCE TEST............................................................................................................................22
6.3 SECURITY TEST......................................................................................................................................22
6.4 AUTOMATED TEST...............................................................................................................................22
6.5 STRESS AND VOLUME TEST...................................................................................................................22
6.6 RECOVERY TEST..................................................................................................................................22
6.7 DOCUMENTATION TEST........................................................................................................................22
6.8 BETA TEST..........................................................................................................................................23
6.9 BUSINESS ACCEPTANCE TEST (BAT).....................................................................................................23
7 ENTRY AND EXIT CRITERIA...........................................................................24
7.1 TEST PLAN..........................................................................................................................................24

7.1.1 Test Plan Entry Criteria................................................................................................................................24
7.2.2 Test Cycle Exit Criteria..................................................................................................................................24
8 TEST DELIVERABLES.....................................................................................25
8.1 DELIVERABLES MATRIX..........................................................................................................................27

8.2 DOCUMENTS........................................................................................................................................27
8.2.1 Test Approach Document.............................................................................................................................27
8.2.2 Test Schedule.................................................................................................................................................28
8.2.3 Test Specifications........................................................................................................................................28
8.3 TEST CASE/BUG WRITE-UPS................................................................................................................28
8.3.1 Test Manager Test Cases..............................................................................................................................28
8.3.2 Test Case Coverage Reports..........................................................................................................................28
8.3.3 Bug Tracking System and Regression Results.............................................................................................29
8.3.4 TFS Bug Tracking Reports............................................................................................................................29
8.4 REPORTS.............................................................................................................................................29
8.4.1 Weekly Status Reports..................................................................................................................................29
8.4.2 Phase Completion Reports............................................................................................................................29
8.4.3 Test Final Report - Sign-Off..........................................................................................................................30
8.4.4 TFS Source Control Archive..........................................................................................................................30
9 RESOURCE & ENVIRONMENT NEEDS.............................................................31
9.1 BASE SYSTEM HARDWARE......................................................................................................................31

9.2 BASE SOFTWARE ELEMENTS IN THE TEST ENVIRONMENT.......................................................................31
9.3 PRODUCTIVITY AND SUPPORT TOOLS.....................................................................................................31
9.4 TESTING TOOLS...................................................................................................................................31
9.4.1 Tracking Tools................................................................................................................................................31
9.4.1.1 Team Foundation Server...........................................................................................................................31
9.4.1.2 MS Test Manager 2010..............................................................................................................................32
9.4.1.3 Configuration Management......................................................................................................................32
9.4.1.4 Issues Database..........................................................................................................................................32
9.4.2 Diagnostic Tools.............................................................................................................................................32
9.4.2.1 ExamDiff Pro..............................................................................................................................................32
9.4.3 Automation Tools..........................................................................................................................................32
9.4.3.1 Microsoft Visual Studio Ultimate/Test Professional 2010.......................................................................32
9.5 TEST ENVIRONMENT..............................................................................................................................33
9.5.1 Hardware.......................................................................................................................................................33
9.5.2 Software.........................................................................................................................................................33
10 RESPONSIBILITIES, STAFFING AND TRAINING NEEDS...................................34
10.1 PEOPLE AND ROLES.............................................................................................................................34

10.2 STAFFING AND TRAINING NEEDS...........................................................................................................35
11 TEST MANAGER TEST CASES.........................................................................36
11.1 TEST CASES.........................................................................................................................................36
11.2 TEST CASE CONTENTS..........................................................................................................................36
 Test Case Title..................................................................................................................................................36
 TCID.................................................................................................................................................................36
 Status...............................................................................................................................................................36
 Classification....................................................................................................................................................36
 Steps................................................................................................................................................................37
 Summary.........................................................................................................................................................37
 Tested User Stories..........................................................................................................................................37
 All Links............................................................................................................................................................37
 Attachments....................................................................................................................................................37
 Associated Automation...................................................................................................................................37
 Parameter Values............................................................................................................................................37
12 BUG MANAGEMENT........................................................................................38
12.1 BUG DOCUMENTATION............................................................................................................................38

12.1.1 Bug Severity and Priority Definition............................................................................................................38
12.1.1.1 Severity List...............................................................................................................................................38
12.1.1.2 Priority List................................................................................................................................................39
12.2 TEST RUNNER BUG ENTRY FIELDS........................................................................................................39
 Bug Title...........................................................................................................................................................39
 Status...............................................................................................................................................................39
 Classification....................................................................................................................................................39
 Planning...........................................................................................................................................................40
 Details..............................................................................................................................................................40
 System Info......................................................................................................................................................40
 Test Cases........................................................................................................................................................40
 All Links............................................................................................................................................................40
 Attachments....................................................................................................................................................40
12.3 BUG REPORTING PROCESS.......................................................................................................................41

13 DOCUMENTATION.......................................................................................... 42
1 Introduction
This document identifies the XXXX Software Quality Assurance Department’s methodology as
implemented across all projects. This test approach describes the high-level strategies and
methodologies used to plan, organize, and manage testing of software projects within XXXX. This
test approach also includes descriptions of XXXX Software Quality Assurance Department’s role at
various phases of the project development cycle. It also establishes the goals, processes, and
responsibilities required to implement effective quality assurance functions across all XXXX
software development and release projects.
The details outlined in this document provide the framework necessary to ensure a consistent
approach to software quality assurance throughout the project life cycle. It defines the approach
that will be used by the Quality Assurance (QA) personnel to monitor and assess software
development processes and products to provide objective insight into the maturity and quality of
the software. The systematic monitoring of the XXXX software products, processes, and services
will be evaluated to ensure they meet requirements and comply with XXXX policies, standards, and
procedures, as well as applicable Institute of Electrical and Electronic Engineers (IEEE) standards.
The overall purpose of this test approach strategy is to gather all of the information necessary to
plan and control the test effort for testing XXXX applications. It describes the approach to testing
the software, and will be the top-level plan used by testers to direct the test effort.
The approach is designed to create clear and precise documentation of the test methods and
processes that XXXX will use throughout the course of system verification testing.
The strategy covers SQA activities throughout the formulation and implementation phases of the
application mission. SQA activities will continue through operations and maintenance of the
system.
This documenting of the test methods and processes will serve as the basis for ensuring that all
major milestones and activities required for effective verification testing can efficiently and
successfully be accomplished. This plan may be modified and enhanced as required throughout all
future verification testing engagements.

NOTE: For the remainder of this document and for the sake of simplicity and consistency, the AFCE
Software Development Department will be simply referred to as “Development”, and XXXX Software
Quality Assurance Department will be referred to as “QA Testing”.
2 Quality Objectives
2.1 Test Approach Objectives
This Test Approach supports the following objectives:
 Outlines and defines the overall test approach that will be used;
 Identifies hardware, software, and tools to be used to support the testing efforts;
 Defines the types of tests to be performed;
 Defines the types of data required for effective testing;
 Defines the types of security threats and vulnerabilities against which each system will
be tested;
 Identifies and establishes traceability from the Requirements Matrix to test cases and
from test cases to the Requirements Matrix;
 Serves as a foundation for the development of Test Plans and Test Cases;
 Defines the process for recording and reporting test results;
 Defines the process for regression testing and closure of discrepancies;
 Identifies the items that should be targeted by the tests;
 Identifies the motivation for and ideas behind the test areas to be covered;
 Identifies the required resources and provides an estimate of the test efforts;
 List the deliverable elements of the test activities;
 Define the activities required to prepare for and conduct System, Beta and User
Acceptance testing;
 Communicate to all responsible parties the System Test strategy;
 Define deliverables and responsible parties;
 Communicate to all responsible parties the various Dependencies and Risks; and
 Scope.
2.2 High-Level Product Development Objective

The high-level objectives of this project are as follows:
 Update the software to a current technology platform while maintaining the

core functionality of the original product;
 Re-design the user interface to take advantage of current technology standards,
and to deliver a better user experience;
 Provide a method for delivering updates to the product electronically and
seamlessly to the user; and
 Allow internal non-technical personnel to manage the creation and maintenance
of new versions.
2.2.1 Primary Objective
The primary objective of testing our application systems is to:
 Identify and expose all issues and associated risks;
 Communicate all known issues to the project team; and
 Ensure that all issues are addressed in an appropriate matter before release.
As an objective, this requires careful and methodical testing of the application to first ensure all
areas of the system are scrutinized and, consequently, all issues (bugs) found are dealt with
appropriately.
2.2.2 Secondary Objective

A secondary objective of testing our application systems is to:
 Assure that the system meets the full requirements of our customer(s);
 Maintain the quality of the product; and
 Remain within the cost range established at the project outset.
At the end of the project development cycle, the user should find that the project has met or
exceeded all of their expectations as detailed in the requirements. Any changes, additions, or
deletions to the Requirements document, Functional Specification, or Design Specification will
be documented and tested at the highest level of quality allowed within the remaining time of
the project and within the ability of the test team.
2.3 Scope of Applications Under Test

The scope of this quality assurance effort is to validate the full range of activities related to the
functionality of all XXXX applications-under-test (AUT); as they undergoes development, re-
designing and re-building.
This test plan describes the unit, subsystem integration, and system level tests that will be
performed on components of the applications. It is assumed that prior to testing, each
subsystem to be tested will have undergone an informal peer review and only code that has
successfully passed a peer review will be tested.
Unit tests will be initially done by the software designer agency (i.e. Eureka, Avectra, etc.) and
subsequently by the XXXX Development Department; performing secondary unit testing,
boundary checking and basic black box testing.

3 Test Methodology
3.1 Test Stages
Overview
There are four major milestones in the Development Cycle: Planning Phase, Design Phase,
Development Phase, and Stabilization Phase. The Planning Phase culminates in the completion
of the Planning Docs Milestone (Requirements plus Functional Specs). The Design Phase
culminates in the completion of the Design Specs and Test Plan/Test Specs. The Development
Phase culminates in the Code Complete Milestone. The Stabilization Phase culminates in the
Release Milestone.
During the first two phases, QA Testing plays a supporting role, providing ideas and limited
testing of the planning and design documents. Throughout the final two stages, QA Testing
plays a key role in the project.
Milestone 1 - Planning Phase

During the first phase of the Development Cycle, QA Testing should focus upon the
Requirements (User Stories) and Functional Specs. QA Testing reviews these documents for
their comprehensibility, accuracy, and feasibility. Specific tasks that QA Testing may carry out
during this phase include:
 Assessing the impact of Requirements on testing;

 Providing metrics factors (preliminary schedule, estimated test case and bug counts,
etc.);
 Identifying project infrastructure issues; and
 Identifying risks related to the testing and development process.
Milestone 2 - Design Phase

During the second phase of the Development Cycle, QA Testing is focused upon evaluating the
design and is required to produce and distribute its draft Test Plan. To generate the Test Plan,
Test Spec, and Test Cases; QA Testing requires that the Requirements, Functional Spec, Design
Documents, Business Rules, and Project Plan/Schedule be completed and a copy emailed to the
test point person.
During this phase, QA Testing may participate within the design reviews (with Development)
and have access to the Design Spec under construction. This will help QA Testing to better
prepare its Test Plan, Test Spec, and Test Cases. The Test Plan defines much of the detailed
strategy and specific testing information that will be used for testing the application.
The purpose of the Test Plan is to achieve the following:
 Divide Design Spec into testable areas and sub-areas. This should be confused with
more detailed test specs. The plan will also identify and include areas that are to be
omitted (not tested);
 Define testing strategies for each area and sub-area;
 Define bug-tracking procedures;
 Define release and drop criteria;
 Define list of configurations for testing;
 Identify testing risks;
 Identify required resources and related information; and
 Provide a testing schedule.
Milestone 2a - Usability Testing

The purpose of Usability Testing is to ensure that the new components and features will
function in a manner that is acceptable to the customer. Development will typically create a
non-functioning prototype of the UI components to evaluate the proposed design. Usability
Testing can be coordinated by QA Testing, but actual testing must be performed by non-testers
(XXXX business unit members). QA Testing will review the findings and provide the project
team with its evaluation of the impact these changes will have on the testing process and to the
project as a whole.
Milestone 3 - Developing Phase

During the third phase of the development cycle, QA Testing begins to execute its primary role
by identifying issues (bugs). At the beginning of this phase, QA Testing will be spending most of
its time entering requirements (User Stories) and generating test cases (Work Items) in
Microsoft Visual Studio’s Test Professional 2010: Test Manager; which will be housed on the
Team Foundation Server. As this phase progresses; however, QA Testing will receive release
candidates (builds) from Development; increasing functionality to test. By the time the
development phase closes, QA Testing will be primarily executing test cases.
Development’s internal releases during this phase ultimately drive toward a static Alpha build
(recognized as code-complete). While working on the code for an interim milestone (builds
generated as features completed), QA Testing writes the test specification and test cases for that
feature set. During this phase, Development will also be conducting their Unit Testing (White
Box Testing) prior to every internal release to QA Testing.
Milestone 3a - Unit Testing (Multiple)

Unit testing is conducted by Eureka Software Solutions, Inc., Avectra, and some third-party
vendors; however the XXXX Development Department conducts informal Unit Testing to ensure
that proper functionality and code coverage have been achieved both during coding and in
preparation for acceptance into Alpha Testing. Involvement in Unit Testing by the QA Testing
Department during this phase should be in an advisory capacity only. It is the responsibility of
QA Testing to require that a set level of quality control is adopted and maintained by
Development throughout this phase.
The following areas of the project must be unit-tested and signed-off before being passed on to
QA Testing:
 Databases, Stored Procedures, Triggers, Tables, and Indexes;

 Database conversion; and
 .OCX, .DLL, .EXE and other binary formatted executables.
The exit criterion for this milestone is “code-complete”. That is, all functionality and logical and
physical components of the application must be completed and made available to QA Testing
according to the requirements within the drop criteria.
Milestone 3b - Acceptance into Internal Release Testing

Internal releases are issued builds received from Development containing new functionality
(and may include bug fixes). Before accepting a project for Internal Release Testing, QA Testing
must be assured that adequate Unit Testing has been done by Eureka, Avectra, some third-party
vendors and XXXX Development.
A build must pass the following Build Acceptance Test before moving into internal release
testing:
 Sign-off from the Senior Application Engineer/Architect that Unit Testing is

complete on all modules released;
 Verification that the code is located at a specified drop point;
 Verification that Release Notes accompany the new build (discussion of changes,
areas not to test, etc.);
 QA Testing can install the build from a SETUP.EXE file (i.e. CFEExamPrep.exe,
CFEExam.exe, etc.) located at the drop point; and
 Verification that all “Build Acceptance Test” test cases (preferably automated) pass
thereby indicating that the basic components work.
Milestone 3c - Internal Release Testing

Internal release testing is the evaluation of the new UI and functionality that has been
incorporated into the new build. Several cycles of internal release testing will occur (every time
QA Testing receives a new drop of a build). Completion of this phase will occur when code is
complete, and subsequent drops of new builds will include only bug fixes with no new feature
code.
Internal release testing will include:
 Execution of Test Manager test cases;

 Documentation into Test Manager of any variations from expected results; and
 Addition of newly discovered test cases into Test Manager.
Milestone 3d - Acceptance into Alpha Testing

Alpha Testing occurs after code complete has been achieved; all subsequent builds throughout
alpha will thus consist of bug fixes only. Initial receipt of the application defined as code
complete into Alpha Testing requires that all critical path test cases pass to ensure that general
aspects of the application are robust enough to undergo the testing process. The application
must be functionally complete. The drop of the application to QA Testing should meet the same
requirements as any drop criteria (see above).
Passing this milestone indicates that Alpha Testing is ready to commence. Failure into
acceptance requires that the drop be rejected back to Development. This would only occur in

only two instances: one, where the drop criteria had not been properly met; or two, when a bug
of sufficient severity prevents running test cases against the application.
Milestone 3e - Alpha Testing

During the repeated cycles of identifying bugs and taking receipt of new builds (containing bug
fix code changes), there are several processes which are common to this phase across all
projects. These include the various types of tests: functionality, performance, stress,
configuration, etc. There is also the process of communicating results from testing and
ensuring that new drops contain stable fixes (regression). The project should plan for a
minimum of 3-4 cycles of testing (drops of new builds) in this phase with 6-8 for occasions
where even more extreme testing is appropriate.
Throughout the course of Alpha Testing, a brief daily report should be submitted by the QA
Engineer to key senior management personnel indicating testing results to date. Also, frequent
and regular triage meetings shall be held with the project team (more frequent than in previous
phases of the development cycle). QA Testing shall present their bug findings as recorded
within the Test Manager and Team Foundation Server. The objective of the triage meeting is to
determine priorities for bug fixes.
Milestone 4 - Stabilization Phase

During the fourth and final stage of the Development Cycle, QA Testing performs most of the
work (relative to other groups). Here is where testing resource loading is at its peak. Upon
entering this phase, the application has been internally tested module by module, and has
undergone Alpha Testing cycles.
The objective of this phase is to arrive at the Release Milestone with a robust release candidate
(build). There is still an emphasis on testing to continue to identify issues (bugs) and
regression test the bug fixes. All project members may become involved in this process during
this phase.
Milestone 4a - Acceptance into Beta Testing

Upon entering this milestone, QA Testing will continue to provide a brief daily report indicating
testing results to date. When applicable, the report must show that to the best degree
achievable during the Alpha Testing phase, all identified severity 1 and severity 2 bugs have
been communicated and addressed. At a minimum, all priority 1 and priority 2 bugs should be
resolved prior to entering the beta phase.
Important deliverables required for acceptance into Beta Testing include:
 Application SETUP.EXE;
 Installation instructions; and
 All documentation (beta test scripts, manuals or training guides, etc.).
Milestone 4b - Beta Testing/Business Acceptance Testing (BAT)

This milestone process is typically conducted by the various XXXX business unit subject matter
experts (testers). QA Testing and the Sr. Systems Analyst participates in this milestone process
as well by providing confirmation feedback on new issues uncovered, and input based on
identical or similar issues detected earlier. The intention is to verify that the product is ready
for distribution, acceptance by the (business) customer and to iron out potential operational
issues. It is a resource intensive process in which the business unit testers cooperate with
Product Management to develop a focused Business Acceptance Test (BAT) Plan specifying the
objective, scope and duration of the Beta phase.
Throughout the beta test cycle, bug fixes will be focused on minor and trivial bugs (severity 3
and 4). QA Testing will continue its process of verifying the stability of the application through
Regression Testing (existing known bugs, as well as existing test cases). QA Testing will also
assist with confirmation feedback to beta test results (yes it is a bug; yes Development is
working on a fix, etc.).
The milestone target of this phase is to establish that the Application-Under-Test (AUT) has
reached a level of stability. The future web-based version of the product will insure that the
system is operating appropriately for its usage (transaction response times, HTTP hits per
second, throughput, number of simultaneous users, etc.), that it can be released to the client
users. BAT usually involves 1 – 2 weeks of focused testing for an average project and 5 – 8
weeks for a major version release.
Milestone 4c - Release to Production

Release to Production occurs only after the successful completion of the application-under-test
throughout all of the phases and milestones previously discussed above. The milestone target
is to place the release candidate (build) into production after it has been shown that the
application has reached a level of stability that meets or exceeds the client expectations as
defined in the Requirements, Functional Spec., and XXXX Production Standards (not yet
established).
QA Testing will ensure that the Final Release Candidate (RC) Set passes the following test cases:
 Check for extra or missing files (Diff operation on directory);

 Proper date stamps on all files (Diff operation on directory);
 Binary comparison of source files conducted;
 Installation test on clean machine one more time; and
 Basic functionality test (preferably automated smoke tests).
Milestone 4d - Post Release

During this phase, QA Testing archives all project documents, notes, test-ware (automation
scripts, etc.), email, source code, etc. Archiving is the responsibility of the QA Engineer. QA
Testing also prepares a post-implementation review document discussing the testing process
throughout the development cycle.
3.2 Test Levels

Testing of an application can be broken down into three primary categories and several sub-
levels. The three primary categories include tests conducted on every build (Build Tests), tests

conducted at every major milestone (Milestone Tests), and tests conducted at least once every
project release cycle (Release Tests). The test categories and test levels are defined below:
3.2.1 Build Tests
Level 1 - Build Acceptance Tests

Build Acceptance Tests should typically take approximately 1 week. These test cases simply
ensure that the application has been built and can be installed successfully. Other related test
cases ensure that QA Testing received the proper Development Build from the Sr. Systems
Analyst and was informed of any build related matters, including the location of the build drop
point. The objective is to determine if further testing is possible. If any Level 1 test case fails,
the build is returned to the developers un-tested.
Level 2 - Smoke Tests

Smoke Tests are preferably automated; however, manual testing will suffice and typically takes
less than 2 days. These tests cases verify the major functionality at high level. The objective is
to determine if further testing is possible. These test cases should emphasize breadth more
than depth. All components should be touched, and every major feature should be tested
briefly by the Smoke Test. If any Level 2 test case fails, the build is returned to the developers
un-tested.
Level 2a - Bug Regression Testing

Every bug that was marked as “Open” during the previous build, but marked as “Fixed, Needs
Re-Testing” for the current build under test; will need to be regressed, or re-tested. Once the
smoke test is completed, all resolved bugs need to be regressed. It should take between 5
minutes to 1 hour to regress most bugs.
3.2.2 Milestone Tests
Level 3 - Critical Path Tests

Critical Path test cases must pass by the end of every 3-5 Build Test Cycles. Critical Path test
cases are targeted on features and functionality that the user will see and use every day. They
do not need to be tested every drop, but must be tested at least once per milestone. Thus, the
Critical Path test cases must all be executed at least once during the Alpha cycle, and once
during the Beta cycle.
3.2.3 Release Tests
Level 4 - Standard Tests

Standard Test Cases need to be run at least once during the entire test cycle for this release.
These cases are run once, not repeated as are the test cases in previous levels. However,
Functional Testing and Detailed Design Testing can be tested multiple times for each Milestone
Test Cycle (alpha, beta, etc.). Standard test cases usually include Installation, Data, GUI, and
other test areas.

Level 5 - Suggested Test
Suggested Test Cases are those case which would be nice to execute, but may be omitted due to
time constraints.
Most Performance and Stress Test Cases are classic examples of Suggested Test Cases (although
some should be considered standard test cases). Other examples of Suggested Test Cases
include WAN, LAN, Network, and Load Testing.
3.3 Bug Regression

Bug Regression will be a central tenant throughout all testing phases. All bugs that are resolved
as “Fixed, Needs Re-Testing” will be regressed when QA Testing is notified of the new drop
containing the fixes. When a bug passes regression it will be considered “Closed, Fixed”. If a
bug fails regression, QA Testing will notify Development by entering bug notes into Test
Manager. When a Severity 1 bug fails regression, QA Testing should also put out an immediate
email to Development. The QA Engineer will be responsible for tracking and reporting to
Development and product management the status of Regression Testing.
It is recommended that a separate cycle of Regression Testing occur at the end of each phase to
confirm the resolution of Severity 1 and 2 bugs. The scope of this last cycle should be
determined by the test point person and product management.
3.4 Bug Triage

Bug Triages will be held throughout all phases of the development cycle. Bug triages will be the
responsibility of the QA Engineer. Triages will be held on a regular basis with the time frame
being determined by the bug find rate and project schedules. Thus, it would be typical to hold a
few triages during the Planning phase, then maybe one triage per week during the Design
phase, ramping up to twice per week during the latter stages of the Development phase. Then,
the Stabilization phase should see a substantial reduction in the number of new bugs found,
thus a few triages per week would be the maximum (to deal with status on existing bugs).
The QA Engineer, Sr. Systems Analyst, Sr. Application Engineer, Application Engineer, and IT
Director should all be involved in these triage meetings. The QA Engineer will provide required
documentation and reports on bugs for all attendees. The purpose of the triage is to determine
the type of resolution for each bug and to prioritize and determine a schedule for all “To Be
Fixed” bugs. QA Testing will then assign the bugs to Development for fixing and report the
resolution of each bug back into Test Manager/TFS. The QA Engineer will be responsible for
tracking and reporting on the status of all bug resolutions.
3.5 Suspension Criteria and Resumption Requirements

Testing will be suspended on the affected software module when Smoke Test (Level 1) or
Critical Path (Level 2) test case bugs are discovered. A bug report will be filed in Test
Manager/TFS and Development and Product Management will be notified. After fixing the bug,
Development will follow the drop criteria (described above) to provide its latest drop to QA

Testing. At that time, QA Testing will regress the bug, and if passes, continue testing the
module.
Notice that some discretion is in order here on the part of the QA Engineer. To suspend testing,
the bug must be reproducible, it must be clearly defined, and it must be significant.
3.6 Test Completeness

Testing will be considered complete when the following conditions have been met:
3.6.1 Standard Conditions

 When QA Testing, Development and Product Management agree that testing is
complete, the application is stable, and all parties agree that the application meets
functional requirements;
 Script execution of all test cases, in all areas, have passed;
 Automated test cases have passed in all areas;
 All priority 1 and 2 bugs have been resolved and closed;
 Each test area has been signed off as completed by the QA Engineer;
 50% of all resolved severity 1 and 2 bugs have been successfully re-regressed as
final validation; and
 Ad hoc testing in all areas has been completed.
3.6.2 Bug Reporting & Triage Conditions

 Bug find rate indicates a decreasing trend prior to Zero Bug Rate (no new Severity
1/2/3 bugs found);
 Bug find rate remains at 0 new bugs found (Severity 1/2/3) despite a constant test
effort across 3 or more days;
 Bug severity distribution has changed to a steady decrease in Severity 1 and 2 bugs
discovered; and
 No ‘Must Fix’ bugs remaining, despite sustained testing.

4 Software Risk Issues
4.1 Schedule
The schedule for each phase is very aggressive and could affect testing. A slip in the schedule in
one of the other phases could result in a subsequent slip in the test phase. Close project
management is crucial to meeting the forecasted completion date.
4.2 Technical
Since these are new XXXX systems, in the event of a failure, the old system can be used. We will
run our test in parallel with the production system so that there is no downtime of the current
system.
4.3 Management
Management support is required so when the project falls behind, the test schedule does not get
squeezed to make up for the delay. Management can reduce the risk of delays by supporting the
test team throughout the testing phase and assigning people to this project with the required
skills set.
4.4 Personnel
Due to the aggressive schedule, it is very important to have experienced testers on this project.
Unexpected turnovers can impact the schedule. If attrition does happen, all efforts must be
made to replace the experienced individual. This may pose a challenge, since most of our
testers are XXXX employees with primary responsibilities in other business units.
4.5 Requirements
The test plan and test schedule are based on the current Requirements Document. Any changes
to the requirements could affect the test schedule and will need to be approved by senior
management.

5 Test Approach
Functional testing will be conducted during the entire Application Development Life Cycle by
the XXXX Software QA and Development departments. Formal testing will be conducted by
XXXX’s business unit subject matter experts in two cycles; while overall testing of the entire
system will be conducted by the Software QA and Development departments in one final cycle.
The overall testing approach of the project will address and encompass the following tools and
processes:
5.1 Special Testing Tools

 Microsoft Visual Studio 2010 Ultimate - The comprehensive suite of application lifecycle
management tools used by the XXXX Software Development Team to ensure quality
results, from design to deployment. It includes: Integrated Development Environment,
Development Platform Support, Team Foundation Server, MSDN Subscription, Testing
Tools, Database Development, Debugging and Diagnostics, Application Lifecycle
Management, Architecture and Modeling, and Lab Management.
 Visual Studio Test Professional 2010 - An integrated testing toolset included with
Microsoft Visual Studio 2010 Ultimate; that delivers a complete plan-test-track
workflow for in-context collaboration between testers and developers. This tool
includes Test Manager 2010 and Test Runner and is used by the QA Engineer to
facilitate manual/automated testing and traceability.
 Microsoft Visual Studio Team Foundation Server 2010 (TFS) - The collaboration
platform at the core of our application lifecycle management (ALM) process. Team
Foundation Server 2010 automates the software delivery process and enables everyone
on our team to collaborate more effectively, be more agile, and deliver better quality
software while building and sharing institutional knowledge. Project artifacts like
requirements, tasks, bugs, source code, build and test results are stored in a data
warehouse. The tool also contains the reporting, historical trending, full traceability,
and real-time visibility into quality and progress.
5.2 Special Training on Testing Tools

Since Microsoft Visual Studio 2010 Ultimate is a relatively new tool on the market, some special
training will be required. Due to the current time constraints of our projects, this training will
be scheduled and completed in-house utilizing AppDev OnDemand, a self-paced online
courseware learning tool and includes the latest development products and technologies;
including SharePoint 2007, SQL Server 2005/2008, SQL Server 2005 Business Intelligence,
Visual Basic 2005/2008, Visual C# 2005/2008, ASP.NET, AJAX, .NET Framework, Windows
Workflow Foundation, Microsoft Windows Server, Microsoft Office, Microsoft Certification; all
with hands-on lab exercises, sample code, and pre/post exams.
5.3 Test Metrics

The Microsoft Visual Studio Test Professional 2010 application will be configured to collect test
metric data and to analyze the current level of maturity in testing and give a projection on how
we will go about testing activities by allowing us to set goals and predict future trends.
The objective of our test metrics is to capture the planned and actual quantities the effort, time
and resources required to complete all the phases of Testing of the XXXX application
improvement projects.
The SQA Engineer will use the test metrics as a mechanism to know the effectiveness of the
testing that can be measured quantitatively. It is a feedback mechanism to improve the testing
process that is followed currently and will be used to track actual testing progress against the
plan and therefore to be able to be proactive upon early indications that testing activity is
falling behind. The SQA Engineer has created a test metric as a standard means of measuring
different attributes of the software testing process. The metrics are a means of establishing test
progress against the test schedule and may be an indicator of expected future results. Our
metrics will be produced in two forms – Base Metrics and Derived Metrics as outlined below:
Base Metrics
 Number of Test Cases
 Number of New Test Cases
 Number of Test Cases Executed
 Number of Test Cases Unexecuted
 Number of Test Cases Re-executed
 Number of Passes
 Number of Fails
 Number of Test Cases Under Investigation
 Number of Test Cases Blocked
 Number of 1st Run Fails
 Number of Testers
 Test Case Execution Time
Derived Metrics
 Percentage of Test Cases Complete
 Percentage of Test Cases Passed
 Percentage of Test Cases Failed
 Percentage of Test Cases Blocked
 Percentage of Test Defects Corrected
5.4 Configuration Management

Test configuration management will be managed by the Lab Manager tool; included with the
Microsoft Visual Studio 2010 Ultimate package.
Lab Manager can fully provision and ready multiple environments for testing so that build
scripts can explicitly target a particular lab configuration at build time. Lab Management stores
the environments as virtual machine images in a library of pre-built images using System
Center Virtual Machine Manager (SCVMM) to ensure teams always begin their testing from a
known configuration.
The following operating systems and browser will be used in multiple combinations for testing
the XXXX applications:
Operating Systems
 Windows XP, Service Pack 3 or greater
 Windows Vista
 Windows 7
Browsers
 Firefox 3.0
 Internet Explorer 7.0
 Internet Explorer 8.0
5.5 Regression Testing

Regression testing will be conducted at the conclusion of each iteration of testing by the SQA
Department and any assigned business unit subject matter experts. In most cases, the testing
will be based on severity of defects detected.
5.6 Requirements Management

The business requirements will be elicited and managed by the Sr. Systems Analyst. Any
elements in the requirements and design that do not make sense or are not testable will be
immediately documented and reported to the Sr. System Analyst; who will in turn address these
issues with the shareholders for further clarification.
6 Test Strategy
The test strategy consists of a series of different tests that will fully exercise the applications. The
primary purpose of these tests is to uncover the systems limitations and measure its full
capabilities. The list of the various planned tests and a brief explanation are:

6.1 System Test
The System tests will focus on the behavior of the applications. User scenarios will be executed
against the system as well as screen mapping and error message testing. Overall, the system
tests will test the integrated system and verify that it meets the requirements defined in the
requirements document.
6.2 Performance Test

Performance test will be conducted to ensure that the application’s response times meet the
user expectations and meet and/or exceeds the specified performance criteria. During these
tests, response times will be measured under heavy stress and/or volume.
6.3 Security Test

Security tests will determine how secure the applications are. The tests will verify that
unauthorized user access to confidential data is prevented.
6.4 Automated Test

A suite of automated tests will be developed to test the basic functionality of the systems and
perform regression testing on areas of the systems that previously had critical/major defects.
The tool will also assist us by executing user scenarios thereby emulating several users.
6.5 Stress and Volume Test

We will subject the systems to high input conditions and a high volume of data during the peak
times. The Systems will be stress tested using twice (20 users) the number of expected users.
6.6 Recovery Test

Recovery tests will force the system to fail in various ways and verify the recovery is properly
performed. It is vitally important that all data is recovered after a system failure and no
corruption of the data occurred.
6.7 Documentation Test

Tests will be conducted to check the accuracy of the user documentation. These tests will
ensure that no features are missing, and the contents can be easily understood.
6.8 Beta Test

The internal business unit testers will beta test the new systems and will report any defects
they find. This will subject the system to tests that could not be performed in our test
environment.

6.9 Business Acceptance Test (BAT)
Once the applications are ready for implementation, the internal business unit testers will
perform Business Acceptance Testing (BAT). The purpose of these tests is to confirm that the
system is developed according to the specified user requirements and is ready for operational
use.
7 Entry and Exit Criteria
7.1 Test Plan
7.1.1 Test Plan Entry Criteria

The Entrance Criteria specified by the system test controller, should be fulfilled before System
Test can commence. In the event, that any criterion has not been achieved, the System Test may
commence if Business Team and Test Controller are in full agreement that the risk is
manageable.
 All developed code must be unit tested. Unit and Link Testing must be completed and
signed off by development team;
 System Test plans must be signed off by Sr. App Engineer and Sr. Systems Analyst;
 All human resources must be assigned and in place;
 All test hardware and environments must be in place, and free for System test use; and
 The Acceptance Tests must be completed, with a pass rate of not less than 80%.
7.2.2 Test Cycle Exit Criteria
The Exit Criteria detailed below must be achieved before the Round 1 software can be
recommended for promotion to Acceptance status. Furthermore, I recommend that there be a
minimum 1 day effort Final Integration testing AFTER the final fix/change has been retested.
 All High Priority errors from System Test must be fixed and tested;
 If any medium or low-priority errors are outstanding - the implementation risk must be
signed off as acceptable by Sr. App Engineer and Sr. Systems Analyst;
 Project Integration Test must be signed off by Sr. App Engineer and Sr. Systems Analyst;
and
 Business Acceptance Test must be signed off by Business Experts.
8 Test Deliverables
The following artifacts will be testing deliverables, available to the stakeholders:

Deliverable Responsibility Delivery Date
Develop Test cases QA Engineer TBD

Test Case Review QA Engineer, Sr. App Engineer, TBD
Testers
Develop Automated test suites QA Engineer TBD
Requirements Validation Matrix QA Engineer TBD
Execute manual and automated QA Engineer and Testers TBD

tests
Complete Defect Reports Testers TBD
Document and communicate test QA Engineer and Sr. Systems TBD

status/coverage Analyst
Execute Beta tests Testers TBD
Document and communicate QA Engineer and Sr. Systems TBD

Beta test status/coverage Analyst
Execute User Acceptance tests Testers TBD
Document and communicate QA Engineer and Sr. Systems TBD

Acceptance test status/coverage Analyst
Final Test Summary Report QA Engineer TBD
Here is a diagram indicating the dependencies of the various deliverables:

XXXX Business
Requirements Test Case Bug
(PM/BA) Results Results
XXXX Project Test Specs/

Test Plan (QA)
Plan (PM/BA) Outline (QA) Test Cases Bugs
XXXX Test Case

Detailed Design
Functional Coverage Bug Reports
(DEV)
Specs (PM/BA) Reports
Daily Status
Reports
As the diagram above shows, there is a progression from one deliverable to the next. Each
deliverable has its own dependencies, without which it is not possible to fully complete the
deliverable.
8.1 Deliverables Matrix

The following matrix depicts all of the deliverables that QA Testing will use. This matrix should be
updated routinely throughout the project development cycle in the project specific Test Plan.

Deliverable Milestone Sign-Off
Documents
Test Approach Planning
Test Schedule Design
Test Specifications Development
Test Case / Bug Write-Ups
Test Manager Test Cases/Results All
Test Manager Coverage Reports All
Bug Tracking System - Bugs and Regression All

Results
TFS Bug Tracking Reports All
Reports
Daily Status Reports All
Phase Completion Reports All
Test Final Report - Sign-Off Stabilization
TFS Source Control Archive Stabilization
8.2 Documents
8.2.1 Test Approach Document

The Test Approach document is derived from the Project Plan, Requirements and Functional
Specification documents. This document defines the overall test approach to be taken for the

project. The Standard Test Approach document that you are currently reading is a boilerplate
from which the more specific project Test Approach document can be extracted.
When this document is completed, the QA Engineer will distribute it to the IT Director, Sr.
Application Engineer/Architect, Sr. Systems Analyst, Application Engineer, and others as
needed for review and sign-off.
The purpose of the Standard Test Approach document is to:
 Specify the approach that QA Testing will use to test the product, and the
deliverables (extracted from the Test Approach);
 Break the product down into distinct areas and identify features of the product that
are to be tested;
 Specify the procedures to be used for testing sign-off and product release;
 Indicate the tools used to test the product;
 List the resource and scheduling plans;
 Indicate the contact persons responsible for various areas of the project;
 Identify risks and contingency plans that may impact the testing of the product;
 Specify bug management procedures for the project; and
 Specify criteria for acceptance of development drops to QA Testing (of builds).
8.2.2 Test Schedule

The Test Schedule is the responsibility of the QA Engineer and will be based on information
from the Project Scheduler (done by Product Manager). The project specific Test Schedule will
be done in Microsoft Project.
8.2.3 Test Specifications

A Test Specification document is derived from the Test Plan as well as the Requirements,
Functional Spec., and Design Spec documents. It provides specifications for the construction of
Test Cases and includes list(s) of test case areas and test objectives for each of the components
to be tested as identified in the project’s Test Plan.
8.3 Test Case/Bug Write-Ups
8.3.1 Test Manager Test Cases

Test Cases will be documented in Test Manager. Test Cases are developed from the Test
Specifications, Functional Spec., and Design Spec. Test Cases are detailed at the lowest level of
complexity. Results will be tracked as either Pass or Fail in the Test Manager and subsequently
in the Team Foundation Server. There must be an associated bug tracking number for every
Failed Test Case.
8.3.2 Test Case Coverage Reports

Test Case Coverage Reports will be generated in Team Foundation Server’s reporting
component and will provide the current status of test cases and pass/fail information. The
reports can breakdown the status information across the different test case areas, by level
(Smoke, Critical Path, Standard, Suggested, etc.), and in other formats.

8.3.3 Bug Tracking System and Regression Results
Bugs found will be documented and tracked in Team Foundation Server. There will be an
associated Test Case (in Test Manager) for every bug written. Standards for writing up bugs are
detailed in a later section entitled Test Manager and Bug Tracking Standards section of this
document.
8.3.4 TFS Bug Tracking Reports

Reports from the TFS Bug Reports component will be used to communicate information on all
bugs to appropriate project personnel.
The following bug reports will be generated from TFS:
Bug Status Report
Helps you track the team's progress toward resolving bugs and shows the number of
bugs in each state over time, a breakdown of bugs by priority or severity, and the
number of bugs that are assigned to each team member.
Bug Trends Report
Helps you track the rate at which the team is discovering and resolving bugs. Shows a
moving average of bugs discovered and resolved over time.
Reactivation
Helps you track how effectively the team is resolving bugs and shows the number of
bugs that the team resolved over time in relation to the number of bugs that the team
resolved and later reactivated.
8.4 Reports
The QA Engineer will be responsible for writing and disseminating the following reports to the
appropriate XXXX project personnel as required:
8.4.1 Weekly Status Reports

A daily status report will be provided by the QA Engineer to project personnel. This report will
summarize daily testing activities, issues, risks, bug counts, test case coverage, and other
relevant metrics.
8.4.2 Phase Completion Reports

When each phase of testing is completed, the QA Engineer will distribute a Phase Completion
Report to the Product Manager, Development Lead, and other relevant project team members
for review and sign-off.
The document must contain the following metrics:

 Total Test Cases, Number Executed, Number Passes/Fails, Number Yet to Execute;
 Number of Bugs Found to Date, Number Resolved, and Number still Open;
 Breakdown of Bugs by Severity/Priority Matrix;
 Discussion of Unresolved Risks; and
 Discussion of Schedule Progress (are we where we are supposed to be?).
8.4.3 Test Final Report - Sign-Off

A Final Test Report will be issued by the QA Engineer. It will certify as to the extent to which
testing has actually completed (test case coverage report suggested), and an assessment of the
product’s readiness for Release to Production.
8.4.4 TFS Source Control Archive

Once a project release cycle has been completed, all source code, documentation (including
requirements, functional spec, design spec, test plan, etc.), all testware automation, etc. should
be archived into TFS for source control and permanent storage.
9 Resource & Environment Needs

This section presents the non-human resources required for the Test Plan.
9.1 Base System Hardware

The following table sets forth the system resources for the test effort presented in this Test
Plan.
System Resources
Resource Quantity Name and Type
Application Server 1 HP ProLiant DL360 G5
—CPUs 2 x Intel Xeon L5240 Processor
—Memory 4GB RAM
—Hard Disk 1 72GB (Mirrored) HD
—Hard Disk 2 146GB (Mirrored) HD
—Server Name ADMINPREP
—IP Address 150.1.1.88
Test Development PCs TBD TBD
9.2 Base Software Elements in the Test Environment

The following base software elements are required in the test environment for this Test Plan.
Software Element Name Version Type and Other Notes

Windows Server 2008 (64-bit) Operating System
SQL Server 2008 (64-bit) Database Engine
9.3 Productivity and Support Tools

The following tools will be employed to support the test process for this Test Plan.
Tool Category or Type Tool Brand Name Vendor or In-house Version

Test Management Visual Studio TFS Microsoft 2010
Defect Tracking TBD
9.4 Testing Tools
9.4.1 Tracking Tools
9.4.1.1 Team Foundation Server

The TFS bug tracking and reporting component is used by XXXX to enter (bug can also be
entered in Test Manager and Test Runner as well) and track all bugs and project issues. The
component gives meaningful bug reports.
9.4.1.2 MS Test Manager 2010

Test Manager 2010 will be used to track status of test cases, avoid duplication of efforts (re-use
test cases with slight modification) by QA Testing. The QA Engineer is responsible for managing
the Test Manager/Team Foundation Server testing and bug reporting component. User Stories
(requirements) and Work Items (test cases) in TFS will be created for the project by the QA
Engineer and Sr. Application Engineer, when applicable.
9.4.1.3 Configuration Management

With Visual Studio Lab Management you can manage a set of virtual machines as a single entity
called a virtual environment. Each environment consists of one or many virtual machines for
each role required for your application. The environments that you create make up your virtual
lab. Then you use the virtual lab to deploy applications and run tests using Microsoft Test
Manager. These environments are created by using the Lab Center in Microsoft Test Manager.
Applications are deployed to these environments in your lab by using Team Foundation Build.
Tests are run on these environments in your lab from the Test Center by using Microsoft Test
Manager. Development and QA Testing will use Microsoft Lab Center to manage the virtual test
environments and configurations.
9.4.1.4 Issues Database

All projects will have issues that do not belong in Test Manager/TFS, but nonetheless need to be
dealt with. Too often these issues are not written down, and are thus forgotten in the verbal
maze of “Go-Dos”. Thus, we will at some point build a tool to track RFI’s (Request for
Information), COP’s (Change Order Proposals are RFI’s that have a schedule or cost impact), and
CO’s (Change Orders are COP’s which have been approved). Until then, we will put the issues
into Test Manager/TFS as a suggested bug.
9.4.2 Diagnostic Tools
9.4.2.1 ExamDiff Pro

Compare file differences. Registry Dumps, Dir c:\*.* /s, or .ini file snapshot changes over time—
take snapshots both before and after broken state occurs…then compare. (or use fc.exe f/NT
too).
9.4.3 Automation Tools
9.4.3.1 Microsoft Visual Studio Ultimate/Test Professional 2010

Use Visual Studio 2010 for Automation where appropriate (read: repeatable). We have to be
very careful not to be trapped into the allure of automating everything. It is most valuable to
automate Smoke Tests and many Critical Path Tests as they are repeated the most.
9.5 Test Environment

9.5.1 Hardware
QA Testing will have access control to one or more application/database servers (Dev and Test)
separate from any used by non-test members of the project team. QA Testing will also have
access control to an adequate number of variously configured PC workstations to assure testing
a range from the minimum to the recommended client hardware configurations listed in the
project’s Requirements, Functional Specification and Design Specification documents.
9.5.2 Software
In addition to the application and any other customer specified software, the following list of
software should be considered a minimum:
 MS Visual Studio Ultimate Test Professional 2010 (Testing Tool)

 MS Team Foundation Server (Testing Tool Server)
 MS Project
10 Responsibilities, Staffing and Training Needs

This section outlines the personnel necessary to successfully test the XXXX applications. Since staff
size is fixed these number may change.

10.1 People and Roles
This table shows the staffing assumptions for the test effort:
Human Resources
Role Minimum Resources Specific Responsibilities or Comments

Recommended
(number of full-time roles

allocated)
QA Engineer 1 Identifies and defines the specific tests to be
conducted.
Responsibilities include:
 identify test ideas
 define test details
 determine test results
 document change requests
 evaluate product quality
QA Engineer 1 Defines the technical approach to the
implementation of the test effort.
 define test approach
 define test automation architecture
 verify test techniques
 define testability elements
 structure test implementation
Business Unit Testers 6 Implements and executes the tests.
 implement tests and test suites
 execute test suites
 log results
 analyze and recover from test failures
 document incidents
Database Administrator, 1 Ensures test data (database) environment and
Database Manager assets are managed and maintained.
 Support the administration of test data and
test beds (database).
Sr. Application 2 Identifies and defines the operations,
Engineer, Application attributes, and associations of the test classes.
Engineer, and Sr.
Systems Analyst Responsibilities include:
 defines the test classes required to
support testability requirements as
defined by the test team

Human Resources
Role Minimum Resources Specific Responsibilities or Comments

Recommended
(number of full-time roles

allocated)
Sr. Application 2 Implements and unit tests the test classes and
Engineer, Application test packages.
Engineer, and Sr.
Systems Analyst Responsibilities include:
 creates the test components required
to support testability requirements as
defined by the designer
10.2 Staffing and Training Needs

Staffing is fixed for the duration of this project. It is likely most of the staff will assume some
testing role.
11 Test Manager Test Cases

QA Testing has defined standards for the structure of and entry of data into both Test Manager and
Test Manager/TFS. These standards will be adhered to in all projects.
11.1 Test Cases

QA Testing’s test cases will be entered and administered by the QA Engineer. The QA Engineer
will be responsible for the maintenance of data entered into Test Manager and Team
Foundation Server. QA Testing will track and report the success or failure of the test cases.
Tracking of test cases will include when tests were run, by whom, and code version/build
number as well as any comments logged after the test case was run. The QA Engineer will
provide the project team and Product Management with reports.
11.2 Test Case Contents
 Test Case Title

The title should include a brief description of the test case’s purpose.
 TCID
This field is automatically generated by the Test Manager application. It becomes the Test
Case Number and you cannot alter it.
 Status
o Assigned To: The person currently working on the test case.
o State: The workflow state of the test case, such as:
Closed – The test case is no longer required for future iterations of this team project
Design -The test case is being designed and has not yet been reviewed and approved
Ready –The test case has been reviewed and approved and is ready to be run
o Priority: Importance of the test case to the business goals of the product
o Automation Status: Identifies test case as manual or automated and is useful if you
plan to automate in future, such as:
Not Automated - This is a manual test case only

Planned - The plan is to add automation for this test case in the future
Automated - This value is automatically set if an automated test is added to this test case
 Classification
o Area: The area of the product with which the test case is associated and maps to the
feature areas in the application under development
o Iteration: The phase within which the bug will be fixed
 Steps
o You can write step and its expected result in the steps section,
o You can write a common step, or set of common steps by creating “Shared steps”.
Shared steps can be added to other tests. You may want to write common things like
– launching of application under test, logging in to the application, closing the
application as shared steps that you know you will require in many other tests,

o The normal step or shared step can be parameterized – you can actually make your
manual test data independent or can make a single test case to be executed against a
set of data over iterations. Again, common things here could be application URL,
login credentials or any test data that you want to use during the test.
o You can also attach a file to the step – like a document for reference or a screen
shot.
 Summary
This is where you would add a detailed description of the test case
 Tested User Stories

You can attach a test case to the user story (requirement) right when you are creating it by
going here and adding a link to the user story(s) being tested by that test case
 All Links
Bugs can be attached/linked to the test case by using “All links” tab
 Attachments
This is where you can add any file to the test case. For example, you could add a video
recording file, a screen shot file or a log file
 Associated Automation
If you have automated test case, you can link the test case to the automated test method
from the associated automation tab
 Parameter Values
These are the variable values set to replace data on each iteration of the test
12 Bug Management
12.1 Bug Documentation

All bugs will be documented in Test Manager and/or TFS. Bug descriptions will follow the XXXX
test case/bug write-up standard (included below). The QA Engineer will be responsible for the
maintenance of Test Manager/TFS database and bugs therein. All necessary project team
members should have access to TFS.
Every bug entered into TFS’s tracking system will have an associated Test Manager test case
number associated with it. Where a bug does not fit into an existing test case, a new test case
will be created and its number referenced in Test Manager/TFS bug entry. The new test case
will be categorized or listed as a Smoke Test item in the Test Plan. Older cases may be updated
to extend their potential for catching the new bug as long as it doesn’t significantly alter the
objective of the original test case. In the event that a bug is found by a person outside of QA
Testing, the bug should be reported to the QA Engineer who will then assure that further testing
is completed to verify the existence of the bug, refine the repro steps, incorporate it into Test
Manager, and add it to TFS for bug tracking.
12.1.1 Bug Severity and Priority Definition

Bug Severity and Priority fields are both very important for categorizing bugs and prioritizing if
and when the bugs will be fixed. The bug Severity and Priority levels will be defined as outlined
in the following tables below. QA Testing will assign a severity level to all bugs. The QA
Engineer will be responsible to see that a correct severity level is assigned to each bug.
The QA Engineer, Sr. App Engineer, App Engineer, Sr. Systems Analyst and IT Director will
participate in bug review meetings to assign the priority of all currently active bugs. This
meeting will be known as “Bug Triage Meetings”. The QA Engineer is responsible for setting up
these meetings on a routine basis to address the current set of new and existing but unresolved
bugs.
12.1.1.1 Severity List

The tester entering a bug into Bug Tracking System is also responsible for entering the bug
Severity.
Severity ID Severity Level Severity Description

1 Crash The module/product crashes or the bug causes non-recoverable
conditions. System crashes, GP Faults, or database or file corruption,
or potential data loss, program hangs requiring reboot are all
examples of a Severity 1 bug.
2 Major Major system component unusable due to failure or incorrect

functionality. Severity 2 bugs cause serious problems such as a lack of
functionality, or insufficient or unclear error messages that can have a
major impact to the user, prevents other areas of the app from being
tested, etc. Severity 2 bugs can have a work around, but the work
around is inconvenient or difficult.
3 Minor Incorrect functionality of component or process. There is a simple

work around for the bug if it is Severity 3.
4 Trivial Documentation errors or signed off severity 3 bugs.
12.1.1.2 Priority List

Priority Priority Level Priority Description
1 Must Fix This bug must be fixed immediately; the product cannot ship with
this bug.
2 Should Fix These are important problems that should be fixed as soon as
possible. It would be an embarrassment to the company if this bug
shipped.
3 Fix When Have Time The problem should be fixed within the time available. If the bug
does not delay shipping date, then fix it.
4 Low Priority It is not important (at this time) that these bugs be addressed. Fix
these bugs after all other bugs have been fixed.
12.2 Test Runner Bug Entry Fields

The QA Engineer will be responsible for managing the bug reporting process. QA Testing’s
standard bug reporting tools and processes will be used. TFS is the company-wide standard
Bug Logging/Tracking tool. QA Testing and Development will enter their data into Test
Manager/TFS database following the field entry definitions defined below:

 Bug Title
The title should include a brief description of the test case’s purpose.
 Status
o Assigned To - Select the person the bug is being assigned to.
o State - Select whether the bug is “Active” (default) or “Inactive” in the test cycle.
o Reason - Select whether the bug is related to a “New” defect or a “Build Failure”.
o Resolved Reason - Enter a short description of the resolution.
 Classification
o Area - Select the appropriate area in the team project for this bug.
o Iteration - Select the appropriate iteration for this bug.
 Planning
o Stack Rank - The stack rank is used as a way to prioritize your work. The lower the
stack rank the higher the priority the work item is.
o Priority - Select a priority rating from 1 to 4. 1 representing the most urgent.
o Severity - Select the severity of the bug from 1 to 4. 1 representing the most critical.
 Details
The Details display the test steps and detailed actions that were automatically added to a
specific test step, such as input data, expected and actual results, comments, and
attachments.
 System Info
The System Info displays detailed information about the configuration of the computer used
during the test.
 Test Cases
Displays additional test cases related to the bug.
 All Links
Displays test result attachments that are added as links. This includes diagnostic trace data.
 Attachments
A set of files attached to help provide additional information to support the issue.
12.3 Bug Reporting Process

Tester finds an Issue (Bug/Defect)
Developer marks the Tester “Creates a

issue as “Not a Bug” and Bug” in Microsoft Test
reassigns back to tester Runner and assigns
with an explanation for to a Developer for
re-testing analysis
No Is it really a bug?
Yes Yes
Is bug a Prep Course Is bug a netForum

or Vendor issue and issue and outside the
outside the scope of Yes scope of XXXX code
XXXX code customization?
customization?
Developer fixes the

Developer submits
issue, marks it as Developer submits
the Bug to Eureka or
“Resolved” and the Bug to Avectra for
Vendor for analysis/
reassigns to tester for analysis/fix
fix
re-testing
Vendor resolves Developer checks in Avectra resolves

the issue, new Source Code from the issue,
publishes a new build and reassigns publishes a new
Build and marks Bug back to Tester for Build and marks
No No
it as “fixed” re-testing No it as “fixed”
Developer
Developer
receives email
receives email
confirmation
confirmation
from Eureka or
from Avectra
Vendor
Is it really fixed? acknowledging
acknowledging
Bug fix
Bug fix
Yes
Tester marks the Bug as

“Resolved” and continues testing
13 Documentation
The following documentation will be available at the end of the test phase:
 Standard Approach Document
 Test Cases
 Test Case review
 Requirements Validation Matrix
 Defect reports
 Final Test Summary Report


Standard Test Approach

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Standard Test Approach

Загружено:

Авторское право:

Доступные форматы

STANDARD TEST APPROACH

XXXX Software Development Projects

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

Date Version Description Author

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

2.1 TEST APPROACH OBJECTIVES....................................................................................................................7

3.1 TEST STAGES.........................................................................................................................................9

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

6.1 SYSTEM TEST......................................................................................................................................22

7.1 TEST PLAN..........................................................................................................................................24

8.1 DELIVERABLES MATRIX..........................................................................................................................27

9.1 BASE SYSTEM HARDWARE......................................................................................................................31

10.1 PEOPLE AND ROLES.............................................................................................................................34

12.1 BUG DOCUMENTATION............................................................................................................................38

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

2.2 High-Level Product Development Objective

 Update the software to a current technology platform while maintaining the

2.2.2 Secondary Objective

2.3 Scope of Applications Under Test

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

Milestone 1 - Planning Phase

 Assessing the impact of Requirements on testing;

Milestone 2 - Design Phase

The purpose of the Test Plan is to achieve the following:

Milestone 2a - Usability Testing

Milestone 3 - Developing Phase

Milestone 3a - Unit Testing (Multiple)

 Databases, Stored Procedures, Triggers, Tables, and Indexes;

Milestone 3b - Acceptance into Internal Release Testing

 Sign-off from the Senior Application Engineer/Architect that Unit Testing is

Milestone 3c - Internal Release Testing

Internal release testing will include:

 Execution of Test Manager test cases;

Milestone 3d - Acceptance into Alpha Testing

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

Milestone 3e - Alpha Testing

Milestone 4 - Stabilization Phase

Milestone 4a - Acceptance into Beta Testing

Important deliverables required for acceptance into Beta Testing include:

Milestone 4b - Beta Testing/Business Acceptance Testing (BAT)

Milestone 4c - Release to Production

 Check for extra or missing files (Diff operation on directory);

Milestone 4d - Post Release

3.2 Test Levels

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

3.2.1 Build Tests

Level 1 - Build Acceptance Tests

Level 2 - Smoke Tests

Level 2a - Bug Regression Testing

3.2.2 Milestone Tests

Level 3 - Critical Path Tests

3.2.3 Release Tests

Level 4 - Standard Tests

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

3.3 Bug Regression

3.4 Bug Triage

3.5 Suspension Criteria and Resumption Requirements

Prepared by: André J. Jackson, Software QA Engineer Proprietary and Confidential

3.6 Test Completeness

3.6.1 Standard Conditions

3.6.2 Bug Reporting & Triage Conditions