Building the Test Environment

The test environment is comprised of all the conditions,

circumstances, and influences surrounding and affecting the
testing of software. The environment includes the
organization’s policies, procedures, culture, attitudes,
rewards, test processes, test tools, methods for developing
and improving test processes, management’s support of
software testing, as well as any test labs developed for the
purpose of testing software and multiple operating
environments. This category also includes assuring the test
environment fairly represents the production environment to
enable realistic testing to occur.
P. 133
 Four Areas of Focus
• Management Support
• Test Work Processes
• Test Tools
• Testers Competency
 Management Support
• Without adequate management support
testing is rarely performed effectively.
• Management tone and Test process
alignment are the two areas of
management support discussed.
 Management Tone
• Management sets the tone by providing
testers the resources and management
time needed to do their job effectively.
• It is representative of the environment that
management has established that
influence the way testers work.
• Control environment is part of
management tone.
 Control Environment
• Three things to know about it.
3) The control environment is established by the highest levels of
management and works downward through the organization.
4) The test function cannot create the organization’s control
environment, but can influence how that environment is
implemented within the test function.
5) The control environment will influence the way in which testers
perform the work which may be ethical or unethical.
 Three sub-areas of Management
Tone
• Integrity and Ethical Values
• Commitment to Competence
• Management’s Philosophy and Operating
Style
 Integrity and Ethical Values
• An entity’s objectives and the way they are achieved are based on
preferences, value judgments and management styles. Those
preferences and value judgments, which are translated into
standards of behavior, reflect management’s integrity and its
commitment to ethical values.
• Ethical behavior and management integrity are a product of the
“corporate culture.” Official policies specify what management wants
to happen. Corporate culture determines what actually happens.
• Two areas of Integrity and Ethical values: incentives & temptations
and Providing & Communicating Moral Guidance.
 Incentives & Temptations
• Incentives cited for engaging in fraudulent practices and unethical
behavior are:
2) Pressure to meet unrealistic performance targets, particularly for short-
term results.
3) High performance-dependent rewards.

• Temptations for employees to engage in improper acts include:

6) Nonexistant or ineffective controls
7) High decentralization of the organization
8) A weak internal audit function
9) Insignificant penalties for improper behavior

• Removing these incentives & temptations can be achieved by

following sound and profitable business practices.
 Providing and Communicating
Moral Guidance
• The most effective way of transmitting a
message of ethical behavior throughout
the organization is by example.
• Top management should verbally
communicate the entity’s values and
behavioral standards to employees.
 Commitment to Competence
• Competence should reflect the knowledge
and skills needed to accomplish tasks that
define the individuals job.
• Management needs to specify the
competence levels for particular jobs and
to translate those levels into requisite
knowledge and skills.
 Management’s Philosophy and
Operating Style
• Management’s philosophy and operating
style affect the way testing is managed,
including the kinds of business risks
accepted.
 Organizational Structure
• Organizational Structure affects test process alignment

• An entity’s organizational structure provides the framework within

which its activities for achieving entity-wide objectives are planned,
executed, controlled and monitored.
• Activities may relate to what is sometimes referred to as the value
chain: inbound (requirements) activities, operations or production,
outbound (software), deployment and maintenance.
• An entity’s organizational structure depends in part on its size.
• An entity’s activities will be organized to carry out the strategies
designed to achieve particular objectives.
• Assignment of Authority & Responsibility and Human Resource
Policies and Practices are important areas of Organizational
Structure.
 Assignment of Authority and
Responsibility
• Management has an important function with the
assignment of authority and responsibility for
operating activities, and establishment of
reporting relationships and authorization
protocols.
• Alignment of authority and accountability often is
designed to encourage individual initiatives,
within limits.
 Human Resource Policies and
Practices
• Human resource practices send messages to employees
regarding expected levels of integrity, ethical behavior and
competence.
• These include:
3) Standards for hiring the most qualified people
4) Recruiting practices
5) Training policies
6) Rotation of personnel and promotions driven by periodic
performance appraisals demonstrate the entity’s commitment to
the advancement of qualified personnel.
7) Competitive compensation programs that include bonus
incentives that serve to motivate
8) Disciplinary actions that send a message that certain behavior will
not be tolerated.
 Test Work Processes
• Work processes are the policies, standards and procedures in a
quality IT environment.
• Test work processes are those work processes specific to the
testing activity.
• Once management commits to create a work environment
conducive to effective software testing, test work processes need to
be created.
• It is the tester’s responsibility to follow these work processes; and
management’s responsibility if the processes don’t work.
• A process to improve the test work processes should be
implemented to improve the effectiveness and efficiency of the
policies, standards, and procedures.
 Test Work Processes Cont’d
• Policy – Managerial desires and intents concerning either process
(intended objectives) or products (desired attributes).
• Standards – The measure used to evaluate products and identify
nonconformance. The basis upon which adherence to policies is
measured.
• Procedure – The step-by-step method that ensures that standards
are met.
• Policies provide direction, standards are the rules or measures by
which the implemented policies are measured, and the procedures
are the means used to meet or comply with the standards.
• Two test work processes are critical: importance of work processes,
the responsibilities.
 Test Work Processes Cont’d
• It is important to a quality IT environment to
establish, adhere to, and maintain work
processes in the testing activity.
• Major purposes and advantages of work
processes:
3) Improves communication
4) Enables knowledge transfer
5) Improves productivity
6) Assists with mastering new technology
7) Reduces defects and costs
 Test Work Processes Cont’d
• It is important that organizations clearly establish who is responsible for
developing work processes (i.e., policies, procedures, and standards).
• Policies are need in areas such as:

4) Building systems
5) Testing systems
6) Maintaining systems
7) Operating systems
8) Quality of systems
9) Security of systems
10) Allocation of resources
11) Planning for systems
12) Training personnel
 Test Work Processes Cont’d
• Key concepts are:
2) Policies are developed by senior management.
3) Policies set direction but do not define specific
products or procedures.
4) Policies are needed in areas that cause problems.
5) Policies define the areas in which processes will be
developed.
 Test Processes Cont’d
• The workers who use the procedures and are required to comply
with the standards should be responsible for the development of
those standards and procedures.
• Management sets the direction and workers define that direction.
• Key concepts are:
4) Management provides the organizational structure for workers to
develop their own standards and procedures.
5) The program is driven by management policies.
6) Absolute compliance to standards and procedures is required.
7) A mechanism is provided for the continual maintenance of
standards and procedures to make them more effective.
• The software tester should be the owners of test processes – and
thus involved in the selection, development and improvement of
test processes.
 Test Work Processes Cont’d
• Selecting, developing, and acquiring work processes is
an overall IT organization responsibility.
• Normally the function that performs this activity is called
the process engineering function.
• Software testers need to both understand how the
activity operates AND participate when test processes,
and related processes, are selected and put into
practice.
• IT groups should develop a plan for implementing and
operating a process engineering program.
 Test Work Processes Cont’d
• Specific components for the process engineering
program include:
2) Building a Process Engineering Organization
3) Developing a Standard and Procedure for Standards
4) Planning for Standards
5) Writing, Storing, and Retrieving Standards and
Procedures.
6) Enforcing Standards.
 Test Work Processes Cont’d
• The structure that is put into place to develop and update policies,
standards, and procedures must involve both staff and
management.
• Guidelines for establishing an organizational structure:
3) Establish a Process Engineering Committee comprised of the
most senior IT managers.
4) Represent all IT organizational areas on the Process Engineering
Committee
5) Appoint an individual as the Process Engineering Manager.
6) Appoint Ad Hoc Committees to develop individual standards and
procedures.
7) Let the Standards Ad Hoc Committees develop the technical
standard.
 Test Work Processes Cont’d
• Responsibilities of the Process Engineering Manager.
2) Promote the concept of process engineering
3) Be the driving force behind the processes
4) Administer the standards program defined by the
Process Engineering Committee.
5) Be a resource to the Process Engineering Committee
and Ad Hoc Committee.
6) Ensure involved parties are adequately trained.
 Test Work Processes Cont’d
• Role of the process engineering committee is
to:
2) Accept topics for processes
3) Set priority implementation of processes
4) Obtain the resources necessary to develop the
process.
5) Approve or reject developed processes.
 Test Work Processes Cont’d
• Responsibilities of the Ad Hoc Committee:
2) Gain representatives from all involved areas.
3) Ensure that the committee has between three
and eight members in size.
4) Create the standard and procedure.
5) Coordinate reviews of the standard with
involved parties.
6) Periodically review and update the standards
and procedures previously developed by the
Ad Hoc Committee.
 Test Work Processes Cont’d
• The make up of the process engineering committee is
important. Some guidelines in selecting members are:
2) Select the highest-level manager who will accept the
position.
3) Assign individuals who are supportive and enthusiastic
over standards.
4) Make long-term assignments to the Process
Engineering Committee.
5) Select individuals who are respected by their peers
and subordinates.
6) Ensure the appropriate areas of interest are involved.
 Test Work Processes Cont’d
• Guidelines to ensure the appropriate areas of interest
are included:
2) Is every organizational function within IT involved?
3) Are the activities that interface to the IT function
involved; for example, key users and customers?
4) Are activities having a vested interest involved, such
as auditors.
5) Are all the IT “businesses” represented such as the
project leaders, systems programmers, data library,
security, help desk, and so forth?
 Test Work Processes Cont’d
• Prior to creating work processes, the Process
Engineering Committee must develop a standard and
procedure for developing standards and procedures.
• A standard should define:

4) A testing policy – managements objective for testing

5) A test workbench to support the policy (i.e. standards
and processes)
 Test Work Processes Cont’d
• Testers workbench – input products drive the workbench, which
uses procedures and standards to produce output products.
• The tester’s workbench is a pictorial representation of how a specific
test task is performed.
• It is the policies that define the objective for workbenches.
• A function may need several workbenches, such as:
• A budgeting workbench, scheduling workbench, risk assessment
workbench, and tool selection workbench.
• The objective of a workbench is to produce defined output products
in a defect-free manner.
• It is the execution of the workbench that defines worker product
quality, and is the basis for productivity improvement.
 Test Work Processes Cont’d
• Professional test standards.
• Test professionals should be familiar with
the standards published by organizations
such as the ISO, U.S. Dept. of Commerce,
NIST, and IEEE.
• IEEE standards listed on P. 152
 Test Work Processes Cont’d
• Testers are the quality control process for developers.
• If high-quality work is needed, quality control should be
an integral part of the test process.
• Test quality control is performed during the execution of
the process.
• Analysis is performed after the test process is
completed.
• Test analysis can only be performed if adequate data is
maintained during the test process.
 Test Work Processes Cont’d
• Eight test analyses common to the test industry. P154-
158
2) Effectiveness and efficiency of test processes.
3) The test objectives are applicable, reasonable, adequate,
feasible, and affordable.
4) Testers do not have the needed competencies to meet the test
objectives.
5) The test program meets the test objectives.
6) The correct test program is being applied to the project.
7) The test methodology is used correctly.
8) The task work products are adequate to meet the test objectives.
9) Analysis of the results of testing to determine the adequacy of
testing.
 Test Work Processes Cont’d
• Continuous Improvement – process improvement is best
considered as a continuous process, where the
organization moves continually around an improvement
cycle.
• An important step in the improvement cycle is the
execution of data gathering to establish the initial state,
and subsequently to confirm improvements. A testing
process assessment is one excellent way to determine
the status of your current test process.
• Assessment – is the thoughtful analysis of testing
results, and then taking corrective action on the identified
weakenesses.
 Test Work Processes Cont’d
• Testing process assessment is a means of capturing
information describing the current capability of an
organization’s test process.
• A formal entry to the process occurs wit the
compilation of the assessment input, which defines
the:
3) Purpose – why it is being carried out
4) Scope – which processes are being assessed and
what constraints apply.
5) Any additional information that needs to be gathered.
6) The input also defines the responsibility for carrying
out the assessment.
 Test Work Processes Cont’d
• Process assessment is undertaken to measure an organization’s current
processes.
• Process assessment is applicable when:
3) Understanding the state of processes for improvement.
4) Determining the suitability of processes for a particular requirement of
class of requirements.
5) Determining the suitability of another organization’s processes for a
particular contract or class of contracts.
• The framework for process assessment is:
7) Encourages self-assessment
8) Takes into account the context in which the assessed process operates.
9) Addresses the adequacy of the management of the assessed processes
through generic practices.
10) Is appropriate across all application categories and sizes of organization.
 Test Work Processes Cont’d
• Continuous improvement
• One of the most commonly identified
weaknesses in software testing has been the
lack of facts (metrics), and without facts there is
no need to take action (improvement).
• Once appropriate measures are identified,
tracked, and analyzed, then a plan for
continuous improvement can be implemented.
• The process continues until the appropriate and
effective improvement mechanisms are
uncovered and included in the normal process.
 Test Work Processes Cont’d
• Eight steps in the Test Process Improvement Model.
2) Examine the Organization’s Needs and Business
Goals.
3) Conduct Assessment
4) Initiate Process Improvement
5) Analyze Assessment Output and Derive Action Plan.
6) Implement Improvements
7) Confirm Improvements
8) Sustain Improvement Gains
9) Monitor Performance
 Test Work Processes Cont’d
• In establishing the test environment
management must assure that the
mission/goals of the test function are
aligned to the mission/goals of the
organization.
 Test Work Processes Cont’d
• Adapting the Test Process to Different Software Development
Methodologies.
• In the initial days of software development, testing was considered a
phase of development, the testing phase began after software was
developed costing an estimated 10 times per defect compared to a
defect found in the requirements phase.
• When testing is viewed as a life cycle activity testing and
development occur together becoming an integral part of the
development process.
 Test Work Processes Cont’d
• Understanding the software development process is
important for three reasons.
2) Understanding the developmental timetable and
deliverables.
3) Integrating the software
4) Understanding how software is developed.
 Test Work Processes Cont’d
• There are thousands of software development
methodologies including:
2) Waterfall
3) D-shaped
4) Prototype
5) Spiral
6) RAD
7) Incremental
*Book Software Quality Institute described strengths and
weakenesses.
 Test Work Processes Cont’d
• Testers testing software developed by a specific
software development methodology need to:
2) Understand the methodology
3) Understand the deliverables produced when using
that methodology which is needed for test purposes.
4) Identify compatible and incompatible test activities
associated with the developmental methodology.
5) Customize the software test methodology to effectively
test the software base on the specific developmental
methodology used to build the software.
 Test Work Processes Cont’d
• In customizing the software testing process to the different
methodologies the tester needs to know:
2) Will the users be involved in the developmental process?
3) Do the users understand this developmental process?
4) Are the user’s experts in their business activity?
5) Is this project an enhancement to an existing software system
developed using another methodology?
6) Is high reliability an important aspect of this software development
project and does the selected developmental methodology assure
high reliability?
7) Is the amount of changes expected to be implemented in the
software system consistent with the capabilities of developmental
methodology?
 Test Tools Cont’d
• Test Tools – Its difficult to perform testing economically without
the aid of automated test tools.
• The most important aspect of software testing is the process used
to acquire those tools.
• The procedures developed for testers to follow during testing
should include testing tools and techniques.
• Difficulties surrounding the introduction of tools can arise in three
areas:
5) Organizational obstacles
6) Problems arising from the tools
7) Obstacles in the computer environment
 Test Tools Cont’d
• The activities associated with the introduction of tools should
include the following:
2) Identifying the goals to be met by the tool, and assigning
responsibility for the activities required to meet these goals
3) Approving a detailed tool acquisition plan that defines the
resource requirements for procurement and in-house activities
4) Approving the procurement of tools and training, if this is not
explicit in the approval of the acquisition plan.
5) Determining, after some period of tool use, whether the goals
have been met.
 Test Tools Cont’d
• Test management is assigned responsibility for:
2) Identifying tool objectives
3) Approving the acquisition plan
4) Defining selection criteria
5) Making the final selection of the tool or the source
• The test manager is responsible for the following activities:
7) Identifying candidate tools
8) Applying the selection criteria (informal procurement) or preparing
the RFP (informal procurement)
9) Preparing a ranked list of tools or sources
10) Conducting any detailed evaluations or conference room pilots.
 Test Tools Cont’d
• The distribution of responsibilities reduces the chances of
selecting a tool that:
2) Does not meet the recognized needs of the organization.
3) Is difficult to use
4) Requires excessive computer resources
5) Lacks adequate documentation
 Test Tools Cont’d
• Event sequencing includes:
2) employ a methodical process of identifying candidate
tools and selecting among these based on established
criteria. This will avoid some of the pitfalls of
“borrowing” a tool from an acquaintance, etc…
3) Determine the assignment and training of a Tool
Manager who can make minor modifications to both
the computer environment and the tool to provide relief
from version and release incompatibilities with the
operating system.
 Test Tools Cont’d
• Event Sequence steps P. 171-179
2) Goals – identified
3) Tool objectives – desired features and requirements
2a) Acquisition activities for informal/ formal procurment
e.g. by request or by purchase order
3) Procure Tool – cost and trial period with vendor considerations.
4) Evaluation Plan – milestones, initial performance assessments.
5) Implementation Plan – responsibilities and tasks.
6) Training Plan – How and Who to train.
7) Tool Received
8) Acceptance Test – test the tool
9) Orientation – communicate objectives of tool use
10) Modifications
11) Training – scheduling and training on use
12) Use in the operating environment
13) Evaluation Report
14) Determine if goals are met
 Test Tools Cont’d
• Eight Catagories of test tools
2) Automated regression tools
3) Defect management tools
4) Performance/load testing tools
5) Manual tools
6) Traceability tools
7) Code Coverage
8) Test case management tools
9) Common tools that are applicable to testing
 Test Tools Cont’d
• Most testing organizations agree that if the following
three guidelines are adhered to tool usage will be more
effective and efficient.
2) Guideline 1 – Testers should not be permitted to use
tools for which they have not received formal training.
3) The use of test tools should be incorporated into test
processes so that the use of tools is mandatory, not
optional.
4) Testers should have access to an individual in their
organization, or the organization that developed the
tool, to answer questions or provide guidance on using
the tool.
 Testers Competancy
• Test competency is the direct responsibility of the
individual and the organization that employs the
individual.
• It is based on two criteria:
3) Skill sets – defines the totality of skills that need to be
learned.
4) Performance skills – carrying out the skill sets