The Learning Organization: Performance Technology and the Implications for Organizational Effectiveness

William C. Redeen December 1, 1997 Research Paper- PUBP 802

(The interactive web-based version of this document may be found at: http://208.208.159.222/PUBP802/)

Table of Contents
Abstract Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What is Performance Technology? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Research Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Organization of the Paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Trends in Productivity and Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 The Learning Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Measurement: The Missing Link in Performance Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Performance versus Learning Focus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Trends in Performance Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Performance Technology Measures in The Studies Reviewed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 The Relationship of Performance Technology Performance Measurement to Organizational Strategic Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Performance Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Functional Process Improvement & Business Process Re-engineering (BPR) . . . . . . . . . . . . . 10 Activity-based Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Are the Emerging Performance Technology Development Methodologies Compatible? . . . . . . . . . . . 12 Performance Centered Design (PCD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Performance Support Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 An Model of Performance Technology & Organizational Performance . . . . . . . . . . . . . . . . . . . . . . . . 16 The Research Literature: How Effective is Performance Technology? . . . . . . . . . . . . . . . . . . . . . . . . . 18 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Abstract

This paper reviews the current research literature to identify the relationship between performance technology and organizational effectiveness. Previous research has attempted to quantify the effect of the training technology on training effectiveness. Unfortunately, the relationship of these outcomes to organizational effectiveness is typically unclear. With the rapid evolution and convergence of training and performance technologies, there is a renewed potential for establishing a direct relationship between performance technology and organizational effectiveness. In addition, the recent emergence of several key planning methodologies in different disciplinary fields appear to offer an opportunity to address this historic measurement gap between organizational and training effectiveness. This paper proposes a model for integrated training and organizational performance measurement, and summarizes the results of the limited existing empirical studies on performance technology and organizational effectiveness.

Introduction This paper reviews the current research literature on the relationship between performance technology and organizational effectiveness. Much of the historic research has focused on the training component of performance technology, and has addressed training effectiveness without establishing a direct relationship to organizational effectiveness. As a result, the training function has been viewed with scepticism in terms of its contribution to organizational effectiveness.

The recent rapid evolution and convergence of several key training and performance technologies, has created a renewed potential for establishing a direct relationship between performance technology and organizational effectiveness. Specifically, performance technology can now offer on-demand training and performance support at the employees person or work place. For the first time, central collection and measurement of performance is possible using distributed client-server architectures based on open standards, such as the Internet or Intranets. While there is a general consensus that the introduction of this technology is increasing organizational effectiveness, there have been only limited attempts to synthesize the existing research to develop performance measurement and quantitative guidelines to help evaluate the effect of technology on common measures of organizational effectiveness.

What is Performance Technology? According to Martin Wikoff, performance technology emphasizes the systematic analysis of factors that influence behavior and performance, with the objective of measurably improving individual and organizational performance. Performance technology encompasses all tools which enhance human

performance, including traditional training, computer-based training and electronic performance support systems. Recently and for purposes of this paper, performance technology refers to the new information technology continuum. Figure 1 displays the performance technology continuum. According to Raybould, performance technology can be designed with the primary objective of transferring knowledge or of producing performance.1 This paper focuses on the new paradigm of performance-centered design, which emphasizes the design of a training or performance support system to meet specific measurable performance outcomes, both at the individual and organizational level. Learning may occur, but it is not the primary objective. The primary objective is enhanced organizational performance.

Barry Raybould. The Role of Technology in Improving Performance. EPSS.COM, November

1997. 1

Performance-centered systems are technologies that focus on performance as the primary objective. For example, a software application may be designed to allow immediate performance of a job at a high level by integrating the knowledge, information and tools a job performer needs into the software application or a global interface to the organizational knowledge base. Technologies in this category include: Performance Support Tools:: global or specialized software applications to help people perform job tasks such as give an employee review, write a legal document or develop a negotiating strategy Performance-Centered Information Systems: a global system supporting an organizational activity or function, such as a customer call center application. Performance-Centered Intranets- Intranets designed around a functional, topical or task-oriented structure

These integrate a

technologies range of

The Performance Technology Continuum in 1997

Performance Technology

performance support tools t h r o u g h t h e

human/computer interface,
Training Performance Support

creating

partnership

between job performer and

Classroom Training and VTT ComputerBased Training Experts and Colleagues Paper Job Aids

machine.

These

can

includeon-screen help and graphics,

Web-based Training Online References Electronic Performance Support Systems

dialog task

boxes, bars, cards, agent

pushbuttons, wizards, checklists

Limited Implementation

cue and

metaphors to access a Figure 1. The Status of Performance Technology in 1997 common organizational

knowledge base.

Performance technology structures can also be designed with a learning focus in which case the primary objective is that of transferring knowledge to long-term memory. These technologies include: Computer-based training (CBT) 2

Web-based training (WBT): CBT delivered via an Intranet, and Simulation technologies, such as software simulators, business practice simulators or equipment simulators.

Research Issues For performance technology to realize its full potential, the relationship to organizational effectiveness must be clearly defined. This paper addresses the following specific research questions: What tools and methodologies exist to support the measurement of the effectiveness of performance technologies and the relationship to organizational performance? Are the emerging methodologies for performance technology design (Performance-centered Design) and strategic planning (Performance Management) compatible? How can these tools be used congruently to measure and optimize organizational performance? How effective are the new performance technologies? Is there any quantitative data on the relationship of performance technology and organizational effectiveness?

Organization of the Paper This paper is organized into six sections. First, we address recent trends in productivity and the nature of work, establishing the basis for the importance of performance technology. Second, we describe the concept of the Learning Organization, and the related paradigm shift in the concept of training and performance technology. Third, we establish the importance of performance measurement as the mechanism for defining the effectiveness of performance technology. Fourth, we describe emerging performance technology design methodologies and their relationship to organizational performance measurement. Fifth, we document the limited empirical research conducted on performance technology and describe some of the limitations. Finally, we draw conclusions about the potential of performance technologies for enhanced organizational effectiveness and identify areas for further research.

Trends in Productivity and Work Productivity growth is recognized as central to maintaining the current U.S. economic leadership. Fundamental changes in the nature of work and the capabilities of emerging technologies have created a unique opportunity for enhancing individual and organizational productivity.

With the emergence of the information economy, knowledge work is the area that offers the greatest 3

opportunity to increase productivity within the U.S. workforce.2 Knowledge workers make decisions that significantly impact organizational resources and are themselves a significant and costly resource. Knowledge workers compose 43 percent of the white-collar sector, which in turn comprises 67 percent of the service sector.3 Most of these personnel use information systems as a component of their work. Traditionally, knowledge work demands substantial training to establish and maintain competency.

With the recent radical reductions in hardware and software cost, personnel costs and training have become the Figure 2. Information Technology Costs, 1996 largest cost component of Information Systems (IS) implementation and life cycle support.
4

Figure 1 presents IS costs by major category of expenditure, according to the Gartner

Group. Despite this recent data, the production work environment continues to dominate productivity enhancement efforts in spite of evidence that the returns on further refinements do not equal those possible in the knowledge worker environment.

Thus, because of the convergence of these trends, the enhancement of knowledge worker performance has become the paramount focus for increasing productivity in the new information economy.

The Learning Organization The concept of a learning organization, as described by Peter Senge, calls for an organization that fosters group and individual learning through changes in organizational culture and the implementation of systems.

Drucker, Peter F., Management (Harper & Row, 1974).

Roach, Stephen, "Services Under Siege-The Restructuring Imperative," Harvard Business Review (September-October 1991), pp 82-83.
4

The Gartner Group, Annual Survey of Information Technology Costs, 1996. 4

The need for learning organizations is driven by the increasing complexity, dynamism and competitiveness of the global economic environment. Excelling in todays dynamic organizational environment requires more understanding, knowledge, preparation, and agreement than one person's expertise and experience provides. According to David Garvin of Harvard University, "Continuous improvement requires a commitment to learning." 5 Specifically, creation of a learning organization requires the following actions:

Create continuous learning opportunity Promote inquiry and dialogue Encourage collaboration and team learning Establish systems to capture and share learning Empower people towards a collective vision Connect the organization to its environment

The concept of a learning organization provides the foundation for a paradigm shift in the concept of training. In North America in 1995, more than $55.3 billion was spent on formal training and development of employees6. It is estimated that on average, only 10 to 20 percent of training transfers to the job leading to increased employee performance, and thus, organizational performance7. Learning organizations are not about training.8 In fact, many managers believe training has failed to prepare employees for the future and harness organizational knowledge. Managers are demanding that training be held accountable, with specific measurable contributions to organizational performance.

The new paradigm of performance technology emphasizes the support of job performance, with the establishment of specific measures. The new approach establishes linkage to organizational performance measures during the performance technology design process through the use of performance-centered

David Garvin, Building a Learning Organization, Business Credit, 96(1): 19-28. January 1994.

American Society of Training & Development (ASTD), Training & Development Magazine, September 1996. Barry Raybould, EPSS: Unlocking Its Potential in Your Organization, Technical & Skills Training February/ March 1996 Diane Gayeski, From Training Department to Learning Organization. Performance & Improvement, Volume 35 #7, August 1996. 5
8 7

design. The current definition of electronic performance support by Barry Raybould represents this new paradigm:

An electronic infrastructure that captures, stores, and organizes corporate knowledge assets throughout an organization, enabling individuals to achieve required levels of performance in the fastest possible time and with a minimum of support from the people. Performance is achieved by designing the computer/human interface using the principals of Performance-Centered Design (PCD), which focuses on the audiences as performers of work, rather than as users of a system. (Raybould, 1996)

Measurement: The Missing Link in Performance Technology A key element of performance technology is performance measurement. Indeed, improvement is often impossible or ineffective in the absence of measurement. Measurement provides the benchmark against which continuous process improvement may occur.

Until recently, performance technology focused on traditional training. The measurement of training effectiveness had been conducted in relative isolation from organizational performance measurement. With the rapid evolution and convergence of training and performance technologies, there is a renewed potential for establishing a direct relationship between performance technology and organizational effectiveness. Specifically, the shift away from traditional training to electronic performance support (EPS) is forcing the integration of the measurement effectiveness of performance technology and organizations.

Performance versus Learning Focus The key to this shift is the nature of performance technology. According to Raybould, performance technology can be designed with the primary objective of transferring knowledge or of producing performance.9 In the performance support literature it has been shown that it is possible to design an intervention that can generate performance without pre-requisite learning. Learning often occurs as a by-product but it is not the primary objective.

Barry Raybould. The Role of Technology in Improving Performance. EPSS.COM, November

1997. 6

Trends in Performance Measurement The renewed emphasis on performance measurement in the private and public arena, including the passage of the Government Performance & Results Act of 199410, has sought to establish a public strategic planning process driven by a balanced and comprehensive set of performance measures. This new planning process is being referred to as Performance Management (PM).

At the same time, there has been an historic shift in the traditional paradigm of training design. Until recently, a linear process referred to as Instructional Systems Design (ISD) created training systems focused on the achievement of learning objectives. These learning objectives were typically not directly related to the principal organizational performance measures. The new paradigm, referred to as Performance-centered Design (PCD), argues that all training and performance support be driven by specific measurable performance outcomes, thus providing continuous and seamless integration of all human performance development activities in the strategic planning process. Recently, there has been an additional shift which says that performance-centered design should be renamed customer-centered design, where all support should directly impact the cost, quality, effectiveness of a product or service delivered to the ultimate customer.

Performance Technology Measures in The Studies Reviewed Our literature survey indicated a greater focus on establishing measures with a direct relationship to organizational performance. We reviewed three articles which studied the effectiveness of performance technology.

According to Banerji, measures of the effectiveness of performance technology can include:11 error rate time to complete a task the quality of outcome cost of task execution

Public Law: 103-62, An Act to provide for the establishment of strategic planning and performance measurement in the Federal Government, and for other purposes. Banerji, Ashok. Electronic Performance Support Systems. Proceedings of International Conference on Computers in Education (ICCE 95), 1995. 7
11

10

According to Krein and Maholm, they used Mean test Scores and Time to Complete. They also referred to Travel and Per Diem costs. These measures reflect the learning centered paradigm, and have little direct relationship to organizational performance.12

According to Gery, measures may include reduced initial training time, task accuracy, time on task, time to mastery and the ability to achieve day one performance (acceptable performance in the first day of work). Mackay used error rate , task time per task , initial training time and on-the-job training time. 13

Most of these measures addressed individual performance, although most could be extrapolated to an organizational performance measure. A limitation of most of these studies was that they did not attempt to address the other benefits of performance technology, including greater cross-utilization of personnel, ability to quickly reassign or replace personnel, and the benefits of a corporate knowledge base which captures the expertise of the workforce.

The Relationship of Performance Technology Performance Measurement to Organizational Strategic Processes Over the last five years, several new management planning and process improvement methodologies have emerged. These include: Performance Management Business Process Re-engineering Activity-based Management

In the arena of training and performance support, a new paradigms of Performance-centered Design14

Krien, Theodore and Maholm, Timothy. CBT has the Edge in a Comparative Study. Performance & Instruction. August 1990. p.23.
13

12

Mackay, Betty. EPSS technology in Banking. Interactive '94 (proceedings). 1996.

Gloria Gery, Attributes and Behaviors of Performance-Centered Systems, Performance Improvement Quarterly, Vol 8 No.1, 1995. 8

14

and Performance Support Engineering15 have challenged the traditional linear instructional systems design model.

All of these focus on measuring and enhancing some aspect of organizational performance. Although these concepts have developed within different communities, they all possess common elements of measurement. These methodologies are briefly described below.

Performance Management Performance measurement became central to effective public strategic planning. With the passage of the Government Performance and Results Act of 199316 GAO published an executive guide on implementing the Government Performance and Results Act (GPRA), which describes the focus of the law. Specifically, the act established the following requirements: (1) GPRA forces federal agencies to focus on their missions and goals, how to achieve them, and how to improve their structural organizations and business processes; (2) agencies must define their missions and desired outcomes, use strategic planning, involve stakeholders, assess their environments, and align their activities, core processes, and resources to support

mission-related outcomes; (3) agencies need to measure their performance to ensure that they are meeting their goals and making informed decisions; (4) performance measures need to be based on program-related characteristics and performance data must be sufficiently complete, accurate, and consistent; (5) agencies must use performance data to improve organizational processes, identify performance gaps, and set

improvement goals; and (6) GPRA success depends on strong leadership practices that devolve decisionmaking authority with accountability, create incentives, build expertise, and integrate reforms17. management

Performance Management is a broader concept which has evolved from GPRA implementation. Performance Management incorporates the establishment of performance measures under GPRA, but also

Barry Raybould, Performance Support Engineering: An Emerging Methodology for Enabling Organizational Learning, Performance Improvement Quarterly, Vol 8 No.1, 1995: 1. Public Law: 103-62, An Act to provide for the establishment of strategic planning and performance measurement in the Federal Government, and for other purposes. Executive Guide: Effectively Implementing the Government Performance and Results Act., 06/01/96, GAO/GGD-96-118) 9
17 16

15

addresses the need for continuous process improvement, which uses data from good performance measurement.

To be effective, the performance-centered design process must start with organizational performance objectives produced from performance planning. Performance technologists may also want to graphically map the relationship between performance technology measures of effectiveness and organization measures.

Functional Process Improvement & Business Process Re-engineering (BPR) According to Michael Hammer, BPR is:

The fundamental rethinking and radical redesign of business processes to achieve dramatic improvements in critical, contemporary measures of performance, such as cost, quality, service and speed.18

The BPR concept specifically incorporates performance measurement, but also identifies the tools to conduct process redesign. The concept of performance -centered design represents a new set of tools that can help focus training resources on achieving organizational objectives. Performance technology is now just another technology option for process improvement, in the context of BPR. A missing element has been evaluation techniques for decision makers.

Our literature review identified a paper which addressed this issue. Thomas, Baron and Schmidt identified five evaluation and measurement techniques to assess the feasibility and effectiveness of performance technology, in the context of BPR: 19 These included Work Profile Analysis, Direct to Indirect Ratio, Time Saved Times Salary (TSTS), Activity Based Costing (ABC) and Quality Assessment.

Michael Hammer. Reengineering Work: Dont Automate, Obliterate. Harvard Business Review. July/August 1990. pp.104-105. Thomas, B., Baron, J. and Schmidt, W.. Evaluating a Performance Support Environment for Knowledge Workers . USACERL Technical Report 95/32 , September 1995
19

18

10

This research offered a different perspective on measurement, focusing on predictive methodologies for decision makers.

Activity-based Management Activity-based Management (ABM) is business management in which process owners have the responsibility and authority to control and improve operations, and that uses Activity-based Costing (ABC). ABC is a set of management information and accounting methods used to identify, describe, assign costs to, and otherwise report on the operations in an organization. Performance measurement data is of little use without accurate representation of costs on an activity basis, which is the level at which process improvement occurs.

A well-designed ABM/ABC systems directly supports appropriate operational and financial performance measurements, provide cost information for business decisions such as make vs. buy, outsourcing, product pricing, and capital investment justification, and provides timely and actionable information for management. Specifically, ABC produces the following data related to performance measurement 20: Activity Cost - Value added, Cost of Control, Cost of Quality Performance Measurement - Unit Cost, Quality Business Process Analysis - Process Flow, Cost Drivers, Cycle Time, Process Control Business Process Reengineering - Selection, Process Scope, Process Modeling Benchmark Activities - Organization, Best in Class, Competition Planning and Budgeting - Workforce, Target Costing, Core Competencies

Paula Spinner. Using Activity-Based Management to Improve Operations. Department of Defense, 1996, p. 27. 11

20

Until recently, there have been limited measures of activity costs. With Activity-based management, there is a methodology for measuring the cost of specific activities. By combining performance-centered design with activity-based cost information, training and performance support development cost-effectiveness can potentially be measured on a task or activity basis. This provides an important link between the measurement of performance technology and organizational effectiveness.

Input

General Ledger Costs

Trace

Activity Analysis

Resources
Trace

Input

ABC Model
Input Output Input Activity Costs Process Costs Product Costs Unit Costs

Activities
Trace

Outputs

Figure 3. Activity Based Cost Process Model

Are the Emerging Performance Technology Development Methodologies Compatible? This paper evaluated the two emerging training development methodologies as described in the research literature, including: Performance-centered Design Performance Support Engineering Each of these is critically evaluated relative to its potential compatibility with Performance Management, and thus its ability to support direct measurement of performance technology effectiveness.

Performance Centered Design (PCD) Performance-centered design is a development methodology which defines performance outcomes 12

(measures) early in the

training and information systems development process. By defining these

performance outcomes, the complete design process focuses on achieving these specified outcomes, and not some intermediate training effectiveness outcome.

Thus, a level of job or task performance is specified, and the training and performance support system is customized to achieve that result. PCD is an umbrella concept that can result in a solution of job aids, electronic performance support tools, computer-based training, video teletraining or traditional classroom training, depending on the nature of the task to be supported.

An underlying element of the philosophy of PCD is that learning is most effective when performed in the context of actual work. Many organizations report that 85-90% of a person's job knowledge is learned on the job, and only 10-15% is learned in formal training events. PCD recognizes this reality21.

The PCD concept is focused on the integration of training and performance support with information systems and applications. PCD infuses information systems with knowledge, structures tasks, and enables performers to achieve the required level of performance as quickly as possible with minimum support from other people22. Software that is designed around performance is intuitive to its users and enables them to perform their normal work with obvious gains in speed and efficiency without ever attending training classes or looking things up in books. It reflects their own conceptualization of their work and incorporates their language, idioms, metaphors, and understanding of how to perform tasks.

According to Gery, the focus of PCD is on the efficient performance of individual tasks23. Thus, PCD establishes performance measures at a lower level than those measures established under the concept of performance management. To be compatible, the measures established must be coordinated. Figure 1 displays the relative focus of PCD and performance management.

Barry Raybould. Performance Support Engineering: An Emerging Methodology for Enabling Organizational Learning, Performance Improvement Quarterly, Vol 8 No.1, 1995: p.2.
22

21

Craig Marion. What is Performance-Centered Design?.WWW.EPSS.COM. p.2. Gloria Gery, Performance Support: Performance Centered Design. Gery Associates, 1995.

23

13

PCD by its very nature results in greater congruence with organizational performance measures because the task analysis targets a level of task performance. The task performance is based on a desired outcome for the organization as a whole.

Winslow and Bramer of Andersen Consulting explain it like this24:

Traditional transaction-based systems are designed with an emphasis on process and data modeling. In a way, they are designed inside-out. The user interface modeling is derived from the process and data structures. For example, the layout of screens is often a reflection of a record structure, and the menus of the system reflect the functional structure.

PERFORMANCE MEASUREMENT UNDER GPRA

PERFORMANCE MEASUREMENT UNDER PCD

Performance systems are more and more interactive, oriented work Thus,
OUTPUTS OUTCOMES MISSION

ORGANIZATION OUTPUT

toward

actual

circumstances.

these systems need to be designed from the outside-in. The character of process and data modeling in itself does not change so much, but the (outside) user

ACTIVITY OUTPUT

INDIVIDUAL OUTPUT PERFORMANCE MANAGEMENT

interface drives the

modeling (internal)
INPUTS

TASK OUTPUT
Figure 4. Comparison of Performance Measurement Granularity

process modeling and data modeling.

C. Winslow & W. Bramer. Future Work: Putting Knowledge to Work in the Knowledge Economy. NY: The Free Press, 1994. 14

24

According to Gery, When designers have the point of the view of the performer situated in a real work context, success is inevitable. If the point of view does not closely match the situation, usability and performance problems are inevitable."25

According to Gery, PCD is about human performance, not system performance. In fact, the philosophy of PCD is about targeting the attention of software designers away from systems, and on job performance.

Performance Support Engineering A second major emerging training & performance development methodology is performance support engineering. This concept was introduced by Dr. Barry Raybould. In a paper entitled Performance Support Engineering: An Emerging Methodology for Enabling Organizational Learning, Raybould presents a conceptual model called the Organizational Performance/Learning Cycle26. This model describes the dynamics of the organizational learning process and provides a framework for thinking about the technologies and methodologies that enable that process. Raybould argues that previous definitions of EPSS were too limited and restricted in their scope and expands on these earlier definitions to take into account this new model of organizational learning. The new definition clearly distinguishes EPSS from traditional systems development

According to Raybould, this new view of EPSS offers a clear opportunity for organizations to recognize the strategic importance of managing their knowledge assets. Most of the current methodologies in use by different functional groups are limited in their capability to enable this model and the expanded definition of EPSS. The Performance Support Engineering methodology is intended to overcome these limitations.

Rayboulds central thesis focuses on broader strategic aspects of performance design than PCD. He recognizes that there are synergies to be realized from creating a learning organization, and that knowledge is a strategic asset. For example, the new concept of dynamic performance support, where the knowledge and task input are captured fall within the broader concept of performance support engineering.

25

Gloria Gery. Performance Support: Performance Centered Design. Gery Associates, 1995.

Barry Raybould. Performance Support Engineering: An Emerging Methodology for Enabling Organizational Learning, Performance Improvement Quarterly, Vol 8 No.1, 1995: p.2. 15

26

Raybould proposes a broader definition of performance support systems based on a systems approach.

An Electronic Performance Support System (EPSS) is the electronic infrastructure that captures, stores and distributes individual and corporate knowledge assets throughout an organization, to enable individuals to achieve required levels of performance in the fastest possible time and with a minimum of support from other people.

Rayboulds revised image of performance support envisions a dynamic performance support environment where the performance support system not only provides access to knowledge, but captures and disseminates new knowledge.

The implications of performance support engineering for performance measurement are significant. First, there are no incompatibilities between performance centered design and performance support engineering. Both may be applicable. PCD addresses the task level of performance support. Performance support engineering, on the other hand, addresses a new set of organizational performance measures which address an organizations learning capability. For example, these measures might address how quickly information is captured and disseminated.

An Integrated Model of Performance Technology & Organizational Performance Planning and Management Based on our research, Figure 5 displays a proposed integration model for performance technology and organization performance planning and management. From our literature survey, it was clear that while both address performance, the perspective is at completely different level. Performance Technology addresses individual and group performance, which performance management and strategic planning address organizational performance. The two are compatible, but it requires a better understanding on the part of both to ensure goal congruence.

16

MISSION

OUTCOMES

ORGANIZATION OUTPUT

ACTIVITY OUTPUT

Strategic Planning

Performancecentered Design

INDIVIDUAL OUTPUT

TASK OUTPUT

Activity Based Management

Performance Management

1. Identification - What are our opportunities?

2. Change - How can we affect change?

3. Measurement - How are we doing?

CONTINUOUS PROCESS IMPROVEMENT

INPUTS
Figure 5. An Integrated Model of Performance Technology & Organizational Performance Management and Planning

17

The Research Literature: How Effective is Performance Technology? We reviewed three research articles to identify data on the effectiveness of performance technology. The consensus of the research is that performance technology can offer specific measurable benefits to organization effectiveness.

In one of the only control studies on performance supports systems, Banerji compared the performance of different groups in performing a defined sets of information systems tasks. He provided various support tools (on-line help, expert user) and developed two electronic performance support systems variants. His conclusions were that EPSS technology : Leads to 25%-40% savings in cost Decreases 20%-50% training time Decreases delivery time Decreases reading time Leads to 33% reduction in paper documentation Increases memory and retention by 16%-24% Increases productivity by 25%
(c) Dr Ashok Banerji Singapore Polytechnic ICCE 95

Performance Time With Various Supports

Control Groups
Time in Minutes E1 Expert N0 Novice
30 25 20 15 10 5

30%-100%

training

20%-40%

document

Test Groups

N1 Paper docs N2 Expert users help N3 Text-based EPSS N4 Multimedia EPSS

E1

N1

N2 N3 Groups

N4

N0

Figure 6. Performance Time for Various Forms of Support

Figure 6 displays the results of the control groups performance for various measures of effectiveness.

In a control study by Krein and Maholm, computer-based training demonstrated higher test scores and reduced training time. Figure 7 displays the results.

Average Time to Complete Classroom Group (n=61) CBT Group (n=29) 3 hours, 52 minutes

Mean Score on Final Test 13.4

2 hours, 48 minutes 15.6 Figure 7. Effectiveness of CBT vs. Classroom Training

18

Figure 5. Memory Rentention Rates

In a study by Mackay, she testing an EPSS-style front end for a legacy system in bank operations, tested on groups of new and experienced employees. The results included: reduced errors (73% for existing employees, 87% for new), reduced time per task (33% for existing employees, 77% for new), reduced training and on-the-job training time (75% reduction in on-the-job training time).27

Conclusions The emergence of performance planning and performance-centered design methodologies offers an opportunity to increase the accountability of training functions within organizations. The following conclusions are suggested, based on the review conducted and the research questions posed::

There are no inherent incompatibilities between performance management and the emerging training and performance support development methodologies.

The concept of performance-centered design is narrow in scope, addressing performance measures for the performance of specific tasks.

Performance-centered design and performance management functions should coordinate design activities to ensure the compatibility of measures selected.

Performance-centered design and performance support engineering are not mutually exclusive. Both can proceed at the same time.

New measures may be required to address the effectiveness of learning organizations. Performance-centered design can ensure goal congruence between training system design and organizational performance measures.

The performance system developed may incorporate data collection tools to support the performance measurement process.

To achieve this paradigm shift, the strategic planners, trainers and systems developers must coordinate the establishment of performance measures.

There are limited empirical studies of the effect of training and performance support technologies on organizational effectiveness.

The studies that have been conducted indicate substantial increases in productivity and reductions in cost.

27

Mackay, Betty. EPSS Technology in Banking. Interactive '94 (proceedings). 1994. 19

References *"The Learning Organization." Chief Executive. #101 (March 1995): 57-64.

*Chalofsky, Neal E. "A New Paradigm for Learning in Organizations." Human Resource Development Quarterly. 7 #3 (Fall 1996): 287-293.

*DiBella, Anthony J. "Developing Learning Organizations: A Matter of Perspective." Academy of Management Journal. Best Papers Proceedings (1995): 287-290.

*Garratt, Bob. "An Old Idea That Has Come of Age." People Management. 1 #19 (September 21, 1995): 25-28.

*Gephart, Martha A. and others. "Learning Organizations Come Alive." Training & Development. 50 #12 (December 1996): 34-36+.

*Kaplan, Robert S. and David P. Norton. "Strategic Planning and the Balanced Scorecard." Strategy & Leadership. 24 #5 (September/October 1996): 18-24.

*Kuchinke, K. Peter. "Managing Learning for Performance." Human Resource Development Quarterly. 6 #3 (Fall 1995): 307-316.

*Rheem, Helen. "The Learning Organization." Harvard Business Review. 73 #2 (March/April 1995): 10.

*Schein, Edgar H. "Three Cultures of Management: The Key to Organizational Learning." Sloan Management Review. 38 #1 (Fall 1996): 9-20.

*Senge, Peter. "Learning to Alter Mental Models." Executive Excellence. 11 #3 (March 1994): 16-17.

*Senge, Peter. "The Learning Organization Made Plain." [interview] Training & Development. 45 #10 (October 1991): 37-44.

*Senge, Peter. "Mental Models." Planning Review. 20 #2 (March/April 1992): 4-10+. 20

*Senge, Peter. "The Message of the Quality Movement." Executive Excellence. 12 #7 (July 1995): 5-6.

*Shaw, Diana V. and others. "Learning from Mistakes." Quality Progress. 28 #6 (June 1995): 45-48.

American Society of Training & Development (ASTD), Training & Development Magazine, September 1996.

Anthony, G. Michael, "IE's Measure Work, Write Standards for White Collar Workers at Financial Institution," Issues in White Collar Productivity (Industrial Engineering and Management Press, Institute of Industrial Engineers, 1984), pp 84-87.

Argyris, Chris. "Teaching Smart People How to Learn." Harvard Business Review. 69 #3 (May-June 1991): 99-109.

Argyris, Chris. Overcoming Organizational Defenses: Boston: Allyn & Bacon, 1990.

Argyris, Chris. Organizational Learning II: Theory, Method, And Practice. Reading, MA: Addison-Wesley, 1996.

Argyris, Chris. On Organizational Learning. Cambridge, MA: Blackwell, 1993.

Argyris, Chris. (1994), Good Communication that Blocks Learning. Harvard Business Review, July-August. (HBR Reprint No. 94401)

Argyris, Chris. (1991), Teaching Smart People How to Learn. Harvard Business Review, May-June. (HBR Reprint No. 91301)

Banerji, A.K.,(1995). Designing Electronic Performance Support Systems, PhD Thesis, School of Computing, University of Teesside, Cleveland, UK.

Banerji, Ashok. Electronic Performance Support Systems. Proceedings of International Conference on Computers in Education (ICCE 95), 1995. 21

Barker, P.G. (1993). Exploring Hypermedia, Kogan Page, London.

Barker, P.G. and Banerji, A.K. (1993). Designing Electronic Performance Support Systems, 140-142, Proceedings of the 10th International Conference on Technology and Education, Boston, USA.

Barker, P.G. and Banerji, A.K. (1995). Designing Electronic Performance Support Systems, Educational and Training Technology International, Special Edition on Electronic Performance Support Systems, 32(1).

Barker, P.G. and Banerji, A.K. (1994). Evolving Principles of Performance Support, Association for Learning Technology Conference'94, Enabling Active Learning, University of Hull, 19-21 September.

Bernard, Paul, "Structured Project Methodology Provides Support for Informed Business Decisions," IE (March 1986), pp 52-57.

Beruvides, M.G., and D.J. Sumanth, "Knowledge Work: A Conceptual Analysis and Structure," Productivity Management Frontiers-I (Elsevier Science Publishers B.V., 1987), pp 127-138.

Bohl, Don Lee, ed. The Learning Organization in Action. New York: American Management Association, 1994.

Brassard, Michael, The Memory Jogger-A Pocket Guide of Tools for Continuous Improvement (Goal/ QPC, 1988).

Brown, J.S. and Puguid, P. (1991), Organizational Learning and Communities of Practice. Organization Science, 2(1), 40-57. Drucker, Peter F. (1994), The Theory of the Business. Harvard Business Review, September-October.

Bussell, Linda. A Performance Technology Case Study: The Medisoft International/Omnicom Alliance. Performance & Instruction, Volume 34, #5. June 1995.

22

Chase, Nancy. Training effectiveness measures and scoring schemes: A Psychology. v44 n2, Summer 1991, p. 353.

comparison. Personnel

Chavalier, Roger. Improving Efficiency & Effectiveness of Training: A Six Year Case Study of Systemic Change. Performance & Instruction. .May/June 1996.

Chawla, Sarita and John Renesch, eds. Learning Organizations: Developing Cultures for Tomorrow's Workplace. Portland, OR: Productivity Press, 1995. . Blueprint for Quality : How Training Can Turn Strategy into Real Improvement. San Francisco: Jossey-Bass, 1993.

Cleary, B.A., "Company Cares About Customers' Calls," Quality Progress, vol 26, No. 11 (November 1993), pp 69-73.

Cox, Thomas, "The Myth of the Commodity Database or How To Pick the Best Technology for You," Oracle Integrator (January/February 1993), pp 19-21.

Davidove, Eric. Evaluating the Return on Investment in Training. Performance & Instruction. January 1993.

De Geus, Arie. (1988), Planning as Learning. Harvard Business Review, 66(2), March-April, 70-74.

Dennison, D. R., Corporate Culture and Organizational Effectiveness, (New York: NY: John Wiley, 1990).

Drucker, Peter F., Management (Harper & Row, 1974).

Duchastel, Philippe C.; Lang, Joyce. Raise your training ROI. Quality. v36n9. Sep 1997. p. 28-41.

Dumaine, Brian (1994) Mr. Learning Organization. Fortune, October 17, 147-157.

Executive Guide: Effectively Implementing the Government Performance and Results Act., 06/01/96, GAO/GGD-96-118) 23

Gery, G. (1991). Electronic Performance Support Systems: How and Why to Remake the Workplace through the Strategic Application of Technology. Boston, MA: Weingarten Publications.

Gery, G. Performance Support: Performance Centered Design. Gery Associates, 1995

Gery, Gloria, Attributes and Behaviors of Performance-Centered Systems, Performance Improvement Quarterly, Vol 8 No.1, 1995.

Gery, G. J. "Attributes and Behaviors of Performance-Centered Systems," Performance Improvement Quarterly, 8 (1), pp. 47-93, 1995.

Gery, G.J. (1991). Electronic Performance Support Systems - How and Why to Remake the Workplace Through the Strategic Application of Technology, Weiengarten Publications, Boston, USA.

Gore, A., From Red Tape to Results. Creating a Government That Works Better and Costs Less: Report of the National Performance Review, (Washington, DC: GPO, 1993)

Gore, A., From Red Tape to Results. Creating a Government That Works Better and Costs Less: Report of the National Performance Review, (Washington, DC: GPO, 1993)

Hammer, Michael and Champy, James, Reengineering the Corporation, 1993.

Hawkins, P. (1991), The Spiritual Dimension of the Learning Organization. Management Education and Development, 22(3), 172-187.

Helton, B. Ray, "Quality and the Bottom Line, Part 3: The Inner View," The Quarterly Observer (April 1992), p 3.

Helton, B. Ray, "Achieving White-Collar Whitewater Performance by Organizational Alignment," National Productivity Review (Spring 1991), pp 227-244.

Helton, B. Ray, "Quality and the Bottom Line, Part 2: The Company's Side," The Quarterly Observer (March 24

1992), p 5.

Helton, B. Ray, "More of the Right Stuff," The Quarterly Observer (September 1993), p 5. Liggettt, Trevino, and Lavelle, "Activity-Based Cost Management Systems in Advanced Manufacturing Environments," in Parsaei et al. (eds.), Economic and Financial Justification of Advanced Manufacturing Technologies (Elsevier, Amsterdam, 1992).

Hybert, Peter and Perry, Dreama. Beyond the Buzzwords- TQM, HPT, & Shareholder Value. Performance & Instruction. Volume 34, #2. February 1995.

Johnson, H. Thomas, "It's Time to Stop Overselling Activity-Based Concepts," Management Accounting (September 1992), pp 26-35.

Kelly, Robert, and Janet Kaplan, "How Bell Labs Creates Star Performers," Harvard Business Review (July-August 1993), pp 128-139.

Kim, D. H. (1993), The Link between Individual and Organizational Learning. Sloan Management Review, Fall, 37-50. (Reprint 3513)

Kirkpatrick, D. L. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers, 1994.

Krein, Theodore & Maholm, Timothy. CBT has the Edge in a Comparative Study. Performance & Instruction. August 1990.

Laffey, J., Machiraju, N. Rao, Chandhok, R. (1991). Integrated Support and Learning Systems for Augmenting Knowledge Workers. Proceedings of World Congress on Expert Systems, Orlando: Pergamon Press.

Laffey, J., Machiraju, N. Rao, Chandhok, R. (1991). Organizational Memory as a Support for Learning and Performance: Prototypes and Issues. Proceedings of the International Conference on the Learning Sciences, 1991. 25

Leeuw, Frans L.Can Governments Learn?: Comparative Perspectives on Evaluation & Organizational Learning. New Brunswick: Transaction Publishers, 1994.

Mader, R., (1992). Managing and Organising an Electronic Performance System Project, 10th Annual Computer-Based Training Conference and exposition, March 2-5, Orlando, USA.

Marquardt, Michael J. Building the Learning Organization. New York: McGraw-Hill, 1996.

McDermott, Richard. "Designing and Improving Knowledge Work." Journal for Quality and Participation. 18 #2 (March 1995): 72-77.

Noer, David M. Breaking Free: A Prescription for Personal and Organizational Change. San Francisco: Jossey-Bass, 1997.

McDonald, J. Performance support systems for learning. Journal of Educational Technology Systems v. 24 no1 ('95-'96).

McGraw, K.L., (1994). Performance Support Systems: Integrating AI, Hypermedia and CBT to Enhance User Performance, Journal of Artificial Intelligence in Education, 5(1), 3-26.

National Research Council, Information Technology in the Service Society (National Academy Press, Washington, DC, 1994).

Nay, D. C. The Effectiveness of a Multimedia Training System on Organizational Performance. International conference -- 1995 May : Atlanta; GA, Information Resources Management Association: Managing information and communications in a changing global environment. pp. 342-353.

Nevis, E. C., DiBella, A. J. and Gould, J. M. (1995), Understanding Organizations as Learning Systems. Sloan Management Review, Winter 1995, 73-85.

Nonaka, I. (1991), The Knowledge-Creating Company. Harvard Business Review, 69(6),96-104. 26

Norman, D. (1993). Things That Make Us Smart - Defending human attributes in the age of the machine. Reading, MA: Addison-Wesley Publishing Co.

Ostroff, Cheri.. Performance measurement and decision support systems. Management Accounting: Magazine for Chartered Management Accountants. v71 n10, Nov 1993, p. 10.

Parker, Marilyn M., Robert J. Benson, and H.E. Trainor, Information Economics Linking Business Performance to Information Technology (Prentice-Hall, Englewood Cliffs, NJ, 1988).

Partovi, "An Analytic Hierarchy Approach to Activity Based Costing," International Journal of Production Economics, vol 22 (1991), pp 151-161.

Partovi, "An Analytic Hierarchy Approach to Activity Based Costing," International Journal of Production Economics, vol 22 (1991), pp 151-161. Peterson, I., Probing a Computer Productivity Paradox," Science News (1 Jan 1994), p 7. Rittenhouse, Robert G. "Productivity and the Microcomputer," Management of Technology III (Institute of Industrial Engineers, 1992).

Phillips, J. J. Handbook of Training and Evaluation and Measurement Methods (2nd ed.). Houston, TX: Gulf Publishing Company, 1991.

Public Law: 103-62, An Act to provide for the establishment of strategic planning and performance measurement in the Federal Government, and for other purposes.

Puterbaugh, G. "CBT and Performance Support," CBT Directions, pp. 18-25, June 1990.

Ray, James and Sword, Stanley. Reengineering and Human Performance. Performance & Instruction. August 1993.

Raybould, B., (1990). Solving Human Performance Problems with Computers - A Case Study: Building an Electronic Performance Support System, Performance and Instruction, 29, 4-14.

27

Raybould, B. Info-Line: Making EPSS Work for Your Organization. Alexandria, VA: American Society for Training and Development, January 1995.

Raybould, Barry. (1994) Performance Support Engineering: An Emerging Development Methodology for Enabling Organizational Learning. Performance Improvement Quarterly 8(1) pp. 7-22.

The Effectiveness of Computer-based Training. Institute for Defense Analysis, 1991.

Raybould, B. (1990). Solving Human Performance Problems with Computers. Performance & Instruction, 29(11), 4-14.

Raybould, Barry. Solving Human Performance Problems with Computers- A Case Study: Building an Electronic Performance Support System. Performance Improvement Quarterly. November/December 1996.

Roach, Stephen, "Services Under Siege-The Restructuring Imperative," Harvard Business Review (September-October 1991), pp 82-83.

Robinson & Robinson, Training Magazine, March, 1996.

Sassone, Peter G., "Cost Benefit Methodology for Office Systems," ACM Transactions on Office Information Systems, vol 5, No. 3 (July 1987).

Sassone, Peter G.. "A Survey of Cost-Benefit Methodologies for Information Systems," Project Appraisal, vol 3, No. 2 (June 1988), pp 73-84.

Sassone, Peter G., "Office Productivity: The Impacts of Staffing, Intellectual Specialization and Technology" (The Georgia Institute of Technology-School of Economics, September 1991), pp 1-35.

Sassone, Peter G., "Survey Finds Low Office Productivity Linked to Staffing Imbalances," National Productivity Review (Spring 1992), pp 147-158.

Sassone, Peter G., "Three Approaches for Estimating the Value of Office Work," Office Technology and 28

People, vol 6, No. 1 (1992).

Senge, Peter. The Fifth Discipline: the Art and Practice of the Learning Organization. New York: Doubleday, 1990.

*Senge, Peter. "Leading Learning Organizations." Training & Development. 50 #12 (December 1996): 36-37.

Senge, P. (1990). The Fifth Discipline - The Art and Practice of the Learning Organization. New York, NY: Doubleday Currency.

Senge, Peter. (1992), Building Learning Organizations., Journal for Quality and Participation, March.

Shneiderman, B. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Reading, MA: Addison-Wesley Publishing Company, 1987.

Silberman, M., and Auerbach, C. Active Training: A Handbook of Techniques, Designs, Case Examples and Tips. New York, NY: Lexington Books, 1990.

Stata, R. (1990), Organizational Learning: The Key to Managment Innovation. Sloan Managment Review, 30(3), 63-74.

Stone, Deborah and Villachica, Steven. Performance Support for Knowledge Workers: Practical Strategies based on Research and Practice. Performance Improvement Quarterly, Volume 36, #3, March, 1997.

Tate, William. Developing Corporate Competence: A Performance Agenda for Managing Organizations. Brookfield, Vt: Gower 1995.

Thomas, B., Baron, J. and Schmidt, W., USACERL Technical Report 95/32 , Evaluating a Performance Support Environment for Knowledge Workers September 1995.

Thomas, B.E., and J.P. Baron, Evaluating Knowledge Worker Productivity: Literature Review, Interim Report (IR) FF-94/27 (U.S. Army Construction Engineering Research Laboratories [USACERL}, June 1994). 29

Today's Need for Viable Training Measures of Effectiveness. CONFERENCE: 13th -- 1991 Dec Orlando; FL, Interservice/industry training systems conference, 1991 , pp. 401-405

Weston, D. M. (1994). Organizational Learning in Practice. (Publication D94-1817) Menlo Park, CA: SRI International.

Weston, D. M. (1994), Organizational Learning as Strategy. (Publication R827) Menlo Park, CA: SRI International.

Wikoff, Martin. The Quality Movement Meets Performance Technology. Performance & Instruction. Volume 3, #8. September 1994.

Winslow, C. & Bramer, W. (1994). Future Work: Putting Knowledge to Work in the Knowledge Economy. NY: The Free Press.

Wood, Del. A Framework for Re-engineering Traditional Training: Interactive Training and Performance Support. Performance Improvement Quarterly, Volume 35, #8, September 1996.

30