EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

Edition Edition Date Status Class

: : : :

0.1 30 Nov 2001 Draft General Public

EUROPEAN AIR TRAFFIC MANAGEMENT PROGRAMME

DOCUMENT IDENTIFICATION SHEET

DOCUMENT DESCRIPTION
Document Title KPI Measurement, Monitoring and Analysis Guide

EWP DELIVERABLE REFERENCE NUMBER PROGRAMME REFERENCE INDEX AIM/AEP/S-LEV/0008 EDITION : EDITION DATE : Abstract This guide provides an introduction to Key Performance Indicator (KPI) measurement, monitoring and analysis. 0.1 30 Nov 2001

Quality Management KPI Monitoring CONTACT PERSON :

Service Level Measurement

Keywords Performance Data Collection

Indicator Analysis

P. Bosman

TEL : 3333

UNIT :

DOCUMENT STATUS AND TYPE STATUS Working Draft Draft Proposed Issue Released Issue CATEGORY Executive Task Specialist Task Lower Layer Task CLASSIFICATION General Public EATMP Restricted

o o o

o o

o o

INTERNAL REFERENCE NAME :

AHEAD Electronic Filing System

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

DOCUMENT APPROVAL The following table identifies all management authorities that have successively approved the present issue of this document.

AUTHORITY

NAME AND SIGNATURE Ertan Ozkan

DATE 30 Nov 2001

Author/Editor

Conrad Cleasby Programme Manager

30 Nov 2001

Quality Assurance

Page iv

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

DOCUMENT CHANGE RECORD The following table records the complete history of the successive editions of the present document.

EDITION 0.1

DATE 30 Nov 2001 Creation

REASON FOR CHANGE

SECTIONS PAGES AFFECTED All

Edition: 0.1

Draft

Page v

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Page vi

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

TABLE OF CONTENTS

1. 1.1 1.2 2. 2.1 2.2

INTRODUCTION ...................................................................................................... 1 Purpose and scope ................................................................................................ 1 References.............................................................................................................. 1 DATA COLLECTION ............................................................................................... 2 Data Collection Plan............................................................................................... 2 Measurement Techniques...................................................................................... 2 2.2.1 2.2.2 2.2.3 2.3 Event-Driven Measurement ........................................................................ 3 Sampling-Based Measurement................................................................... 5 Simulation................................................................................................... 5

Designing a Data Collection System..................................................................... 5 MONITORING .......................................................................................................... 7 Operational Report ................................................................................................. 7 Real-Time Reports.................................................................................................. 8 Executive Summaries............................................................................................. 8 Customer Reports .................................................................................................. 8 ANALYSING AND INTERPRETING KPIS ............................................................... 9 Variations (Trend) Analysis ................................................................................... 9 Identifying Relationships (Cause-Effect Analysis)............................................. 11 4.2.1 4.2.2 Scatter Diagrams ...................................................................................... 11 Stratification.............................................................................................. 12

3. 3.1 3.2 3.3 3.4 4. 4.1 4.2

4.3 4.4 4.5 4.6 5. A.1

Capability Analysis............................................................................................... 13 Capacity Analysis................................................................................................. 13 Considering Context ............................................................................................ 14 Establishing Priorities.......................................................................................... 15 CONCLUSIONS AND FUTURE WORK................................................................. 16 Cause and Effect Diagram ................................................................................... 17 A.1.1 A.1.2 A.1.3 Usage ....................................................................................................... 17 Steps ........................................................................................................ 17 Example.................................................................................................... 19 Usage ....................................................................................................... 19 Steps ........................................................................................................ 19 Example.................................................................................................... 20 Usage ....................................................................................................... 20 Step-by-Step Example .............................................................................. 21
Draft Page vii

QUALITY TOOLS................................................................................................................ 17

A.2

Check Sheet.......................................................................................................... 19 A.2.1 A.2.2 A.2.3

A.3

Pareto Chart.......................................................................................................... 20 A.3.1 A.3.2

Edition: 0.1

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

A.4

Scatter Diagram.................................................................................................... 22 A.4.1 A.4.2 A.4.3 Usage ....................................................................................................... 23 Steps ........................................................................................................ 23 Example.................................................................................................... 24 Usage ....................................................................................................... 24 Steps ........................................................................................................ 24 Usage ....................................................................................................... 25 Steps ........................................................................................................ 25 Usage ....................................................................................................... 26 Steps ........................................................................................................ 26 Example.................................................................................................... 26 Usage ....................................................................................................... 27 Steps ........................................................................................................ 27 Usage ....................................................................................................... 27 Steps ........................................................................................................ 28

A.5

Run Chart.............................................................................................................. 24 A.5.1 A.5.2

A.6

Affinity Diagram.................................................................................................... 25 A.6.1 A.6.2

A.7

Nominal Group Technique................................................................................... 26 A.7.1 A.7.2 A.7.3

A.8

Interrelations Digraph .......................................................................................... 26 A.8.1 A.8.2

A.9

Force Field Analysis............................................................................................. 27 A.9.1 A.9.2

A.10 Matrix Diagram ..................................................................................................... 28 A.10.1 Usage ....................................................................................................... 28 A.10.2 Steps ........................................................................................................ 28

Page viii

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

1. 1.1

INTRODUCTION Purpose and scope

This guide explores issues related to Key Performance Indicator (KPI) measurement, monitoring and analysis. The main objective is to provide necessary background required for deploying a performance measurement system within the context of quality and service level management. This guide is organised as follows: Chapter 2 presents the issues related to data collection: brief summary of data collection, data collection plan and measurement techniques. In the Chapter 3 how to monitor KPIs is discussed and particularly reporting issues are explored. Chapter 4 gives some guidelines in order to analyse your KPIs. Conclusion and future work are given in the chapter 5. Appendix A contains summary of quality tools that may be useful in the deployment of a performance measurement system.

1.2

References
[1] Foundations of Service Level Management, April 2000, Rick Sturm, Wayne Morris and Mary Jander, SAMS Publications, ISBN 0-672-31743-5. [2] Change Management the 5-step action kit, C. Rye, ISBN 0749433809, Kogan Page Limited, 2001. [3] Operational Performance Measurement Increasing Total Productivity, W. Kaydos, ISBN 1574440993, CRC Press LLC, 1999. [4] Applications of Performance Measurement, Paul Arveson, The Balanced Scorecard Institute, http://www.balancedscorecard.org/appl/index.html, 1998. [5] Basic Tools for Process Improvement, Module 7, Data Collection, 1996.

Edition: 0.1

Draft

Page 1

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

2.

DATA COLLECTION
Data collection helps you to assess the health of your system and processes. To do so, you must identify the Key Performance Indicators (KPIs) you will measure, how you will measure them and what you will do with the data you collect. Every improvement effort relies on data to provide a factual basis for making decisions for improvement. Data collection enables a team to formulate and test working assumptions and develop information that will lead to the improvement of the KPIs of the product, service or system. Data collection improves your decisionmaking by helping you focus on objective information about what is happening, rather than subjective opinions. In other words, I think the problem is... becomes The data indicate the problem is... To collect data uniformly, you will need to develop a data collection plan. The elements of the plan must be clearly and unambiguously defined. Data collection can involve a multitude of decisions by data collectors. When you prepare your data collection plan, you should try to eliminate as many subjective choices as possible by operationally defining the parameters needed to do the job correctly. Your data collectors will then have a standard operating procedure to use during their data collection activities.

2.1

Data Collection Plan

While preparing data collection plans you have to answer the following questions: Why do we want the data? What will we do with the data after we have collected them? You must decide on a purpose for collecting the data. This action focuses you on the specific quality characteristic you want to improve, and sets the stage for where you will collect the data. Where will we collect the data? The location where data are collected must be identified clearly. Who will collect the data? The answer is simple: Those closest to the data (e.g., the process workers) should collect the data. These people have the best opportunity to record the results. They also know the process best and can easily detect when problems occur. But remember, the people who are going to collect the data need training on how to do it and the resources necessary to obtain the information such as time and measurement tools. How do we collect the right data? You have to specify the amount and frequency of data collection. However you need to remember that you are collecting data for the purpose of future improvement efforts. Therefore you have to take into consideration the cost of obtaining the data, the availability of data and the consequences of decisions made on the basis of the data when determining how much data should be obtained and how frequently it should be collected.

2.2

Measurement Techniques
There are three main techniques for collecting data: Event-Driven Measurement Sampling Based Measurement

Page 2

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

Simulation 2.2.1 Event-Driven Measurement In case of event-driven measurement the times at which certain events happen are recorded and then desired statistics are computed by analysing data. Sometimes this is called as event management in the literature. Although the event-driven measurement varies from company to company, there are three distinct and common steps: To detect events (e.g., failure of a computer, error in a publication) To record time and nature events (e.g., to record time and nature of computer failure or error in a publication) To take corrective actions by using procedure that outline how the event should be managed (e.g., to make the computer up and running again or to correct error in a publication) Events can be detected and recorded by Agents Human beings Agent. An agent is a piece of software designed to collect data about the status and functionality of a device, system or application for reporting purposes. Among many tools the popular examples are First Sense, Empire, OpenView (HP), etc. Agents capture data directly from the hardware elements underlying the service (network, bridges, routers, switches, hubs, etc.) or they gather input from software programs that affect overall service availability (applications, databases, middleware, etc.) They report events directly as they occur. Examples include hardware and software failures, broken routers, etc. Human Beings. In this case the event is detected by the people involving in the process. The recording is generally done by using checksheets. Checksheets are structured forms that enable people to collect and organise data systematically. Checksheets may be computerised (e.g., workflow management or document management systems contain similar forms). Because each checksheet is used for collecting and recording data unique to a specific process or system, it can be constructed in whatever shape, size and format are appropriate for the data collection task at hand. There is no standardised format that you can apply to all checksheets. Instead, each checksheet is a form tailored to collect the required information. However you may use the following guidelines while developing useful checksheets: Involve the process workers in developing the checksheet for their process. Label all columns clearly. Organise your form so that the data are recorded in the sequence seen by a person viewing the process. This reduces the possibility of data being recorded in the wrong column or not being recorded. Make the form user-friendly. Make sure the checksheet can be easily understood and used by all of the workers who are recording data.
Edition: 0.1 Draft Page 3

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Week 14- 2001 Monday Tuesday Wednesday Thursday Friday Total 1. Input Hopper Faults 5 2. A4 Paper Tray Fault 10 3. A3 Paper Tray Fault 3 4. Internal Paper Jams 5. Duplex Processing Faults 6. Stapling Faults 7. Output Collation Faults 2 25 5 0

1 7 6 4 5 2
Figure 1. A Sample Checksheet for a Photocopy Machine

Create a format that gives you the most information with the least amount of effort. For example, design your checksheet so that data can be recorded using only a check mark, slant mark, number or letter. Provide enough space for the collectors to record all of the data. Designate a place for recording the date and time the data were collected. These elements are required when the data are used with Run Charts or other tools which require the date and time of each observation. Provide a place to enter the name of the individual collecting the data.
Page 4 Draft Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

Allow enough space so data collectors can write in comments on unusual events. 2.2.2 Sampling-Based Measurement In the sampling-based measurement one selects a sample of a certain product or publication and checks the selected sample. For example in order to measure correctness, a set of AIS publications (i.e., not all) is taken and checked for correctness. The correctness is determined by using this selected sample. This technique is particularly useful when you don t have enough tools or manpower in order to collect data. 2.2.3 Simulation This technique is generally used in automated environments in order to measure service availability. The basic idea is to generate synthetic service/product requests at regular intervals and collecting availability and performance measures based on tracking these requests. This technique has some certain advantages: It does not require external participation (e.g., your customers) It does not require real-time agents that can be heavy and expensive in terms of computing power and investment.

2.3

Designing a Data Collection System

The design of a data collection system depends on the following: What has to be collected? What mechanisms are already in place? If there are many unknowns about what data is required, it is probably best to start with a manual data collection system. After the system has been used and the system design has stabilised, it can be automated. The first problem is to get everyone to report reasonably accurate data. This will be a challenge, no matter how much instruction is provided. Most people get into the routine very quickly, but some will require considerable support and instruction. Intensive follow-up and checking of detail is needed to be sure all problems are reported and that they are reported correctly. During first several weeks of implementation, supervisors should review all data sheets and look into any entries that look questionable. When KPIs are first implemented, there will many questions about what they mean, where they come from and how they should be interpreted, even if all this was explained before starting. Managers and the system developers should carefully listen to any questions and objections because they may indicate where the system needs to be improved. If KPIs are not being used and analysed, there are only four possible explanations: There is a lack of leadership of the program The users don t understand information

Edition: 0.1

Draft

Page 5

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

The information is not relevant user s needs. The information is incorrect or unreliable The following should be considered during the design of a Data Collection System: Make it Easy to Report Data. Make it as easy as possible to record or enter data. Don t add steps and people to a production process to capture necessary data. Build reporting into the process by modifying forms and procedures. Don t Overkill. Don t take the approach of collecting every bit of available data and rearranging it into massive reports that no one can use. Instead first determine what information is needed and then develop the system to supply it. Reports and graphs should be designed for specific users and purposes. Since different managers must make different decisions, they need different information. Give every the graphs and summary reports of all KPIs that are relevant to them. Decentralise the Measurement System. Don t try to build a centrally controlled, one-size-fits-all system. It will be cumbersome, slow and inefficient. There are good reasons for decentralising measurement systems: Measures and data systems needed in different processes and subprocesses are diverse that building a system to accommodate all the needs would be practically impossible. Systems customised for different functions are more efficient and more effective than more general solutions The best approach appears to be to take a decentralised approach but keep closely coupled functions under the same umbrella so the data will share a common structure and can be easily interrelated. Level of Detail. The amount of detail needed to identify the root causes of problems is typically more than what is required to establish accountability. In theory any thing (e.g., process) could be measured so extensively that the root cause of any problem could be quickly isolated. However, in general, it is very expensive. Therefore it is required to select right level of details that makes the best trade-off between how much data to collect and how often the detailed data is need. Another approach is to find the root-cause iteratively. In the iterative the probable causes can be identified in suspected areas and additional data can be collected to finally determine the real causes of problems. Combining Measure to Create a Single Composite Index. All managers would like to have one KPI that would indicate when everything was not in fine shape and tell them what to do about it. Unfortunately it is not possible since complex systems cannot be controlled with simple measurement systems. However it is still useful to composite KPIs for a department or a process since they can help keep the relative importance of individual KPIs. The easiest way of constructing a composite KPI is to assign a weighting factor to each component and calculate the weighted average. Security of Confidential Information. For KPIs to be effective as motivator, everyone must be kept abreast of performance. While it is best for KPIs to be available for everyone, it may be necessary to keep some information confidential for competitive reasons.

Page 6

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

3.

MONITORING
KPI reporting is an important communication vehicle while monitoring KPIs. However the biggest danger with KPI reporting is to create information overload by developing and distributing too many reports, charts and tables to too many people. The quality of information (and its value) is, in general, inversely proportional to the volume information. For effective communications, reports has to fulfil the following requirements: Relevant to the person receiving it. This requirement has two aspects: (1) Making sure that managers get all information that is relevant to them (2) and also they get nothing that is not relevant to them. Information not needed or not used is just another form of waste. Well organised. Cause-effect relationships, process relationships and the relative importance of KPIs to organisation or operating unit should be readily apparent. Understood by those using it. Information that isn t understood is just another form of waste (useless noise) Kept as brief as possible. Since everyone s time is limited and valuable, the shorter a report is the more likely it will be used. Wading through pages of numbers to find important points is not effective use of any manager s time. Reports should provide information to lines of business and customer community. The audience for certain information affects the format of reports, different reports may be required to cover different aspects of KPIs and to satisfy the interests and focus of various audiences. The following types of reports are generally used for monitoring: Operational Reports Real-Time Reports Executive Summaries Customer Reports

3.1

Operational Report
The format and content of the operational reports vary with the purpose of analysis to be done (Please see the following chapter for details and examples): Trend Analysis. Such reports should present a single KPI as a function of time. The report should be supported by a simple line graph or Run Chart in order to make trend analysis. Cause-Effect Analysis. In case of cause-effect analysis it is checked whether there is a relationship between two or more KPIs. Therefore such reports should contain values of KPIs in question. They are generally supported with a scatter diagram or stratification table. Capability Analysis. Such reports are used to keep track of the KPI values when you start making a change in order to improve the system or

Edition: 0.1

Draft

Page 7

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

processes. These reports will be very similar to ones used in the trend analysis. However in this case the KPI values are presented as a function of change (i.e., before and after the change(s)) not time. Capacity Analysis. These reports are only applicable to KPIs that are used to measure capacity (e.g., number of publications per month) and associated quality KPIs (e.g., rate of errors in publications). Such reports are used to find out where the capacity saturates and at which capacity you can still produce high quality products.

3.2

Real-Time Reports
Real-time reporting and proactive notification of problems increase customers' confidence and flexibility. They generally cover service provision problems happened due to changes in the environment such as: Scheduled outages Unavailability under heavy security attacks (such as virus, hackers, etc.) Strikes Etc. Such kind of notification should show which end customers are affected as well as applications, locations and lines of business that are affected. It should also show the nature of the problem and its symptoms along with when service is anticipated to return to normal.

3.3

Executive Summaries
You should keep in mind one thing when tailoring reports for executive managers: they have no time! Such reports should provide an overall assessment of achieved performance levels including quantitative and qualitative reports. It should provide quick summaries of performance levels and make effective use of graphs and charts to impart this information. Relations achieved performance level with any business impact is an important aspect of the executive summary. The executive summary should be self-contained, particularly for end-of-period reports aimed at senior management and lines of business. If there are KPIs for which problems have been experienced, they should be highlighted with references to any supporting documentation or detailed reports.

3.4

Customer Reports
These reports should provide customers with summarised reports on service and product delivery. If there are KPIs for which problems have been experienced and which are important to customers (e.g. service availability), they should be highlighted. These reports should also cite any steps it takes to improve customer service.

Page 8

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

4.

ANALYSING AND INTERPRETING KPIS

You may design KPIs very well and in a reliable manner. However if the data and information are not properly analysed and interpreted, the benefits will be limited. Main objectives of KPI Analysis are as follows: Identifying opportunities and problems Determining priorities Taking action to improve Making decisions to re-allocate resources Changing or adjusting strategy Providing feedback to change behaviour Recognising and rewarding accomplishments Although rigid rules for analysing and interpreting KPIs and their related data cannot be defined, some guidelines will help assure that the data is analysed correctly and the right conclusions are drawn.

4.1

Variations (Trend) Analysis

All KPIs will exhibit some variation. At the lower levels of detail, this variation can be quite large even if everything is under control. The first rule to follow when interpreting KPIs is to not react to short-term deviations until reasons for the deviation are understood. If the deviation is within the normal range, there has been no change in performance at all. If it is a very large deviation, something unusual has happened and the cause should be determined. In most cases, special problems or circumstances are known by those responsible for the KPI. A simple line graph or Run Chart will provide a good sense of the normal variation. This is one reason why KPIs should always be put on run charts instead of relying solely on reports. In order explore how the variance can be analysed; lets assume that you measure the time to reach your office each morning:

Measurements for Time to Reach (Minutes) Day Time 1 25.3 2 22.1 3 24.4 4 26.8 5 27.3 6 26.6 7 24.2 8 22.0 9 21.3 10 23.9

Edition: 0.1

Draft

Page 9

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

30 28 26 24 22 20 18 16 14 12 10 1 2 3 4 5
Days Figure 2. Example of Stable Trend

Time to Reach (mins)

10

Although there is a fluctuation between 21.3 and 27.3 minutes, the trend is quite stable around median value 24.3 and data points do not show a particular and steady trend. Therefore there is no reason that you try to find out why it took you 27.3 minutes in the day 5. Lets assume that you continue your measurements next ten days and you obtain the following measures.

Measurements for Time to Reach (Minutes) Day Time 11 18.1 12 17.6 13 17.2 14 15.1 15 14.4 16 14.0 17 12.6 18 12.2 19 14.5 20 15.3

New measurements show a descending trend. The trend is confirmed with reasonable number of consecutive data points. You can conclude that your time to reach office has been reduced.

Page 10

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

30 28 26 24 22 20 18 16 14 12 10
1 3 5 7 9 11 13 15

Time to Reach (mins)

Trend

17

Days Figure 3. Example of Change in Trend

4.2

Identifying Relationships (Cause-Effect Analysis)

Identifying relationships between variables is important for understanding how a things work and also the causes of problems. Looking for relationships should be part of analysing any KPI, especially those that have several external or internal variables that might affect the performance. This also includes customers, because particular customers can influence quality measures such as complaints and general satisfaction. While identifying relationships between two variables X and Y there are three possibilities that should be considered: X and Y are not related at all. The apparent relationship is the result of pure coincidence. X and Y are related, but X does not cause Y or vice-versa. Instead they are affected by another variable(s). X and Y have a cause-effect relationship. There are two popular methods used for identifying relationships: scatter diagrams and stratification.

4.2.1

Scatter Diagrams Scatter Diagrams are a simple way of identifying whether a relationship exists between two variables and, if so, the strength of the influence of one variable upon the other, e.g. the effect of increases in temperature on the consumption of domestic drinking water. Analysis with Scatter Diagrams is done through plotting the data from each data set on a graph. Horizontal axis of the graph is scaled for the cause variable and vertical axis for the effect variable (maximum values applicable to each axis will be determined by reference to values within the data sets). If there is a strong relationship between two variables, it will probably be indicated by the Scatter Diagram. If the relationship is weak, it may not be readily apparent,

Edition: 0.1

Draft

19

Page 11

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

especially if there are other factors influencing the behaviour of the variable of interest.

Figure 4. Correlation Examples with Scatter Diagrams for variables X and Y.

4.2.2

Stratification Stratifying data is another way of identifying relationships between variables. Stratification consists of cutting the data into layers according to the different variables in question. For example, to analyse customer complaints about toasters it might be appropriate to look at them by model number, which plant made them and which retail chain sold them. Lets take the following table as an example:

Model % 106A 117B 419B 777A

Plant %

Chain % 0.6 0.8 0.7 0.8 2.1

1.1 1 2.6 2 0.8 3 0.5 4

0.9 K 0.8 S 2.4 P 0.9 T W

The analysis indicates a possible relationship with model 117B, plant 3 and store chain W but these relationships are not certain and require further investigation since it is possible that: The chain W sold more 117B units than others or

Page 12

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

The plant 3 produced more117B units or There is nothing wrong with 117B units but chain W mistakenly misrepresented the units in its advertisements. Therefore although stratification illustrates possible relationships it is not definitive at all and a further investigation should be carried out.

4.3

Capability Analysis
Understanding process capability is important for both control and planning purposes. For large and complex processes the capability question is What performance level can be maintained by the process? In order to determine this performance level it may be necessary to select a period that seems to represent normal operating conditions and derive a performance target from this period. However this target only applies to the process in its current configuration. Often real question is At what point should attempts to incrementally improve the capability of a process be abandoned in favour of a radical restructuring (or reengineering) of the process. Lets take the following table, which illustrates improvements provided by each successive change in a production process, as an example: Event Ratio of Errors in Publications 10.0 5.0 4.1 3.4 2.8

Start Change 1 Change 2 Change 3 Change 4

It might seem further significant improvements are not feasible the after change 4. However it should be noted that determining when a process has reached its practical limit for incremental improvement is a matter of judgement. If the person making that judgement understands how well the process is performing and its improvement history, that determination will probably be quite accurate.

4.4

Capacity Analysis
How much work a process can do in a given period is its production capacity. It is a type of capability. Knowing this figure is obviously essential for effective planning and management. When production capacity is exceeded, the following will happen: If the process is limited by equipment capacity, work will pile up in front of the limiting steps of the process production delay will increase. If the process is limited by labour capacity, work will pile up in front of the bottleneck step(s) in the process, but the work may also get done while the quality of the work suffers rework and rejects will increase. Identification of capacity for the first case is relatively straightforward. The saturation point can be determined after plotting production data (e.g., number of publications):

Edition: 0.1

Draft

Page 13

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Production Units/Month Range where production saturates

Time Months Figure 5. Capacity Analysis (I)

For the second case an estimate of production capacity can be derived from quality and production data:

Production Units/Month

Error Rate %

Errors start increasing in this region. The capacity cut-off is a value in that region

Time Months Figure 6. Capacity Analysis (II)

4.5

Considering Context
KPIs do not exist in a vacuum. They are affected by anything that affects an organisation or its production processes. Weather, strikes, supply line disruptions, unusual customer requests, competitorsactions and many others can cause large deviations in the KPIs. That is why it is a good practice to note significant changes in environmental factors or unusual circumstances on charts when they occur. Besides explaining what caused particular behaviour, these notes can help managers predict what will happen under similar conditions in the future. However although context is important making allowances for poor performance can also be overdone. Minor events should never be used to make excuses for large changes in the performance. The same is true when performance is unusually good. This could also be caused by favourable circumstances. Even if the circumstances are not controllable, understanding what happened can lead to new opportunities.

Page 14

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

4.6

Establishing Priorities
Because there will always be more problems and opportunities than there are resources available to pursue them, managers must always think in terms of priorities. Priorities for improving performance or changes in these priorities should be one of the regular outputs of analysing KPIs. Assuming a measurement system has the capability of determining the relative impact of KPIs, priorities should be relatively clear in terms of costs or profit opportunities. However priority decisions must be supported with the following: Potential risk Investment required Payback period How well projects support strategic objectives Availability of resources Etc. Priorities must be evaluated from the broader perspective of the total organisation to avoid sub-optimisation and to assure resources are allocated to the areas of most return.

Edition: 0.1

Draft

Page 15

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

5.

CONCLUSIONS AND FUTURE WORK

This guide discusses issues and problems that can be encountered during measurement, monitoring and analysis of KPIs. This guide covers the following topics: How to prepare data collection plans Tips and tricks to design a data collection system Measurement techniques Monitoring and reporting Different types of analysis that can be done via KPIs: trend analysis, causeeffect analysis, capability analysis, capacity planning, etc. The provided information will assist AIS organisations in order to fulfil the associated ISO9000:2001 requirement (i.e., section 8). It will be also useful while implementing service level management. It should be noted that the main objective is not only to measure but also to take corrective and preventive actions by analysing performance levels achieved for KPIs. The document is still in its early stages and it requires some detailed examples especially for KPI monitoring and analysis.

Page 16

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

QUALITY TOOLS
This appendix is taken from The Quality Tools Cookbook by Prof. Sid Sytsma Ferris State University and Dr. Katherine Manley - Ferris State University . They used the following references: Brassard, M. and Ritter, D. (1994). The Memory Jogger, Goal/QPC, Methuen, MA Brassard, M. (1989). The Memory Jogger+, Goal/QPC, Methuen, MA Brassard, M., Tucker, S. and Oddo, F. (1993). Total Quality Management in Education, Goal/QPC, Methuen, MA Byrnes, M., and Cornesky, R. (1994). Quality Fusion: Turning Total Quality Management into Classroom Practice, Cornesky and Associates, Port Orange, FL Chang, R. and Niedzwiecki, M. (1993). Continuous Improvement Tools Vol. 1, Richard Chang Associates, Irvine, CA Chang, R. and Niedzwiecki, M. (1993). Continuous Improvement Tools Vol. 2, Richard Chang Associates, Irvine, CA Cornesky, R. (1993). The Quality Professor, Magna Publications Inc., Madison, WI Cornesky, R., and McCool S. (1992). Total Quality Improvement Guide for Institutions of Higher Education, Madison, WI E. B. Dean's Internet page at e.b.dean@larc.nasa.gov Ishikawa, K (1982). Guide to Quality Control, Quality Resources, White Plains, NY. McCloskey, L.A, and Collett, D.N. (1993). TQM a Basic Text, Goal/QPC, Methuen, MA The tools given in this appendix are supposed to assist AIS organisation while analysing their KPIs.

A.1

Cause and Effect Diagram

A Cause and Effect Diagram (Fishbone Diagram) is an analysis tool to display possible causes of a specific problem or condition.

A.1.1 Usage Identifying potential causes of a problem or issue in an orderly way (e.g., Why is the production process suddenly producing so many defects?)

Summarising major causes under four categories (e.g., People, Machines,

Methods, and Materials or Policies, Procedures, People, and Plant) A.1.2 Steps Prepare a flip chart or an overhead transparency of the following template:
Edition: 0.1 Draft Page 17

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Error! Unknown switch argument. Write the issue (problem or process condition) on the right side of the Cause and Effect Diagram. Identify the major cause categories and write them in the four boxes on the Cause and Effect Diagram. You may summarise causes under categories such as: o o o Methods, Machines, Materials, People Places, Procedures, People, Policies, Surroundings, Suppliers, System, Skills

Brainstorm potential causes of the problem. As possible causes are provided, decide as a group where to place them on the Cause and Effect Diagram. It is acceptable to list a possible cause under more than one major cause category. Review each major cause category. Circle the most likely causes on the diagram. Review the causes that are circled and ask "Why is this a cause?" Asking "why" will help get to the root cause of the problem. Reach an agreement on the most probable cause(s). A.1.3 Example Error! Unknown switch argument.

A.2

Check Sheet
The Check Sheet is a data-gathering and interpretation tool.

A.2.1 Usage Distinguishing between fact and opinion (example: how does the community perceive the effectiveness of the school in preparing students for the world of work?) Gathering data about how often a problem is occurring (example: how often does printing machine out of order?) Gathering data about the type of problem occurring (example: What is the most common type of word processing error created by the employeesgrammar, punctuation, transposing letters, etc.?)

Page 18

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

A.2.2 Steps Clarify the measurement objectives. Ask questions such as "What is the problem?", "Why should data be collected?", "Who will use the information being collected?", "Who will collect the data?" Create a form for collecting data. Determine the specific things that will be measured and write this down the left side of the check sheet. Determine the time or place being measured and white this across the top of the columns. Collect the data for the items being measured. Record each occurrence directly on the Check Sheet as it happens. Tally the data by totalling the number of occurrences for each category being measured. A.2.3 Example

Edition: 0.1

Draft

Page 19

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Week 14- 2001 Monday Tuesday Wednesday Thursday Friday Total 1. Input Hopper Faults 2. A4 Paper Tray Fault 10 3. A3 Paper Tray Fault 3 4. Internal Paper Jams 25 5. Duplex Processing Faults 5 6. Stapling Faults 7. Output Collation Faults 0 2 5

1 7 6 4 5 2
Figure 7. A Sample Checksheet for a Photocopy Machine

A.3

Pareto Chart
A Pareto Chart is a special form of a bar graph and is used to display the relative importance of problems or conditions.

A.3.1 Usage Focusing on critical issues by ranking them in terms of importance and frequency (e.g., Which problem with Product X is most significant to our customers?) Prioritising problems or causes to efficiently initiate problem solving (e.g., Solution of what production problem will improve quality most?)
Page 20 Draft Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

Analysing problems or causes by different groupings of data (e.g., by department, by machine, by team) Analysing the before and after impact of changes made in a process (e.g., Has the initiation of a quality improvement program reduced the number of defectives?) A.3.2 Step-by-Step Example Determine the categories of problems or causes to be compared. Begin by organising the problems or causes into a narrowed down list of categories (usually 8 or less). Select a Standard Unit of Measurement and the Time Period to be studied. It could be a measure of how often something occurs (defects, errors, tardies, cost overruns, etc.); frequencies of reasons cited in surveys as the cause of a certain problem; or a specific measurement of volume or size. The time period to be studied should be a reasonable length of time to collect the data. Collect and Summarise the Data. Create a three-column table with the headings of "error or problem category", "frequency", and "percent of total". In the "error or problem category" column list the categories of problems or causes previously identified. In the "frequency" column write in the totals for each of the categories over the designated period of time. In the "percent of total" column, divide each number in the "frequency" column by the total number of measurements. This will provide the percentage of the total. Error Category Punctuation Grammar Spelling Typing TOTAL Frequency 22 15 10 3 50 Percentage 44% 30% 20% 6% 100%

Create the framework for the horizontal and vertical axes of the Pareto Chart. The horizontal axis will be the categories of problems or causes in descending order with the most frequently occurring category on the far left (or at the beginning of the horizontal line). There will be two vertical axes-one on the far left and one on the far right. The vertical axis on the far left point will indicate the frequency for each of the categories. Scale it so the value at the top of the axis is slightly higher than the highest frequency number. The vertical axis on the far right will represent the percentage scale and should be scaled so that the point for the number of occurrences on the left matches with the corresponding percentage on the right. Error! Unknown switch argument. Plot the bars on the Pareto Chart. Using a bar graph format, draw the corresponding bars in decreasing height from left to right using the frequency scale on the left vertical axis. To plot the cumulative percentage line, place a dot above each bar at a height corresponding to the scale on the right vertical axis. Then connect these dots from left to right, ending with the 100% point at the top of the right vertical axis.

Edition: 0.1

Draft

Page 21

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Error! Unknown switch argument. Interpret the Pareto Chart. Use common sense-just because a certain problem occurs most often doesn't necessarily mean it demands your greatest attention. Investigate all angles to help solve the problems-What makes the biggest difference? What will it cost to correct the problems? What will it cost if we don't correct this problem?

A.4

Scatter Diagram
A Scatter Diagram is used to interpret data by graphically displaying the relationship between two variables.

A.4.1 Usage Validating "hunches" about a cause-and-effect relationship between types of variables (example: Is there a relationship between the production speed of an operator and the number of defective parts made?) Displaying the direction of the relationship (positive, negative, etc.) (example: Will increasing assembly line speed increase or decrease the number of defective parts made?) Displaying the strength of the relationship (example: How strong is the relationship between assembly line speed and the number of defective parts produced) A.4.2 Steps

Collect two pieces of data (a pair of numbers) on a process, or product.

Create a summary table of the data. Draw a diagram labelling the horizontal and vertical axes. It is common that the "cause" variable be labelled the horizontal (X) axis and the "effect" variable be labelled the vertical (Y) axis. The values should increase up the vertical scale and toward the right on the horizontal scale. The scale on both the X and Y-axes should be sufficient to include both the largest and the smallest X and Y values in the table. Plot the data pairs on the diagram by placing a dot at the intersections of the X and Y coordinates for each data pair. Error! Unknown switch argument. Interpret the scatter diagram for direction and strength. o Interpreting the direction: Data patterns may be positive, negative, or display no relationship. A positive relationship is indicated by an ellipse of points that slopes upward demonstrating that an increase in the cause variable also increases the effect variable. A negative relationship is indicated by an ellipse of points that slopes downward demonstrating that an increase in the cause variable results in a decrease in the effect variable. A diagram with a cluster of points such that it is difficult or impossible to determine whether the trend is
Draft Edition: 0.1

Page 22

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

upward sloping or downward sloping indicates that there is no relationship between the two variables. o Interpreting the strength: Data patterns, whether in a positive or negative direction, should also be interpreted for strength by examining the "tightness" of the clustered points. The more the points are clustered to look like a straight line the stronger the relationship.

A.4.3 Example Error! Unknown switch argument.

A.5

Run Chart
Run charts are used to analyse processes according to time or order. Run charts are useful in discovering patterns that occur over time.

A.5.1 Usage Simplest display of trends over time Data plotted in time order An aid to understanding basic characteristics of a process. A.5.2 Steps Gathering Data. To begin any run chart, some type of process or operation must be available to take measurements for analysis. Measurements must be taken over a period of time. The data must be collected in a chronological or sequential form. You may start at any point and end at any point. For best results, at least 25 or more samples must be taken in order to get an accurate run chart. Organising Data. Once the data has been placed in chronological or sequential form, it must be divided into two sets of values x and y. The values for x represent time and the values for y represent the measurements taken from the manufacturing process or operation. Charting Data. Plot the y values versus the x values by hand or by computer, using an appropriate scale that will make the points on the graph visible. Next, draw vertical lines for the x values to separate time intervals such as weeks. Draw horizontal lines to show where trends in the process or operation occur or will occur.

After drawing the horizontal and vertical lines to segment data, interpret the
data and draw any conclusions that will be beneficial to the process or operation. Some possible outcomes are: o o o Trends in the chart Cyclical patterns in the data Observations from each time interval are consistent

Edition: 0.1

Draft

Page 23

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

A.6

Affinity Diagram
An Affinity Diagram is a creative process, used with or by a group, to gather and organise ideas, opinions, issues, etc.

A.6.1 Usage Adding structure to a large or complicated issue (example: what are the central issues in the development of a particular new product?) Breaking down a complicated issue into broad categories (example: What are the major steps in the completion of a complex project?) Gaining agreement on an issue or situation (example: How should a new product be marketed?) A.6.2 Steps State the issue or problem to be explored. Start with a clear statement of the problem or goal and provide a time limit for the session-usually 45-60 minutes is sufficient. Brainstorm ideas for the issue or problem. Each participant should think of ideas and write them individually on index cards, sticky notes, or have a recorder write them on a flip chart. Collect the cards or sticky notes, mix them up and spread them out (or stick them) on a flat surface (e.g., desk or wall). Index cards can easily be secured to a wall with a putty-type adhesive. Arrange the cards or sticky notes into related groups. For approximately 15 minutes allow participants to pick out cards that list related ideas and set them aside until all cards are grouped. Create a title or heading for each grouping that best describes the theme of each group of cards.

A.7

Nominal Group Technique

A nominal group technique is a structured process that identifies and ranks the major problems or issues that need addressing.

A.7.1 Usage Identifying the major strengths of a department/unit/institution (example: making decisions by consensus when selecting problem solutions in a business) Providing each participant with an equal voice (example: defusing a domineering influential employee who tends to control the discussion and dominate the process) A.7.2 Steps Request that all participants (usually 5-10 persons) write or say the problem or issue they feel is most important.
Page 24 Draft Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

Develop a master list of the problems or issues. Generate and distribute to each participate a form that numbers in no particular order the problems or issues. Request that each participant rank the top five problems or issues by assigning a #5 points to their most important perceived problem and #1 points the least important of their top five. Tally the results by adding the points for each problem or issue. The problem or issue with the highest number is the most important one for the total team. Discuss the results and generate a final ranked list for action planning. A.7.3 Example Five possible solutions to a problem exist. Six people must decide which solution should be attempted first. The solutions are called A, B, C, D, and E. The people are Bill, Bob, Henry, Peter, Paul, and Mary. Each of the six people order the potential solutions. The following matrix is developed.
Solution A B C D E Bill 1 5 4 2 3 Bob Henry Peter Paul Mary 2 1 4 3 5 5 2 5 5 4 3 3 3 1 2 1 4 1 2 1 4 5 2 4 3 TOTAL 16 26 16 11 21

We would begin by trying solution B, followed by E.

A.8

Interrelations Digraph
An Interrelations Digraph is a graphical representation of all the factors in a complicated problem, system, or situation. It is typically used in conjunction with one of the other quality tools, particularly the affinity diagram. Frequently the header cards from the affinity diagram are used as the starting point for the interrelations digraph.

A.8.1 Usage Identifying key or driver issues from a list of important issues. Identifying the most important problems for solution when the number of problems exceeds the resources available to solve all of them. Identifying the root cause of existing problems. Identifying key factors needed to make a decision when there is insufficient information available to make a data-driven decision. A.8.2 Steps State clearly the issue or problem. Write it on a card and stick it to the centre of a board. Determine the factors related to the issue. Frequently these will be the headers from a previously completed affinity diagram. Place cards containing these factors in a circle around the issue card.
Edition: 0.1 Draft Page 25

AIM/AEP/S-LEV/0008

KPI Measurement, Monitoring and Analysis Guide

Determine if cause-effect relationships exist between any of the cards. If so, draw an arrow from the "cause" card to the "effect" card. Do this for all cause/effect relationships that you can find. Analyse the relationships. Cards that have most arrows going from them tend to be root causes. Cards that have most arrows going to them are root effects.

A.9

Force Field Analysis

The Force Field Analysis is an analysis tool that uses a creative process for forcing agreement about all facets of a desired change.

A.9.1 Usage Clarifying and strengthening the "driving forces" for change (example: What things are "driving" us toward process improvement?) Identifying obstacles or "restraining forces" to change (example: What is "restraining" us from achieving increased sales?) Encouraging agreement on relative priority of factors on each side of the balance sheet A.9.2 Steps 1. Create a flip chart or overhead transparency with a form similar to the following: 2. Discuss and come to agreement with the group (usually 5-7 people) on the current situation and the goal. Write this in the appropriate space on the form. 3. Brainstorm the "driving" and "restraining" forces. Driving forces are things (actions, skills, equipment, procedures, culture, people, etc.) that help move toward the goal. Restraining forces are things that can inhibit reaching the goal. 4. Sort on common themes and/or prioritise the driving and restraining forces. 5. Discuss action strategies to eliminate the restraining forces and to capitalize on the driving forces.

A.10

Matrix Diagram
A Matrix Diagram is a tool that is used to systematically organise information that must be compared on a variety of characteristics in order to make a comparison, selection or choice.

A.10.1 Usage Assigning tasks to complete a project. Make comparisons between competing alternatives that involve multiple characteristics. Prioritise combinations of new and old activities that maximised the number of total objectives met.

Page 26

Draft

Edition: 0.1

KPI Measurement, Monitoring and Analysis Guide

AIM/AEP/S-LEV/0008

A.10.2 Steps Determine the factors that are important for making a correct selection or assignment. Select the type of matrix to be used. L-shaped matrices are used for twofactor comparisons; T-shaped are used for 3-factor comparisons in that display indirect and direct relationships; Y-shaped are used for 3-factor comparisons showing direct relationship only. Select the relationship symbols to be used. Complete the proper matrix using the appropriate factors and symbols. Examine the matrix and draw the appropriate conclusion.

End of Document

Edition: 0.1

Draft

Page 27