Академический Документы
Профессиональный Документы
Культура Документы
discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/258999189
CITATIONS READS
0 600
2 authors:
Some of the authors of this publication are also working on these related projects:
Book. Interactive Modeling and Simulation in Business System Design. View project
All content following this page was uploaded by Ella Roubtsova on 25 July 2014.
Keywords: Key Performance Indicators (KPIs), goal modeling, concept modeling, executable process modeling, proper-
ties of KPIs, validation
Abstract: Competition for funding between organizations attracts attention to their Key Performance Indicators (KPIs).
KPIs are usually designed with a top-down approach as families of measures for a group of business units
and often do not take into account the difference in goals and business processes of organizations at the
strategic, tactical and operational level. This results in unreliable, inefficient and sometimes inconsistent
KPIs. Validation of KPI properties is typically postponed until the KPI is implemented, and databases are
populated with values. The reason is the absence of intuitive and simple methods for KPI modelling that
relate the strategic and tactical models and executable operational models. We propose such a method for KPI
modelling and validation of their properties. Our method combines ideas of goal, conceptual and executable
process modelling. Models at all levels are derived from KPI definitions. The conceptual modelling techniques
are used to relate the strategic and tactical models. The synchronous semantics of protocol modelling is used to
relate the tactical and the operational models. The executable operational and tactical models enable derivation
the KPI values, testing KPIs against the desired properties and identification of ambiguities in KPI definitions
that need to be resolved to improve KPIs.
AND AND
Goal Goal
Goal
A Referred Person has <<Monitor access> Measure access to
Estimate the size of the
access to psychological psychological
Population r of people
therapies therapies
needed psychological
<<Monitor effectiveness of treatment therapy
Goal
Goal
A Referred Person after
Measure effectiveness of
treatment has improved
treatment
conditions
Concept
Access Dashboard
Attributes
KPI1:Number of people with disorder
<<Derive>>
identified by the annual survey <<Derive>>
KPI4:Number of people with disorder who in
the entered treatment
HI1: Access Rate
not justified by the goals of the system (Gossler and focused on the business semantics.
Sifakis, 2003). Analysis of intermediate states may - Secondly, we need to model KPIs that count
be relevant for validation of asynchronous implemen- entities in a specified state. The semantics of derived
tation, but the KPIs are situated at the tactical and states is needed for KPI modelling as KPIs must
strategic level, at the level of visible states of the sys- reflect the state of business processes from which
tem. the measurements are obtained to drive the KPIs.
Activity diagrams and UML state machines do not
We have chosen the protocol machines for the have semantics allowing one concept to derive its
following reasons: states from the state of other concepts (OMG, 2003).
- Firstly, we want to minimize complexity of semantic - Thirdly, we should be able to model crosscutting
modelling of business concepts. Protocol modelling sub-processes as KPIs may collect information from
uses a data-extended synchronous CSP-parallel com- different elements of the system. Crosscutting sub-
position (McNeile and Simons, 2006).CSP stands processes are repeated parts of a business process that
for Communicating Sequential Processes. The initial cannot be decomposed using hierarchy or sequential
form of this composition operator was proposed decomposition. Activity diagrams and UML state
in (Hoare, 1985). The initial operator was extended machines cannot easily specify the crosscutting
for machines with data by (McNeile and Simons, sub-processes as the composition techniques of
2006). Because of the synchronous composition of the diagrams are restricted with sequential and
its parts a protocol model presents only quiescent hierarchical composition (Gossler and Sifakis, 2003).
states of the system. This allows us to model only Separating crosscutting sub-processes can be used
visible state and essentially decrease the state space for efficient modelling of KPIs. Combining different
of the model. All states can be justified by the system measures at different abstraction level presented with
goals and subgoals. Modelling and reasoning can be
crosscutting parts simplifies the KPI models. Proto- DateOfSurvey the value of the attribute of the proto-
col modelling supports the CSP parallel composition col machine Survey.
of crosscutting sub-processes in the business process By accepting or refusing this event the protocol
that simplifies the KPI modelling. model communicates with the environment. This
communication is simulated in Modelscope tool sup-
A protocol machine is an object life-cycle model. porting Protocol Modelling (McNeile and Simons,
It is presented as a state transition structure with a data 2005).
storage that defines the ability of a system to interact A protocol machine may also describe a part of the
with its environment by accepting or refusing events life cycle of an object. The metacode is started with
from the environment. A protocol machine can be the key word BEHAVIOURFor example, the concept
seen as an object that exists even without its creation Refereed Person is presented as the protocol machine
in its initial state. An object goes into its active state Referred Person. This protocol machine includes
accepting a creating event. All protocol machines are behaviour Treatment. Both protocol machines
composed with the data-extended synchronous CSP- Referred Person and Treatment are synchronized
parallel composition. with an event EnterTreatmentAndAssess. The sep-
aration of the concept Treatment simplifies the quan-
tification on the states of the Referred Person used
The protocol model of the program for IAPT.
for KPI definition.
Protocol machines used for modelling of KPIs are
graphically shown in Figure 2. Each concept from OBJECT ReferredPerson
NAME PersonName
Figure 1 has the corresponding protocol machine at
INCLUDES Treatment
Figure 2 with the same name. The boxes in Figure 2 ATTRIBUTES PersonName: String,
represent protocol machines, where the attributes of DateOfReferring:Date,
concepts are shown in bubbles. The graphical pre- STATES referred, 28DaysWaited, left,
sentation does not provide all the elements of the enteredTreatmentAndInitiallyAssessed,
model. The complete protocol model of the pro- completedTreatmentAndAssessed
TRANSITIONS @new*Refer =referred,
gram for IAPT is presented with its metacode and
referred*Decline=left,
small java functions and can be found in (Roubtsova, left*Return=referred,
2013). For instance, the metacode of an instance of referred*Wait=28DaysWaited,
the Survey concept is a protocol machine Survey. 28DaysWaited*Leave=left,
28DaysWaited*EnterTreatmentAndAssess=
OBJECT Survey
28DaysWaited
NAME SurveyName
ATTRIBUTES SurveyName: String,
BEHAVIOUR Treatment
Population:Integer,
ATTRIBUTES CasenessBefore:Boolean,
DateOfSurvey:Date
CasenessAfter:Boolean
STATES created
STATES enteredTreatmentAndInitiallyAssessed,
TRANSITIONS @new*CreateSurvey=created
completedTreatmentAndAssessed,left
TRANSITIONS
EVENT CreateSurvey
@new*EnterTreatmentAndAssess
ATTRIBUTES Suvey:Survey,
=enteredTreatmentAndInitiallyAssessed,
SurveyName:String,
enteredTreatmentAndInitiallyAssessed*Leave=left,
Population:Integer,
enteredTreatmentAndInitiallyAssessed*
DateOfSurvey:Date
CompleteTreatmentAndAssess
It is assumed that the survey is periodic (the pe- =completedTreatmentAndAssessed
riod is not given), and we add the tacit attribute to the
EVENT Refer
Survey: DateOfSurvey:Date. Only the Survey in-
ATTRIBUTES ReferredPerson:ReferredPerson,
stance closest to the date of KPI monitoring is used PersonName:String,
for KPIs calculation. DateOfReferring:Date
Each instance of the Survey is created by ac- EVENT Decline
cepting an event Create Survey. Only the Survey ATTRIBUTES ReferredPerson:ReferredPerson
in state ”created” can provide the values of its EVENT Return
ATTRIBUTES ReferredPerson:ReferredPerson
attributes of the LevelOfNeed and Population
EVENT Wait
for performance indicators. The acceptance of ATTRIBUTES ReferredPerson:ReferredPerson
an event Create Survey brings with its attribute EVENT Leave
Population the number of people who have depres- ATTRIBUTES ReferredPerson:ReferredPerson
sion and(or) anxiety disorders and with its attribute EVENT EnterTreatmentAndAssess
ATTRIBUTES ReferredPerson:ReferredPerson, public class AccessDashboard extends Behaviour{
CasenessBefore:Boolean //KPI 1 Level of Need
EVENT CompleteTreatmentAndAssess
ATTRIBUTES ReferredPerson:ReferredPerson, public int getLevelOfNeed() {
CasenessAfter:Boolean int LevelOfNeed=0;
Attributes of the protocol machines Referred // choose the date three years ago
Person and Treatment are shown in bubbles in Calendar cal = Calendar.getInstance();
Figure 2. Attributes CasenessBefore:Boolean cal.add(Calendar.YEAR, -100);
and CasenesAfter:Boolean model the procedure Date dd = cal.getTime();
of assessment of the patient’s conditions. Each
Date Dashd =
transition is labelled with an external event. As
this. getDate("StartOfReportingQuarter");
events are the structures that carry data, they al-
low the model to updatethe attributes in the life cy- Instance[] existingSurvey =
cle of an instance of the Referred Person. The selectInState("Survey", "@any");
value of the DateOfReferring is entered with event
Refer. CasenessBefore is inserted with event for
(int i = 0; i < existingSurvey.length; i++) {
EnterTreatmentAndAssess. CasenessAfter is en-
Date SD=
tered with event CompleteTreatmentAndAssess. existingSurvey[i].getDate("DateOfSurvey");
if
5. Deriving the KPIs from the protocol models of (SD.compareTo(dd)>0 && SD.compareTo(Dashd)<0)
{LevelOfNeed=
concepts. An instance of a dashboard protocol ma-
existingSurvey[i].getInteger("Population");
chine models a KPI report request. The metacode of dd=SD;
the Access Dashboard is shown below. }}
return LevelOfNeed;
OBJECT AccessDashboard
}}
NAME DashboardName
ATTRIBUTES, AccessDashboard reads the state of protocol ma-
DashboardName:String, chine Survey (but does not change it). It finds
StartOfReportingQuarter:Date, the Survey with the closest date and takes the value
!LevelOfNeed:Integer,
!NumberReferredPersons:Integer,
of the Population of this Survey and assigns it
!NumberReferredPersonsWaited:Integer, to own attribute LevelofNeed. The search func-
!NumberOfEnteredTreatment:Integer, tion selectInState("Survey", "@any") selects
!AccessRate: Integer, the set of surveys to choose the latest survey from this
STATES created set.
TRANSITIONS Protocol modelling has predefined select func-
@new*CreateAccessDashboards=created
tions useful for definition of tactical KPIs:
The protocol machines AccessDashbord and - selectInState (”BehaviourName”,”State”) returns
RecoveryDashboard present KPIs monitoring the an array of instances, all of which include the speci-
access and recovery by selecting and counting the in- fied behaviour;
stances of the protocol machines. Derived attributes - selectByRef(”BehaviourName”,”AttributeName”)
of dashboard protocol machines marked by the ex- returns an array of instances, all of which include the
clamation symbol ”!” represent individual KPIs. The specified behaviour (or object) and have the specified
event CreateAccessDashboard is used to insert the attribute.
value of the starting date of the reporting quarter into
the attribute StartOfReportingQuarter:Date. 6. Validating the KPI properties by using the exe-
Each dashboard protocol machine reads the state cutable protocol model, goal and conceptual mod-
of protocol machines Survey and Referred Person els. At the moment, when the KPIs are protocol
and derives the values of own attributes presenting modeled as valid numeric algorithms they can be an-
KPIs. alyzed and tested. The algorithms for KPIs are pre-
Each derived attribute presenting a KPI has a cor- sented in java files and implement exceptions, e.g. for
responding derivation function. The functions are the division by zero for Access Rate and Recovery
stored in the java files extending behaviour of dash- Rate.
board protocol machines. For example, the attribute Quatifiability. Quantification means deriving a
KPI1: LevelOfNeed is calculated using its request number or a conclusion from a set of instances of se-
function presented below. lected concepts in the models. As we apply the select
Access Dashboard
Survey Population =Population
Create Dashboard DateOfSyrvey=DateOFSurvey
(start of reporting
quarter) created
Create Survey
(DateOfSurvey,
Population)
created
KPI1 - Latest Survey. Population
KPI3a - Number of Referred Persons
DateOfReferring within the reported
quarter
KPI3b - Number Referred Persons in any
state except “left” DateOfReferring is
Referred Person
within the reporting quarter.
KPI 4 - Number Of Entered Treatment
DateOfReferring
(in state entered treatment or completed Enter
treatment ) within the reporting quarter Treatment
HI1Access Rate=KP14/KPI1 Refer and Assess
(DateOf Reffering)
Wait
28 days
referred
waited
Recovery Dashboard
Return
Decline Leave
Create Dashboard
(begin of reporting
quarter) created left
functions for KPI derivation, this means that quantifi- Linearity. In order to test linearity of a KPI to
cation is already built into the KPI derivation proce- a number of instances, we need a model with N in-
dure. stances and a model with N + 1 instances. The tests
Sensibility to the change. In order to test the sen- are collected during the execution. Because the pro-
sitivity to change for any KPI, the protocol model is tocol model has only the quiescent states, the KPI-
populated with instances. attributes of the dashboard, are derived from other
protocol machines at the same moment. For exam-
Let us assume that we want to test how ple, all KPIs of AccessDashboard are derived at the
val the KPI1:Level of Need is changed when StartOfReportingQuarter.
the new instance of Survey appears. We cre-
ate two instances of the protocol machine Reliability of the business process used for KPI
Survey with different DateOfSurvey and definitions. Execution may question semantic relia-
one instance of the AccessDashboard with bility of KPIs because of incompleteness of the busi-
StartOfReportingQuarter. We need to test ness process used in the KPI definition.
that the closest instance of the Survey will be For example, the procedure of testing Caseness
chosen to update the KPI1. We do not need to create is not specified in document of the IAPT (Improving
a database to test the KPI definition in protocol Access to Psychological Therapies, 2013). The strate-
modeling. We use only two instances of the pro- gic HI:Recovery Rate depends on the quality of the
tocol machine Survey needed for validation of the procedure of testing Caseness both before and after
sensitivity to the change of the DateOfSurvey. treatment. The Recovery Rate is increased if more of
healthy people with assessed as ”false” will enter the Person after treatment has improved conditions”.
treatment. The Recovery Rate is increased if more of The improvement corresponds to the growth of the
sick people are wrongly assessed as ”false” will leave Recovery Rate, but the growth may be manipulated
the treatment. In other words, the quality of testing is by the procedure of the assessment Caseness both
the point of attention for management of the business before and after treatment. If this procedure is in-
process using this KPI family. In order to improve dependent of the process of treatment and well de-
reliability of the HI: Recovery Rate, the procedure fined/specified to avoid manipulations, the Recovery
of the Caseness testing should be specified. Rate is improvement oriented. If the procedure of
Efficiency of KPI sets. It is more difficult to val- Caseness assessment belongs to the treatment pro-
idate efficiency as there are usually several different cess, and this treatment process gets funding on the
ways to collect data from the model for a KPI calcu- basis of this KPI, then the value of Recovery Rate
lation. The KPI is efficient in organization if it is sim- can be manipulated to meet the planned values by as-
ple and well understood. In this way, the efficiency is sessing healthy people as sick before the treatment
related with semantic reliability. and sending them for the treatment or by assessing
In general, the KPIs should not duplicate each sick people as healthy after the treatment.
other. As we analyse the working programme, the
duplications were already avoided. In the IAPT pro-
gramme, the KPI2 and KPI6a duplicate other KPIs. 4 Related Work and Conclusions
They were found superfluous already by organiza-
tions trying to apply the set of the IAPT KPIs. The
An overview of the KPI specific modelling tech-
validation could be done on the model.
niques is presented in (Strecker et al., 2012). The
Improvement orientation of KPIs. The most im- KPI specific modelling techniques, mentioned by
portant property of KPIs is improvement orientation. (Strecker et al., 2012), (Frank et al., 2009) and
There is a danger of replacing the improvement orien- (Popova and Sharpanskykh, 2010), are based on con-
tation of KPIs with the plan orientation. In this case, ceptual modelling and propose metamodels for KPI
the KPIs may be used for manipulating numbers. The design. They are aimed to integrate the enterprise
value of an improvement oriented KPI cannot be ma- models with a model of performance measurement
nipulated in the attempt to meet its planned value. systems and use the integrated model as a basis for
Our case study presents examples of both an im- further analysis at different organizational levels of
provement oriented KPI and a possibly plan oriented abstraction. The methods do not have means for exe-
KPI. cution of process models and validating the properties
The HI AccessRate = of KPIs.
(NumberO f EnteredTreatment/LevelO f Need) is an ex- The aim of our method is different as we take a de-
ample of the improvement oriented KPIs. It cor- signed set of KPIs as an input from a document or a
responds to the goal: ”A Referred Person has ac- standard and then model and analyse the abstract busi-
cess to psychological therapies.” The improvement ness process derived from this KPI set and the prop-
means positive growth of the ratio of treated people erties of KPIs. We relate KPIs only to the abstract
to the people needing treatment. Modelling shows business processes derived from the KPI definitions
that the numerator and denominator of the KPI are and therefore, simplify the analysis.
objective values that grow through the model execu- We don’t restrict our modelling techniques with
tion and cannot be manipulated in the defined pro- conceptual modelling. Our method combines ele-
cess. The LevelOfNeed comes from an independent ments of goal modelling, conceptual modelling and
process Survey. The NumberOfEnteredTreatment protocol modelling. The need of a practical combina-
is a summation of individually Referred Persons, tion of these methods to validate KPI properties led
which are independent of the treatment providing us to the choice of the synchronous protocol mod-
The HI RecoveryRate = elling technique. As a protocol model is a combi-
Numbero f PeopleMovingToRecovery/ nation of process model and data model, it contains
(Numbero f PeopleCompletedTreatment − useful procedures for communication of protocol ma-
NumberO fCasenessPeopleBe f oreTreatment) chines with data and advanced procedures for deriva-
may become plan oriented and open to manipulations. tion of states of one protocol machine from the states
For validation of the improvement orientation of this of others. These advanced protocol modelling opera-
indicator, we use both the goals associated with KPIs tors are supported with the Modelscope tool (McNeile
and the model of the underlying process. and Simons, 2005). The model is executed and tested.
The KPI corresponds to the goal ”A Referred The combination of techniques proposed in this
paper supports the requirements for the method for Hoare, C. (1985). Communicating Sequential Processes.
KPI modelling as it presents a) an abstraction of the Prentice-Hall International.
available conceptual models enabling modelling of IAB Europe (2013). http://www.iabeurope.eu/news/launch-
KPIs and understanding their relationships; b) intu- of-a-committee-on-standards-and-kpis-for-brand-
itive models of KPIs of different levels. Our method and-audience-campaigns.aspx.
enables model execution and therefore, validation of Improving Access to Psychological Therapies (2013).
IAPT Key Performance Indicator (KPI). Tech-
the desired properties of KPIs. The goal model and nical Guidance for Adult IAPT Services.
the executable protocol model support the validation http://www.iapt.nhs.uk/silo/files/iapt-kpi-technical-
of the properties of KPIs including semantic reliabil- guidance-201213-v20-.pdf.
ity and improvement orientation. The validated pro- KPIStandard (2013). http://kpistandard.com/html/EN/Sectors.
cesses and KPIs can be used for implementation mod- McNeile, A. and Roubtsova, E. (2010). Aspect-Oriented
elling. Development Using Protocol Modeling. LNCS 6210,
In the future work, we plan to further develop pages 115–150.
techniques to test semantic reliability and efficiency McNeile, A. and Simons, N. (2005). http://www.-
of a KPI and adapt our method for design and anal- metamaxim.com/.
ysis of tactical, strategic and complex KPIs (Robert McNeile, A. and Simons, N. (2006). Protocol Modelling.
Kaplan, and David Norton, 2001) used in industry. A Modelling Approach that Supports Reusable Be-
We also plan to integrate our method with the havioural Abstractions. Software and System Model-
ing, 5(1):91–107.
methods for system implementation modelling. The
models of implementation are usually asynchronous. Neely, A., Mills, J., Platts, K., Richards, H., Gregory, M.,
Bourne, M., and Kennerley, M. (2000). Performance
All asynchronous deviations of their behaviour from measurement system design: developing and testing a
the visible system behaviour presented with protocol process-based approach. International Journal of Op-
models should be required or accepted by users. In erations and Production Management, 20(10):1119 –
such a way using protocol models may contribute to 11452.
reliability of requirements and models of implemen- OMG (2003). Unified Modeling Language: Superstructure
tations. version 2.1.1 formal/2007-02-03.
Parmenter, D. (2010). Key Performance Indicators, Devel-
oping, Implementing and Using Winning KPIs. John
Wiley & Sons, New Jersey.
REFERENCES Peter Kueng (2000). Process performance measurement
system - a tool to support process-based organizations.
Andrews, P. (2002). An Introduction to Mathematical Logic TOTAL QUALITY MANAGEMENT, 11(1):67–85.
and Type Theory: To Truth Through Proof. Berlin: Popova, V. and Sharpanskykh, A. (2010). Modeling organi-
Kluwer. zational performance indicators. Information systems,
Berler, A., Pavlopoulos, S., and Koutsouris, D. (2005). 35(4):505–527.
Using key performance indicators as knowledge- Regev, G. and Wegmann, A. (2011). Revisiting Goal-
management tools at a regional health-care authority Oriented Requirements Engineering with a Regulation
level. IEEE Transactions on Information Technology View. Springer. LNBIP, 109):56–69.
in Biomedicine, 9(2):184–192.
Robert Kaplan, and David Norton (2001). Transforming the
Dardenne, A., van Lamsweerde, A., and Fickas, S. (1993).
Balanced Scorecard from Performance Measurement
Goal-directed requirements acquisition. Sci. Comput.
to strategic management: Part I. Accounting Horizons,
Program., 20(1-2):3–50.
pages 87–104.
Frank, U., Heise, D., and Kattenstroth, H. (2009). Use of a
Roubtsova, E. (2013). Protocol Model of the KPIs from
domain specific modeling language for realizing ver-
”The programme ”Improving Access to Psychological
satile dashboards. In: Tolvanen JP, Rossi M, Gray
Therapies. http://www.open.ou.nl/elr/IAPT.zip.
J, Sprinkle J (eds) Proceedings of the 9th OOPSLA
workshop on domain-specific modeling (DSM). Strecker, S., Frank, U., Heise, D., and Kattenstroth, H.
(2012). MetricM: A modelling method in support of
Garengo, P., Biazzo, S., and Bititci, U. S. (2005). Perfor-
the reflective design and use of performance measure-
mance measurement systems in SMEs: A review for
ment systems. Springer, Information Systems and e-
a research agenda. International Journal of Manage-
Business Management, 10:241–276.
ment Reviews, 7:25–47.
Golfarelli, M. (2009). From User Requirements to Concep-
tual Design in Data Warehouse Design - a Survey. In
Data Warehousing Design and Advanced Engineering
Applications: Methods for Complex Construction. L.
Bellatreche (Ed.), pages 1–14.
Gossler, G. and Sifakis, J. (2003). Composition for Compo-
nent Based Modeling. Springer,LNCS, 2852):443–46.