See

discussions, stats, and author profiles for this publication at:

https://www.researchgate.net/publication/259925122

How to Measure Public

Administration Performance

Article in Public Performance & Management Review March 2012

DOI: 10.2307/23208797

CITATIONS READS

7 365

3 authors, including:

Wouter Van Dooren Zsuzsanna Lonti

University of Antwerp Organisation for Economi
66 PUBLICATIONS 454 16 PUBLICATIONS 160
CITATIONS CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Wouter Van Dooren on 18 July 2014.

The user has requested enhancement of the downloaded file.

 HOW TO MEASURE PUBLIC
ADMINISTRATION PERFORMANCE
A Conceptual Model with Applications for Budgeting,
Human Resources Management, and Open Government
WOUTER VAN DOOREN
CHIARA DE CALUWE
University of Antwerp
ZSUZSANNA LONTI
Organization for Economic Cooperation and Development

ABSTRACT: The economic crisis provides some insights on the role of

measurement systems. As shown by the ongoing discussion of credit rating agencies
by political actors and in the news media, measurement is not a neutral device, but
an active agent in societal processes. Comparative measurements of international
public administration are as frail, technocratic, and overly aggregated as bond
ratings, but nonetheless are increasingly used by journalists, aid organizations,
foreign investors, and, indeed, rating agencies to hold governments accountable.
Better measurement is needed in public administration performance. This article
builds on study reports from the OECDs Government at a Glance project to
address the issue of how to measure public administration performance. The fields
of budgeting, human resources management, and open government illustrate both
the potential and the challenge of such measurements.

KEYWORDS: budgeting, human resources management, open government,

public administration performance

The economic crisis provides some insights on the role of measurement systems
in society. Most notably, the role of credit rating agencies has been discussed by
political actors and in the news media.1 This discussion shows that measurement
is not a neutral device, but an active agent in societal processes. As in the case
of bond ratings, comparative measurement of international public administration
seems frail, technocratic, and overly aggregated (Arndt, 2007; Arndt & Oman,
2006; Van de Walle, 2006). Nonetheless, these indicators are increasingly being
used by journalists, aid organizations, foreign investors, and, indeed, rating agen-
cies for holding governments accountable. Hence, better measurement of public
administration performance is needed. The discussion that follows builds on study

Public Performance & Management Review, Vol. 35, No. 3, March 2012, pp. 489508.
2012 M.E. Sharpe, Inc. All rights reserved.
1530-9576/2012 $9.50 + 0.00.
DOI 10.2753/PMR1530-9576350306 489
490 PPMR / March 2012

reports from the Organization for Economic Cooperation and Developments

(OECDs) Government at a Glance project to address the issue of how to measure
public administration performance. The fields of budgeting, human resources
management, and open government are used to illustrate both the potential and
the challenge of measuring public administration performance.
The current economic crisis is also a crisis of measurement.2 In a 2005 cartoon
by Randy Glasbergen, a banker offers a couple an interest-only mortgage, bal-
loon mortgage, reverse mortgage, upside-down mortgage, inside-out mortgage,
loop-the-loop mortgage, and a spinning double-Axel mortgage with a triple-Lutz
(Mortgage Cartoon Glasbergen, 2005). The cartoon appeared two years ahead of
the collapse of Northern Rock in the UK and three years before the height of the
crisis in 2008. However, Glasbergen and other, more systematic observers, such as
Roubini and Taleb, were well aware of problems in the housing mortgage market.
Measurement systems of both the public and private sectors failed to detect this
undercurrent. The economic measurement models used by the national banks and
the planning bureaus did not capture the trend, just as the credit agencies failed
to adequately gauge the real value of the assets under scrutiny. Today, national
creditworthiness ratings are being discussed in response to notable downgrades
of several eurozone countries along with Standard & Poors U.S. credit rating
downgrade.
Some critiques target the failure of measurement systems to reflect reality. An
additional step is to seek the causes of the crisis in the measurement systems. The
AAA ratings for complex collateralized debt obligations (CDOs), for instance, are
blamed for having been at least a catalyst for the crisis. Crotty (2009) asserts that
the recent global financial boom and crisis might not have occurred if perverse
incentives had not induced credit-rating agencies to give absurdly high ratings
to illiquid, nontransparent, structured financial products. Interestingly, the same
mechanism of bad ratings by the agencies before the crisis and overly strict ratings
after the crisis has also been documented for the East Asian financial meltdown in
1997 (Ferri, Liu, & Stiglitz, 1999). Moreover, rating agencies power and lack of
accountability to the public was also discussed before the crisis (Kerwer, 2005).
For present purposes, there is no need for a thorough analysis of the ins and outs
of the economic crisis. Yet the crisis does remind us that measurement systems
are not neutral controls for policymakers in the cockpit of society. Measurement
systems are themselves agents, and influence behavior through their workings.
When measurement systems are used to hold organizations to account, it should
be kept in mind that they influence the behavior of account givers as well as ac-
count holders. Therefore, both a critical attitude toward existing indicators as well
as continuous efforts to improve measurement seem warranted. This is also true
concerning performance measurement for public administration.
Existing indicators on public administration performancewhich usually go
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 491

under the banner of governance indicatorshave been subject to some criticism.

This is true even at the national level, but far more so for the purpose of interna-
tional comparisons (Arndt, 2007; Van de Walle, 2006). One of the main critiques
of governance rankings targets their validitydo they measure what they claim
to measure? Rankings such as the World Bank governance indicators comprise
a broad and unintelligible compilation of opinion surveys, risk assessments, and
sector indicators (Arndt & Oman, 2006). Typically, nonperceptual indicators on
the systems core functions are lacking. As a result, there is a risk of constructing
an image of a countrys performance based on indicators that do not accurately
reflect its performance. If these performance indicators are used for accountabil-
ity purposes in a subsequent phase of governance, the consequences of invalid
measurement become real.
The present article does not offer further critique but, rather, proposes possible
measurement improvements. The text is based on several research reports commis-
sioned by the OECDs governance directorate, which is currently undertaking the
large project called Government at a Glance. The focus is on doing measurement,
rather than on further elaborating the functions and dysfunctions of measurement
(for the latter, see, e.g., Hood, 2006; Radin, 2006). No subjective stance is taken
on the utility or potentially negative effects of using performance measures.
For public administration more than for fields such as education or health, an
additional difficulty for measurement is the conceptual confusion on what public
administration performance actually means. This article begins by exploring
the conceptual foundation for indicators of public administration performance.
Based on this foundation, it discusses possible performance indicators in three
areas of public administration: budgeting, human resources management, and
open government.

The Conceptual Foundation of Public Administration Performance

As suggested above, it is first necessary to sort out a number of definitional is-

sues. The primary task is to define performance. There seems to be a common
mainstream understanding of performance, but this definition is process-oriented
and nonsubstantial. In order to measure public administration performance, a
substantial definition of performance is required.

THE MAINSTREAM DEFINITION OF PERFORMANCE

Performance is usually defined in terms of the outcomes and outputs that follow
from a public production process (Hatry, 1999). This model seems to provide the
dominant vocabulary used by public administration researchers and practitioners
when discussing performance. Although some terminological issues remain, and
although some analysts will primarily emphasize the importance of contextual
492 PPMR / March 2012

factors, the main building blocks have become widely accepted foci of public
administration theory and practice.
Outcomes are the result of activities that convert inputs to outputs. The transfor-
mation of inputs, such as financial and human resources, to activities is mediated
by the structure of government, cultural predispositions, and institutional and
managerial arrangements. Outputs are the goods and services that public orga-
nizations supply in response to demand. Outcomes are the consumption of the
goods and services (intermediate outcomes) as well as the effects this consumption
entails (final outcomes).
The criterion for assessing outcome is added value (Moore, 1995). The added
value of a private firm is the result of the aggregation of individual decisions to
consume a service or good at a given price. A firms profit can be conceptualized
as its outcome in society, since it is the sum of the values that individuals attach
to a good or service, minus the costs of production. Free-rider problems3 and the
positive and negative externalities of consumption,4 however, mean that one cannot
rely on individual consumption decisions for all goods and services (Musgrave,
1959). For public services, although the criterion of added value remains intact, the
notion of public value replaces private value. In the absence of monetary profits,
it is much more difficult for public organizations to assess outcomes.

SUBSTANTIVE APPROACHES TO PERFORMANCE

A problem with the mainstream definition of performance is its nonsubstantial na-
ture. The definition of performance as outputs and outcomes of public services does
not tell us what these outputs and outcomes should be. Different ideologies will
have different views about public services, such as whether or not to redistribute
outputs, how many services to provide, and whether or not regulation is needed.
As such, the mainstream definition is a purely analytical concept. Different actors
can define performance differently without invalidating its conceptual definition
in terms of output and outcome (Van Dooren, Bouckaert, & Halligan, 2010).
Operationalizing public administration performance calls for a thick substan-
tive approach to performance that includes a variety of public values. The tendency
of measurement systems to develop tunnel vision, similar to the narrow focus of
credit-rating agencies, could be countered by maintaining an open conceptual
view on what defines public administration performance. In a seminal article on
the New Public Management (NPM), Hood (1991) proposes a classification with
three clusters of public values. He makes the point that NPM reforms almost exclu-
sively stress public values of one type, often to the detriment of other values. The
public-values literature expanded further in the following decade. Jorgensen and
Bozeman (2007), for instance, developed an inventory of more than 70 different
values in the public-values universe. For reasons of parsimony, Hoods three broad
value groups are utilized, since they also seem to reflect general dimensions of
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 493

Table 1. Approaches to Performance and

Consequences for Development of Indicators

Product Procedure Regime

How can
performance be Sufficiency (provision (Mostly) deficiency (absence of Deficiency (system
observed? of services) fairness and honesty) failure)
What is the
incidence of these
observations? (Mostly) repeated (Mostly) repeated One-time
Developing
indicators Feasible Difficult Very difficult

Source: Based on Van Dooren, Bouckaert, & Halligan, 2010.

performance (Van Dooren, Bouckaert, & Halligan, 2010). Three broad but distinct
substantial definitions of performance might be:
s 0RODUCT PERFORMANCE REmECTS SUCCESS IN MATCHING RESOURCES TO DElNED TASKS
Government has to operate in an economical, efficient, and effective way. Product
performance reflects what Piotrowski (2010) calls mission-based values.
s 0ROCEDURAL PERFORMANCE REmECTS SUCCESS IN KEEPING GOVERNMENT FAIR AND HONEST
Government has to pursue honesty, fairness, and mutuality through the preven-
tion of distortion, inequity, bias, and abuse of office. Procedural performance is
not mission-based (Piotrowski, 2010).
s 2EGIME PERFORMANCE REmECTS SUCCESS IN KEEPING THE PUBLIC SECTOR ROBUST AND
resilient. Government has to operate even in adverse worst-case conditions
and to adapt rapidly in response to crisis and change. Regime performance is
another form of non-mission-based performance.
It is generally easier to measure product performance than process or regime
performance (see Table 1). For production, resources are allocated to defined
tasks and indicators are a means to assess goal attainment. This is not to say
that measurement is unproblematic, but at least conceptually it is unambiguous.
Processes in production cycles are usually repeated, which allows for learning
in measurement. Measuring levels of fairness and honesty becomes substantially
more complicated. Such measures will predominantly be about the absence of
fairness and honesty, measuring fraud, corruption, favoritism, and so forth. Typi-
cally, these activities are under the radar. They are, however, recurring events,
which makes the development of indicators somewhat easier. Indicators for ro-
bustness and resilience are even more complicated, since failures and worst-case
conditions that really put a strain on robustness are uncommon. As an alternative
to outcome measures, measures of capacity could be developed, but they are not
performance indicators (Hall, 2008).
The selected public administration dimensions discussed below touch upon two
of the three approaches to public values. Human resources management and bud-
494 PPMR / March 2012

geting are typically cyclical, recurring activities. A human resources management

department has to hire, evaluate, motivate, and terminate staff on a continuous basis.
Timely and accurate budgets have to be developed by budget departments every
year. Human resources management and budgeting activities have a good deal of
product performance, and hence it can be expected that measuring performance
on these dimensions is viable. The main product of a budget department is the
budget. For a human resources department, hires are an important product. Open
government, on the contrary, has a more procedural, nonmission focus, and hence
more difficulties can be expected in developing indicators. For regime performance,
it is not at all clear whether it is possible to develop performance indicators; and
in consequence this dimension is left out of the scope of the present article.

Public Administration Performance

The discussion in this section defines public administration performance. The

development of performance indicators for public administration requires an
understanding of two defining features of the nature of public administration.
First, public administration is about enabling rather than delivering. Public
administration almost never provides final goods and services. Public administra-
tion, however, is a precondition for the successful operation of other government
departments. It is government for government rather than government for the
citizens. That takes nothing away from its importance. Public service delivery is
a chain of inputs and outputs. Clearly, public administration arrangements are to
be found earlier in the chain. Schools need to be staffed and financed before they
can provide teaching.
The chain of impact is schematically represented in Figure 1. Public admin-
istration processes, including typical horizontal functions such as financing and
human resources management, are a precondition for functions performed by
line departments and agencies. Public administration outputs are inputs for the
functional processes. For instance, ethics training sessions are an output for an
ethics division but an input for a social security department. The outcome of the
training sessions is better awareness of ethics issues within the administration
of social security. In the same way, the number of administered allowances is an
output of the social security administration that is an input for societal processes.
If the allowance protects people from poverty, then it is a policy outcome.
The step from a public administration process such as ethics to a policy outcome
such as poverty in society is a rather big leap. It would be hard to demonstrate
how integrity training has an impact on poverty statistics. However, by taking
into consideration the intermediate processes in the chain, it might be possible to
bridge the gap. Public administration processes should, in the first place, improve
the quality of public administration and enable others to govern society in a more
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 495

Figure 1. A Chained Approach to Outcome

direct way. Public administration performance should reflect this position.

A second typical feature of public administration is its crosscutting nature.
Precisely because it is an enabler, public administration has an impact on all other
policy sectors. This also explains why it is so difficult to implement government-
wide administrative policies (Verhoest, Bouckaert, & Peters, 2007). Often, such
policies are perceived to run counter to the vested interests and practices of the
policy sectors. For measurement, the crosscutting nature of public administration
complicates data collection and standardization.
Measuring the enabling and crosscutting roles of public administration has
two consequences. First, the outputs of public administration processes are the
inputs for substantive processes in line departments and agencies. In order
to identify output, it is necessary to ask what products and services public
administration processes deliver to line agencies. Second, the outcomes of
public administration primarily have to reflect its ability to facilitate policy
sectors. To identify outcome, it is necessary to ask whether public administration
processes succeed in enabling performance in other sectors.

MEASURING PUBLIC ADMINISTRATION PERFORMANCE:

BUDGETING, HUMAN RESOURCES MANAGEMENT, AND OPEN
GOVERNMENT
The conceptual approach described above will now be applied to three important
dimensions of public administration: budgeting, human resources management,
and open government. Maturity of measurement differs for the three dimensions.
Measurement in budgeting is quite well developed, which allows for relatively
robust and comparative indicators. Human resources management is also a dimen-
sion in which several measurement efforts exist, but these are not yet developed
for international comparative purposes. Open government is, for measurement
496 PPMR / March 2012

concerns, a field of experimentation. Some prospects for indicator development

are discussed below. The implications for their use are considered in the discus-
sion and also in the conclusions.
Budgeting
A budget is defined as a comprehensive statement of government financial plans,
including expenditures, revenues, deficit or surplus, and debt. The budget is the
governments main economic policy document, indicating how it plans to use
public resources to meet policy goals (OECD, 2006). Budgeting activities follow a
cycle of events or stages in making decisions about the budget, and implementing
and assessing those decisions. The cycle usually has four stages: formulation, ap-
proval, execution, and audit. What are the outcomes of these budgeting activities?
What constitutes successful and unsuccessful budgeting?
Budgeting has multiple objectives (Gray, Jenkins, & Segsworth, 2001; Sterck,
2007). Three conventional budget functions are authorization (legal function), al-
location (policy function), and management (management function). By approving
the budget, the legislature authorizes the executive to spend money and to collect
taxes. The question, then, is whether budgets succeed in informing executives
about the decisions they have to make. More generally, the budget has to inform
the general public about governments spending and revenue collection inten-
tions. A second objective of a budget is to allocate resources to policy fields while
maintaining fiscal balance at the macrolevel. The question is whether the budget
is successful in allocating the right amount of resources to public services and,
at the same time, is successful in ensuring a structural balance between revenues
and expenditures. A third function is management. Managers have to use budgets
to control their organization.
Performance indicators for budgeting could first look at the variance between
projected (intended) and actual revenues and the variance between projected and
actual expenditures (Colonna & Puma, 2003). These measures give an indication
of the quality of the estimates. If the variance becomes too high, the impact of
the budget as a tool for policy, management, and legislative control erodes. At
the microlevel, managers may no longer be able to plan spending and investment.
Moreover, overestimation of expenditures may lead to end-of-the-year spending
also known as December fever (Douglas & Franklin, 2006; Tarschys, 2002).
Managers typically spend all of their appropriated budgets, partly because they
fear sanctions in the subsequent budget round. At the macrolevel, economic and
monetary policy will be based on faulty assumptions (Rubin, 1997). The Govern-
ment Performance Project (Maxwell School of Citizenship and Public Affairs,
2002) uses the following measures;
a. Revenue estimation accuracy = (actual revenue estimated revenue)/estimated
revenue
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 497

b. Expenditure estimation accuracy = (actual expenditure estimated expenditure)/

estimated expenditure

Expenditures and revenues can be over- or underestimated. Cautious budgeting

would suggest a higher tolerance for the underestimation of revenues and the over-
estimation of expenditures. A budget in the public sector is also a tool for legisla-
tive control over the executive. The legislatures appropriations provide authority
under law to the executive to spend public funds up to a set limit and for a specified
purpose. In this respect, a budget reflects not only a projection but also an intention
(Kristensen, Groszyk, & Bhler, 2002). Overestimation of expenditures implies
that appropriated budgets are not used. This may be the result of bad budgeting,
but may also point to implementation problems and thus an infringement on the
legislatures decision to spend a given amount on a particular program.
A second set of budget performance indicators reflects budgeting transparency.
Blondal (2003) observes that the budget is the governments principal policy docu-
ment, wherein the governments policy objectives are reconciled and implemented
in concrete terms (i.e., the authorization function). Budget transparencyopenness
about policy intentions, formulation, and implementationis therefore at the
core of good public administration and may even influence more general political
indicators, such as voter turnout (Benito & Bastida, 2009).
Researchers at international institutions and nonprofits, as well as academics,
have attempted to develop indices of budget transparency that combine several
indicators (Alt & Lassen, 2006; De Renzio & Masud, 2011). According to Blondal
(2003), transparency has three essential elements. The first is the systematic and
timely release of budgetary data. This is what is traditionally associated with bud-
get transparency. Budget transparency is seen as an output, that is, the release of
budgetary data. The second element is an effective role for the legislature. It must
be able to scrutinize the budget reports and independently review them. It must be
able to debate and influence budget policy and be in a position to effectively hold
the government to account. The third element is an effective role for civil society,
through the media and nongovernmental organizations. Citizens must be in a posi-
tion to influence budget policy and must be in a position to hold the government
to account. In many ways, it is a role similar to that of the legislature albeit only
indirectly. The second and third elements discuss the roles of the legislature and
society, respectively. Active scrutiny by the legislature and society may be the
result of efforts to increase budget transparency, but also of characteristics of the
polity or broader trends (Sterck, 2007).
Fiscal stability may be a third budgeting performance indicator. As argued
above, one of the key budget functions is allocation: to match spending priori-
ties with estimated revenues. Many institutional arrangements contribute to this
objectivefiscal rules, economic assumptions, expenditure frameworks, fiscal-risk
498 PPMR / March 2012

assessments, and so on. Therefore, a balanced budget seems to be a valid indica-

tor of the outcomes of these processes. There is, however, a delicate balance to
maintain in practice as well as in the indicator set. Institutions for safeguarding
balanced budgets, combined with the growth of entitlements and taxing limitations,
can lead to an erosion of allocative efficiency (Marlowe, 2009).
Audits close the budgeting cycle. They are expert examinations of legal and
financial compliance or performance (Raaum & Morgan, 2001). Performance of
compliance audits might, for instance, be a decrease in the number of corrections
auditors have to put forward. The results of such performance measures, however, are
usually hard to interpret. Does a higher number of infringements imply that auditors
detect more mistakes or that they fail to help administrations to avoid mistakes?
Several supreme audit offices have attempted to measure audit outcomes (for
an overview, see Lonsdale, Wilkins, & Ling, 2011). The UK National Audit Office
measures the financial savings due to auditing work relative to the money invested
in the auditing agency. It identified a savings of 582 million in 2007 as a result
of audit work, which exceeded the 8:1 target by some 22 million (National Audit
Office, 2007). The National Audit Office, however, stresses that many benefits are
not financial, and thus not measured. The U.S. Government Accountability Office
(GAO) reports financial benefits of $46 billion, which is a 94:1 ratio relative to every
dollar invested in audit (2007). The GAO also measures nonfinancial benefits. It
measures (1) how often agencies act on GAO information to improve services to the
public, (2) when information the GAO provides to the Congress results in statutory
or regulatory changes, and (3) when core business processes improve at agencies and
government-wide management reforms are advanced by the GAOs work. Beyond
this, the GAO measures the percentage of past recommendations implemented. In
fiscal year 2007, 82% of the 2003 recommendations had been implemented. A final
outcome indicator from the GAO report relates to client satisfaction. It counts the
number of congressional hearings at which the GAO testified. The underlying argu-
ment is that congressional attention is an indication of responsiveness and impact.
Budgeting has been subject to substantial international standardization, in particular
within the European Union (Eurostat, 2001; United Nations, 1993). Hence, some
relatively robust international comparative-performance indicators can be derived
from the budgeting system. Most EU countries seem to follow the Eurostat guidelines,
with the notable exception of Greece, which has resorted to unprecedented creative
accounting. The reaction of the European Union seems to be to impose more control
and more intensive measurement.5 Again, an argument pro measurement.
Human Resources Management
One of the most prominent public administration issues is public personnel
management. The OECD (2007) identifies four dimensions of human resources
management policy: (1) workforce planning and management, (2) management
of staff performance, (3) flexibility and coherence of human resources manage-
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 499

ment rules across governments, and (4) core values. The latter were discussed
above in the section dealing with the guiding principles of outcome indicators.
The first three dimensions are used as a point of departure for the definition of
performance indicators.
(1) The objective of good workforce planning and management is to ensure an
appropriately dimensioned, appropriately structured, and representative workforce,
able to meet changing labor needs in the context of changing demands and rapid
developments in the wider labor market (OECD, 2007). A number of indicators
can be derived.
The workforce needs to be appropriately dimensioned and structured, with the
right size and education. At first sight, size and education are input indicators.
Yet it is the adverb appropriately that gives this objective an outcome flavor.
Well-functioning human resources management arrangements should bring about
a workforce that is adequately sized and educated in order to ensure undisrupted
and financially sustainable service delivery. Indicators that reflect the appropriate-
ness can be considered outcomes.
An important outcome of successful human resources management policies
in a competitive labor market is governments being an employer of choice
(Vandenabeele, 2008). A 2002 OECD study assessed this issue by asking
whether governments expect problems in recruitment and retentionnow or
soon. The study also asked whether there are critical skill shortages (OECD,
2002).
Duration of hiring is another potential indicator of workforce management.
The U.S. Office of Personnel Management used this indicator to assess the human
resources management performance of government agencies (2007). It defines the
indicator as the agencys percentage of hires within a 45-day timeframe. It uses
a hiring time-frame model to uniformly define the indicator.
The U.S. GAO uses hire rate and acceptance rate to assess the impact of hu-
man resources management practices. The hire rate is the ratio of the number of
people hired to the number the agency planned to hire as defined in its workforce
plan. The acceptance rate is the ratio of the number of applicants accepting of-
fers to the number of offers made. Acceptance rate is seen as a proxy for GAOs
attractiveness as an employer and an indicator of competitiveness in bringing in
new talent (GAO, 2007).
Many governments strive for a workforce that reflects society (Gualmini,
2008). Evidence of the impact of diversity on performance efficiency is diffuse
(Andrews, Boyne, Meier, OToole, & Walker, 2005), but representation also
reflects other public values, such as social mobility of minorities. Indicators on
equal opportunities reflecting the gender balance and minority representation are
thus potential outcome indicators.
(2) The objective of good staff performance management is to ensure a suitably
empowered and highly motivated civil service that is flexible and collaborative
500 PPMR / March 2012

and provides services in a cost efficient manner (OECD, 2007). This outcome
suggests a second set of performance indicators. First, consider staff satisfac-
tion. A satisfied workforce could be seen as an outcome for a human resources
department. Sick leave/absenteeism is another example. A reasonable measure
of successful human resources management is a low absenteeism rate. External
factors that need to be taken into account might include the age structure of the
workforce (younger people are healthier) and the task environment (idea-based
and relational tasks are probably less exhausting and more motivating than routine
physical jobs) (Hackman & Oldham, 1980). A third example may be indicators
on retention and turnover. Organizations invest in hiring and training people, and
thus presumably want to retain them. High retention rates may reflect workplace
attractiveness. Clearly, an important external influence will be the condition of
the general labor market and the resulting degree of private-sector competition
for critical skills.
(3) The objective of a good balance between human resources management
arrangements is to ensure that transaction costs in negotiating shared responsi-
bilities between governments are minimized, and that an active labor market is
supported for staff with distinctive public sector skills and competencies (OECD,
2007). A third set of indicators is implied in this outcome. An active labor market
is related to workforce mobility. First, there may be a learning effect. Mobility
broadens horizons, leads to the circulation of ideas, and in this way is an antidote
against tunnel vision. Second, mobility is in some instances a strategy against cor-
ruption and unethical behavior, especially in inspection services. The time spent
in a particular sensitive position is too short to develop unsound relations with
target groups (Abbink, 2004). Third, mobility may be a matter of allocation. Some
departments may have staff shortages while others have redundancies. Internal
labor markets should sort out these imbalances.
The kinds of indicators in this field are typically found in many larger orga-
nizations, both private companies and public bureaucracies. Hence, there seems
to be fertile ground for solid international and cross-sectoral comparison. The
main problem for comparative purposes is that, unlike for budgeting, there are
no internationally agreed upon standards and definitions.
Open Government
A third field of public administration to be highlighted as a case of comparative
measurement concerns the openness of government. Open government mostly re-
lates to procedural performance, and therefore performance measurement becomes
more difficult. It is a field characterized by measurement experimentation.
The OECD identifies four dimensions of accountability and openness (OECD,
2001). Transparency is the ability of the government to ensure that its actions,
and those who are responsible for its actions, are exposed to public scrutiny and
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 501

Inclusive government

Responsive government

Accessible government

Transparent government

Figure 2. Open and Accountable Government: Four Dimensions

challenge. Accessibility in the public sector is achieved when the government can
ensure everyones capacity to obtain information and to utilize services at any
time, anywhere, and in a user-friendly manner. A responsive government is one
that can and will react to new ideas, demands, and needs of citizens. Inclusiveness
aims to ensure the broadest base of participation possible.
The four dimensions of openness and accountability are assumed to build on
each other (see Figure 2). Transparent government is a precondition for accessible
government. Citizens need to know their government in order to access it, and
citizens need to access government in order for government to respond to them.
Inclusiveness seems to be a different dimension, which assesses to what extent
government is transparent for everyone, accessible for everyone, and responsive
to everyone. The underlying logic of this scheme resembles Arnsteins ladder of
citizen participation (Arnstein, 1969): the higher on the ladder, the higher the level
of participation. In this model, it can be argued, open and accountable governments
are at the top of the pyramid.
Transparency is a key tenet of contemporary performance and accountability
relations (Dubnick, 2005; Piotrowski & Rosenbloom, 2002). Freedom of in-
formation laws are a key institutional arrangement for achieving transparency.
Performance indicators might reflect whether citizens (and companies and media)
actually refer to these laws or whether they remain dead letters. The catch is that in
very transparent public administration systems, there is no need to use information
laws. Interpretation of performance information in context and through dialogue
seems warranted (Moynihan, 2008).
The publication of annual reports, performance data, strategic plans, and legis-
lative timetables is a means of opening up government activity to the public eye.
Performance indicators could measure the extent to which target groups are actually
using available information. A good definition of target groups is necessary. It is
not realistic to expect ordinary citizens to read annual reports and strategic plans
in their leisure time. It is realistic, however, to expect media and interest groups to
502 PPMR / March 2012

scrutinize public documents.6 These mediators play a crucial role in safeguarding

transparency in democratic societies. Expert assessments are likely to be a good
way to proceed in measuring the quality of these reports.
Accessibility. Countries mainly safeguard accessibility through administrative
law, but ombudsmen, citizen or service charters, and policies to reduce red tape also
have accessibility as an objective. It is very difficult to measure the direct impact
of these arrangements. Is the number of complaints handled by the ombudsman an
indication of problems with accessibility, or does it simply mean the ombudsman
is better known? A more general indication of accessibility problems may be the
length of the waiting list for services. The problem of this indicator is that waiting
lists usually cannot be attributed to open government policies. Although the latter
may help to put the issue on the agenda, waiting lists are more likely caused by
lack of capacity or extraordinary demands on a service.
Responsiveness is pursued through public consultation or active public par-
ticipation throughout the policymaking process. According to several authors,
responsiveness should be more than a mere client orientation. They stress the
importance of citizen-government collaboration (Vigoda, 2002) and coproduction
(Bovaird, 2007), which reflect the quality and depth of the relationship rather than
the power or the number of participants. Measuring the performance of partici-
pation processes is thus a delicate matter. One possible indicator could measure
the range of backgrounds and personal profiles of the participants. In an optimal
scenario, participation is widespread and representative, but again, the interpreta-
tion of the results is not straightforward.
To conclude, the performance of arrangements aimed at an open and account-
able government is especially hard to measure. Outcomes can only be observed
within societyusing public opinion data from citizens, businesses, and privileged
observers such as interest groups. These data are perceptual and nonstandardized.
The standardized surveys that do exist, such as Eurobarometer and the World Values
Survey, often face the problem of translating core concepts such as legitimacy and
trust (MacQuarrie, 2008). The fallback option is to measure citizens use of open
government arrangements under the premise that arrangements need to be used
in order to be effective. Here, international comparison is plagued by differences
in the institutional framework that arranges access to government.

Discussion and Conclusion

The present article began with the observation that the economic crisis is also a crisis
of measurement in the ratings industry. Yet ratings are unlikely to disappear. More
generally, measurement seems to be a key tenet of modern society (Porter, 1995;
Scott, 1998). Performance indicators are increasingly used to make complex realities
tangible and to hold actors accountable. The vast array of activities and performance
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 503

TXDOLW\RIPHDVXUHPHQW

KLJK

EXGJHWLQJ

ORZ KLJK
XVH
+50

JRYHUQDQFH FUHGLW
RSHQJRY
LQGLFDWRUV UDWLQJV

ORZ

Figure 3. Measurement Quality and Use

in hospitals, schools, universities, local governments, and the like are captured by a
limited number of indicators in performance contracts, league tables, and star ratings
(Bevan & Hood, 2006; Moynihan, 2009; Radin, 2006). The story of the rating agencies
also shows that the impact of these indicator regimes can be substantial. In order to
ensure that accountability for performance is providing the right effects in government
and society, the quality and diversity of indicators becomes crucial.
This is also the case for public administration indicators. The quality of public
administration indicators is often problematic, in particular when they are internation-
ally comparative. Nonetheless, they are increasingly used, among other purposes, to
inform funding, investment decisions, the rating of government bonds, and media
coverage (United Nations Development Programme, 2004). As a result, the develop-
ment of diverse and valid comparative indicators seems to be warranted. This article
builds on OECD initiatives with a conceptual discussion of indicators in the fields
of budgeting, human resources management, and open government.
The cases show that public administration performance measurement is at
least conceptually conceivable for all three fields. There are differences between
the cases, however. For budgeting, good comparative performance indicators can
be developed. The high level of standardization and cyclical nature of budgeting
contributes to the potential. For human resources management, the development
of performance indicators is also relatively straightforward, but international
comparison may be more difficult due to the lack of standardization. For open
504 PPMR / March 2012

government, the procedural character seems to be a hindrance to the development

and interpretation of indicators. For all three fields, performance indicators will
only make sense in context, suggesting that there is need for dialogue on indica-
tors (Moynihan, 2008). Accountability for performance should not be confined
to accountability for performance indicators.
The article mainly deals with the validity and robustness of international compari-
sons of public administration. Yet, as is argued above, the use of performance indicators
is an equally important dimension. Although use was not the main object of the article,
it ends with some reflections on how measurement quality and use may interact.
Figure 3 combines measurement quality and its use and gives a rough assessment
of where to locate the cases of budgeting, human resources management, and open
government. It also plots the general governance indicators (for an overview, see Van
de Walle, 2006) and the credit ratings of CDOs that sparked the economic crisis.
International comparative data on open government and human resources man-
agement tend to be of relatively low quality (somewhat better on quality for human
resources management). This, however, is not problematic, because the low intensity
of use leaves room for experimentation. Budgeting has higher quality, but international
comparisons of budget data are used quite intensively by international institutions. The
case of low use and high quality is mainly a missed opportunity. The most problematic
quadrant is high use of low-quality data. The credit ratings clearly were to be situated
in this category, and several governance indicators also seem to head this way.
Use and quality of data are intertwined. Once data are used intensively and
have an impact, they become institutionalized. At that point, it becomes difficult
to improve quality, because changes in definitions and methods also touch upon
interests and challenge routines. Efforts to alter GDP calculationsanother inten-
sively used indicatorare a case in point (Van Dooren, 2009). In other words, it is
very difficult to move upward from the lower-right quadrant in Figure 3. Another
implication is that being in the lower-left quadrant should be cherished. This is the
period in which the DNA of the performance indicators can be established through
experimentation. For indicators of public administration performance, there is still
room for experimentation. Finally, the admittedly rough analysis of the quadrants
may call for more longitudinal research on how indicator sets are developed over
time, how invalid data are able to construct a reality, what determines the use of
valid data, and where the indicator sets originate.

Notes
1. The Obama administration has been very critical of the quality of Standard & Poors ratings
since the U.S. downgrade (President Obamas statement, 2011). In the media, there have been
some less critical accounts of the role of rating agencies (e.g., Tasker, 2011), together with more
critical and very critical perspectives (e.g., Credit-rating agencies, 2011; Rushe, 2011).
2. The views expressed in this article are entirely those of the authors and do not reflect
any opinion of the OECD or its member governments.
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 505

3. A free rider is someone who enjoys the benefits of a public good without bearing the
cost. The image of someone who uses public transport without paying is illustrative.
4. Externalities are costs and benefits attributable to an activity that are not reflected in the
price of the goods or services being produced.
5. This is mainly done through the establishment of the so-called European semester, a
more frequent follow-up of national budgets by the EU, with new incentives and sanctions for
countries that do not provide qualitative budget data (Europa Press Release, 2011)
6. There is a parallel with private-sector annual reports. Only a selected audience of fi-
nancial journalists, analysts, and interest groups have the time and competence to assess these
reports.

References
Abbink, K. (2004). Staff rotation as an anti-corruption policy: An experimental study.
European Journal of Political Economy, 20(4), 887906.
Alt, J.E., & Lassen, D.D. (2006). Transparency, political polarization, and political budget
cycles in OECD countries. American Journal of Political Science, 50(3), 530550.
Andrews, R., Boyne, G.A., Meier, K.J., OToole, L.J., Jr., & Walker, R.M. (2005). Rep-
resentative bureaucracy, organizational strategy, and public service performance: An
empirical analysis of English local government. Journal of Public Administration
Research and Theory, 15(4), 489504.
Arndt, C. (2007). The politics of governance ratings. Maastricht: Maastricht Graduate
School of Governance.
Arndt, C., & Oman, C. (2006). Uses and abuses of governance indicators. Paris: Organiza-
tion for Economic Cooperation and Development.
Arnstein, S.R. (1969). A ladder of citizen participation. Journal of the American Planning
Association, 35(4), 216224.
Benito, B., & Bastida, F. (2009). Budget transparency, fiscal performance, and political
turnout: An international approach. Public Administration Review, 69(3), 403417.
Bevan, G., & Hood, C. (2006). Whats measured is what matters: Targets and gaming in
the British health care sector. Public Administration, 84(3), 517538.
Blondal, J. (2003). Budget reform in OECD member countries: Common trends. OECD
Journal on Budgeting, 2(4), 725.
Bovaird, T. (2007). Beyond engagement and participation: User and community coproduc-
tion of public services. Public Administration Review, 67(5), 846860.
Colonna, A., & Puma, J. (Eds.). (2003). Paths to performance in state and local government:
A final assessment of the Maxwell School of Citizenship and Public Affairs. Syracuse,
NY: Campbell Public Affairs Institute.
Credit-rating agencies: Judges with tenure. (2011). Economist, August 11. Available at
www.economist.com/node/21525936/, accessed August 2011.
Crotty, J. (2009). Structural causes of the global financial crisis: A critical assess-
ment of the new financial architecture. Cambridge Journal of Economics, 33(4),
563580.
De Renzio, P., & Masud, H. (2011). Measuring and promoting budget transparency: The
open budget index as a research and advocacy tool. Governance, 24(3), 607616.
Douglas, J.W., & Franklin, A.L. (2006). Putting the brakes on the rush to spend down
end-of-year balances: Carryover money in Oklahoma state agencies. Public Budgeting
& Finance, 26(3), 4664.
Dubnick, M. (2005). Accountability and the promise of performance: In search of mecha-
nisms. Public Performance and Management Review, 28(3), 376417.
Europa Press Release. (2011). European semester: A new architecture for the new EU
506 PPMR / March 2012

economic governanceQ&A. Available at http://europa.eu/rapid/pressReleasesAction.

do?reference=MEMO/11/14/, accessed August 2011.
Eurostat. (2001). Handbook on price and volume statistics in the national accounts. Lux-
embourg: European Communities.
Ferri, G., Liu, L.G., & Stiglitz, J.E. (1999). The procyclical role of rating agencies: Evi-
dence from the East Asian crisis. Economic Notes, 28(3), 335355.
Gray, A., Jenkins, B., & Segsworth, B. (2001). Budgeting, auditing, and evaluation: Func-
tions and integration in seven governments. New Brunswick, NJ: Transaction.
Government Accountability Office. (2007). Performance and accountability report, 2007.
Washington, DC.
Gualmini, E. (2008). Restructuring Weberian bureaucracy: Comparing managerial reforms
in Europe and the United States. Public Administration, 86(1), 7594.
Hackman, J., & Oldham, G.R. (1980). Work redesign. Reading, MA: Addison-Wesley.
Hall, J.L. (2008). The forgotten regional organizations: Creating capacity for economic
development. Public Administration Review, 68(1), 110125.
Hatry, H.P. (1999). Performance measurement: Getting results. Washington, DC: Urban
Institute Press.
Hood, C. (1991). A public management for all seasons. Public Administration, 69(1),
319.
Hood, C. (2006). Gaming in target world: The targets approach to managing British public
services. Public Administration Review, 68(4), 515521.
Jorgensen, T.B., & Bozeman, B. (2007). Public values: An inventory. Administration &
Society, 39(3), 354381.
Kerwer, D. (2005). Holding global regulators accountable: The case of credit rating agen-
cies. Governance, 18(3), 453475.
Kristensen, J., Groszyk, W., & Bhler, B. (2002). Outcome-focused management and
budgeting. OECD Journal on Budgeting, 1(4), 735.
Lonsdale, J., Wilkins, P., & Ling, T. (2011). Performance auditing: Contributing to ac-
countability in democratic government. Cheltenham, UK: Edward Elgar.
MacQuarrie, C. (2008). Putting the demos back in democracy: Declining citizen trust in
government and what to do about it. Ottawa: School of Political Studies, University of
Ottawa.
Marlowe, J. (2009). Public financial engineering and its discontents. Public Performance
& Management Review, 32(4), 626630.
Maxwell School of Citizenship and Public Affairs. (2002). Paths to performance in state
and local government: A final assessment from the Maxwell School of Citizenship and
Public Affairs. Syracuse: Maxwell School of Citizenship and Public Affairs.
Moore, M.H. (1995). Creating public value strategic management in government. Cam-
bridge: Harvard University Press.
Mortgage Cartoon Glasbergen [image] (2005). Available at www.glasbergen.com/real-
estate-cartoons/, accessed 2007.
Moynihan, D.P. (2008). The dynamics of performance management: Constructing informa-
tion and reform. Washington, DC: Georgetown University Press.
Moynihan, D.P. (2009). Through a glass, darkly. Public Performance & Management
Review, 32(4), 592603.
National Audit Office. (2007). Annual report, 2007. London.
OECD. (2001). Citizens as partners: Information, consultation and participation in policy
making. Paris.
OECD. (2002). Public service as an employer of choice. Paris.
OECD. (2006). OECD budget practices and procedures database phase II: Final glos-
sary. Paris.
Van Dooren, De Caluw, and Lonti / MEASURING PUBLIC ADMINISTRATION PERFORMANCE 507

OECD. (2007). OECD reviews of human resources management in government: Belgium.

Paris.
Office of Personnel Management. (2007). Performance and accountability report: Fiscal
year 2007 (pp. 1229). Washington, DC.
Piotrowski, S.J. (2010). An argument for fully incorporating nonmission-based values into
public administration. In R. OLeary, D.M. Van Slyke, & S. Kim (Eds.), The future of
public administration around the world: The Minnowbrook perspective (pp. 2733).
Washington, DC: Georgetown University Press.
Piotrowski, S.J., & Rosenbloom, D.H. (2002). Nonmission-based values in results-oriented
public management: The case of freedom of information. Public Administration Review,
62(6), 643657.
Porter, T.M. (1995). Trust in numbers: The pursuit of objectivity in science and public life.
Princeton: Princeton University Press.
President Obamas statement on credit downgrade (2011). Washington, DC: White
House. Available at www.whitehouse.gov/photos-and-video/video/2011/08/08/
president-obamas-statement-credit-downgrade?page=6&v=accessibility/, accessed
September 2011.
Raaum, R.B., & Morgan, S.L. (2001). Performance auditing: A measurement approach.
Altamonte Springs, FL: Institute of Internal Auditors.
Radin, B.A. (2006). Challenging the performance movement: Accountability, complexity,
and democratic values. Washington, DC: Georgetown University Press.
Rubin, I.S. (1997). The politics of public budgeting: Getting and spending, borrowing and
balancing. Chatham, NJ: Chatham House.
Rushe, D. (2011). How do you rate the rating agencies? Guardian, March 31. Available at
www.guardian.co.uk/commentisfree/cifamerica/2011/mar/31/ratings-agencies-credit-
crunch/, accessed August 2011.
Scott, J.C. (1998). Seeing like a state: How certain schemes to improve the human condi-
tion have failed. New Haven: Yale University Press.
Sterck, M. (2007). The impact of performance budgeting on the role of the legislature: A
four-country study. International Review of Administrative Sciences, 73(2), 189203.
Tarschys, D. (2002). Time horizons in budgeting. OECD Journal on Budgeting, 2(2),
77104.
Tasker, P. (2011). How to make monkeys out of rating agencies. Financial Times, August
11. Available at www.ft.com/intl/cms/s/0/b8d489a8-c363-11e0-b163-00144feabdc0.
html#axzz1Ve5yc100/, accessed August 2011.
United Nations. (1993). Revised system of national accounts: SNA 1993. New York.
United Nations Development Programme. (2004). Sources for democratic governance
indicators. Oslo.
Vandenabeele, W. (2008). Government calling: Public service motivation as an element
in selecting government as an employer of choice. Public Administration, 86(4),
10891105.
Van de Walle, S. (2006). The state of the worlds bureaucracies. Journal of Comparative
Policy Analysis: Research and Practice, 8(4), 437448.
Van Dooren, W. (2009). A politico-administrative agenda for progress in social measure-
ment: Reforming the calculation of governments contribution to GDP. Journal of
Comparative Policy Analysis: Research and Practice, 11(3), 309326.
Van Dooren, W., Bouckaert, G., & Halligan, J. (2010). Performance management in the
public sector. London: Routledge.
Verhoest, K., Bouckaert, G., & Peters, B.G. (2007). Janus-faced reorganization: Specializa-
tion and coordination in four OECD countries in the period 19802005. International
Review of Administrative Sciences, 73(3), 325.
508 PPMR / March 2012

Vigoda, E. (2002). From responsiveness to collaboration: Governance, citizens, and

the next generation of public administration. Public Administration Review, 62(5),
527540.

Wouter Van Dooren is an assistant professor of public administration in the

Department of Political Science, University of Antwerp, Belgium. He researches
performance measurement and management and recently coauthored Performance
Management in the Public Sector (Routledge, 2010).

Chiara De Caluw is a researcher in the Department of Political Science, Uni-

versity of Antwerp, Belgium. She is working on a Ph.D. dissertation concerning
performance by public administration.

Zsuzsanna Lonti is the head of the statistics and indicators unit in the Budgeting
and Public Expenditures Division of the OECD Governance Directorate, Paris.
She leads the Government at a Glance project.

View publication stats