Abstract

This article explores distinctive research patterns of public administration and accounting disciplines concerning public sector performance measurement (PSPM). Our review shows that accounting researchers from Europe investigate reasons for limited PM use and factors explaining a rational or symbolic PM use, inspired by organization theory and institutional theory and conducting case/eld studies. Public administration researchers from Europe and the USA prefer to study PM design and PM impact respectively, mainly using surveys in combination with various theories, like political theory. Public administration research from the USA examines the types of performance indicators in PM systems and contingent factors for PM design. Public administration research from Europe shows an interest in evaluating public sector reforms like Best Value and explaining learning processes for improvement. We argue that PSPM research could benet from interdisciplinary efforts and intensied mutual communication between public administration and accounting.

DISTINCTIVE RESEARCH PATTERNS ON PUBLIC SECTOR PERFORMANCE MEASUREMENT OF PUBLIC ADMINISTRATION AND ACCOUNTING DISCIPLINES
ge Johnsen and G. Jan van Helden, A Jarmo Vakkuri
G. Jan van Helden Faculty of Economics and Business, PO Box 800, 9700 AV Groningen, The Netherlands Tel: 31 503633673, Fax: 31 503638252 E-mail: g.j.van.helden@rug.nl ge Johnsen A Faculty of Social Sciences, Oslo University College, P.O. Box 4 St Olavs plass, NO-0130 Oslo, Norway E-mail: aage.johnsen@sam.hio.no Jarmo Vakkuri Department of Economics and Accounting, University of Tampere, FI-33014, Finland E-mail: jarmo.vakkuri@uta.

Key words
Accounting, performance measurement, public administration, research patterns

Vol. 10 Issue 5 2008 641651 Public Management Review ISSN 1471-9037 print/ISSN 1471-9045 online 2008 Taylor & Francis http://www.tandf.co.uk/journals DOI: 10.1080/14719030802264366

642

Public Management Review

INTRODUCTION Performance measurement has been a core issue in the public sector for many decades, but its importance has been reinforced by the more recent New Public Management and Reinventing Government movements with their emphasis on planning and control of outputs and outcomes. Several disciplines (as well as professions) are relevant in studies of public sector performance measurement (PSPM), notably economics, public administration, accounting and sociology. We focus on two research disciplines, one rooted in public administration and the other in (public sector management) accounting. These two academic disciplines do not necessarily overlap, indicating an absence of mutual communication. This impression inspired us to conduct a literature review on PSPM research. It goes beyond the scope of this short article to investigate comprehensively differences and similarities in PSPM research between public administration and accounting. Therefore, our analysis focuses on distinctive research patterns of these disciplines in terms of content as well as use of theories and methods. Our literature review includes articles on PSPM that were published in the 20005 volumes of four public administration journals and four accounting journals. After a justication of the research design, the ndings of our review are analysed and discussed. RESEARCH DESIGN Our review focuses on public sector performance measurement. In this article the public sector includes various tiers of government (i.e. local, central and intermediate), governmental agencies (such as the police force and schools) and also non-governmental public sector organizations, mainly from the health care sector. Performance measurement (PM) is understood as an instrument for indicating efciency and effectiveness and due to the public sector context also equity. PM is primarily used for increasing decision-making rationality in organizations. However, ritualistic uses of PM information cannot be excluded. PM not only implies measurement and analysis, but also reporting the resulting information to relevant administrative and political bodies. PM serves various functions, particularly accountability, policy (re)design, as well as planning and control (Mayston 1985; Likierman 1993; de Bruijn 2002; Johnsen 2005). In this sense performance measurement largely overlaps with performance management. It is both measurement of performance and management with measures of performance. We selected four public administration (PA) journals Journal of Public Administration Research and Theory (JPART), Public Administration Review (PAR), Public Administration (PA) and Public Management Review (PMR) and four accounting (AC) journals Accounting, Organizations and Society (AOS), Journal of Management Accounting Research

van Helden et al.: Public sector performance measurement

643

(JMAR), Financial Accountability and Management (FAM) and Management Accounting Research (MAR). Table 1 shows the origins of the journals (US or European) and the disciplines (PA or AC). All four PA journals rank among the highest in their eld, according to their impact scores. AOS is one of the four international top AC journals (it also has a substantial impact score). Given that the American top journals in AC Accounting Review, Journal of Accounting Research and Journal of Accounting and Economics are strongly nancial accounting oriented and publish hardly any articles about the public sector, we included two other AC journals JMAR and MAR with very good reputations (Lowe and Locke 2005). Finally, FAM was selected as an AC journal because it is a well-respected international niche journal on public sector management and accounting. Our analysis is based on a set of PSPM articles, selected from the 20005 volumes of the journals. Our initial set of articles was seventy-nine, but the analysis focuses on sixty-eight articles.1 First, because PSPM research from US authors published in AC journals is underrepresented, we had to disregard this group of authors. Second, also articles coming from authors outside Europe or the USA mainly from Australia and New Zealand were excluded. The analysis thus distinguishes three groups of authors: (1) (2) (3) Accounting researchers from Europe. Public administration researchers from Europe. Public administration researchers from the USA.

Research methods employed in the articles were classied as: literature review; analytic; survey; documentary analysis; case/eld study; reection; and others. This categorization was inspired by earlier literature reviews by Shields (1997) and van Helden (2005). Although these categories refer to different research dimensions, such as empirical research (documentary analysis, case/eld studies and surveys) and theoretical research (analytic and literature reviews), they can be derived from the selected articles. Moreover, whenever two categories are apparent in one article for instance a literature review and a survey this is accounted for (see counting procedures later). Based on prior reviews (Shields 1997; van Helden 2005) we identied the following categories for theories: economics; organization theory; neo-institutional sociology; political theory; PM theory; other theories; and no theory. PM theory can be regarded

Table 1: Journal selection according to origins and disciplines of journal Public administration US-originated Europe-originated PAR and JPART PA and PMR Accounting AOS (partly) and JMAR AOS (partly), FAM and MAR

644

Public Management Review

as a second level (more applied) theoretical stance, whereas the other theories are distinctive disciplines. We distinguished PM stages and PM topics. PM stages relate to the various stages within a PM life cycle; particularly design, implementation, use and impact. Similar distinctions are made by Likierman (1993) and Johnsen (2005), and in empirical studies of PM some stages of this life cycle are recognizable (Cavalluzzo and Ittner 2004; van Dooren 2005). The pilot stage of our review revealed that this categorization of PM stages although sensible incompletely covered the research topics. We decided to include the PM topic as well as the PM stage. Accordingly, each PM stage involves various research topics. For the research topics a pre-determined set of categorizations could not be developed. Only a retrospective categorization was possible. Our counting procedure in dealing with articles that use a combination of two or more methods (or theories) was as follows. If two methods (theories) are addressed, each counted for one half, and if three methods (theories) are addressed, each counted for one third and so on. ANALYSIS We discuss the nature of public sector performance measurement research by identifying the main themes addressed in our sample of articles. Although articles from both AC and PA researchers display an interest in all the PM stages that is, design, implementation, use and impact each category of authors has its own favourite PM stage: AC researchers from Europe are strongly interested in PM use, European PA researchers in PM impacts and PA researchers from the USA in PM design. Focusing on these distinctive research interests, we will review the current body of knowledge in PSPM research.2 US public administration researchers on PM design Research on PM design is prominent among American PA researchers: 43 per cent of the articles address PM design, whereas the three other PM stages implementation, use and impact only receive attention from about 20 per cent of the articles each. These researchers address two main questions, that is, the types of performance indicators (including the way they are specied) and contingent factors that inuence PM design. Edwards and Thomas (2005) describe the development and operation of a performance measurement system, called the Atlanta Dashboard, in comparison with other such systems in US cities. Although the Dashboard loosely follows the ideas of the Balanced Scorecard, it deliberately emphasizes outcomes, because of their relevance, and omits internal process perspectives. In contrast, Kelly (2005) criticizes the imitation of private sector ideas as the Balanced Scorecard, because value creation in the

van Helden et al.: Public sector performance measurement

645

public sector needs to be balanced among appropriate, competing values acknowledging the multiple levels of managerial accountability to citizens, elected ofcials, public employees and professional standards. Kelly and Swindell (2002) examine correlations from a sample of cities and municipalities in the USA with standardized measures and citizen survey results for four service areas. Their study did not nd a systematic correlation between internal (administrative) and external (citizen satisfaction) performance measures. Some articles intend to nd answers to rather pragmatic questions about performance management. Rubenstein et al. (2003) advocate, for instance, the use of performance measures adjusted for the effect of non-controllable factors and the average value of controllable factors (policy variables), instead of using raw performance measures. Folz (2004) recommends that best performing benchmarking partners have to be selected from organizations having a certain target service level. Another group of articles is concerned with survey research that aims to nd relationships between contingent variables and PM design. In their nation-wide survey of US counties Wang and Berman (2001) found, for instance, that central management involvement signicantly determines deployment of output measures, while mission orientation as well as external support from politicians and citizens signicantly inuences deployment of outcome measures. Research by Julnes and Holzer (2001) reveals that PM design of organizations at various layers of US government is primarily inuenced by external requirements, resources, goal orientation, information capabilities and involvement of internal groups (see also Melkers and Willoughby 2005). As indicated before, PA authors from the USA are conducting surveys for their contingency studies, whereas studies on the design of Performance Indicator (PI) systems are mostly based on case studies. Some articles have a reective nature. Both PM theory and political theory are well represented in PA papers by American authors. European AC researchers on PM use About 50 per cent of the articles written by European AC researchers concerns PM use. Three aspects are important to this research. The rst group of articles examines why politicians and public managers use performance information only to a limited extent. Ter Bogt (2004) observes that aldermen of Dutch municipalities prefer informal and verbal communication to budgets and reports with output-oriented performance information (see also ter Bogt and van Helden 2000). This author also shows that politicians evaluate top managers based on a broad set of criteria in which quantitative performance information only plays a minor role (ter Bogt 2003). A second group of articles investigates the extent to which PM use is rational or symbolic. Brignall and Modell (2000) argue that conicts between powerful stakeholders with diverging interests induce a decoupling of performance information at various levels

646

Public Management Review

of the organizational hierarchy, which makes a symbolic PM use likely. Only when one of the stakeholders is dominant in comparison with the others, a cascading down of performance information serving the interests of the dominant stakeholder throughout the hierarchy will lead to a rational PM use. Some studies point at specic forms of PM use also challenging rationality (compare Modell 2001). In their analysis of the UK Best Value project Bowerman et al. (2000) introduce the idea of defensive benchmarking, meaning that public sector organizations use benchmarking information to avoid critique from external bodies rather than as an incentive for improvement. Llewellyn and Northcott (2005) examine social, institutional and political processes through which UK hospitals move to the level of average performance. A third group of authors investigates contingencies of PM use. Van Dooren (2005) conducts a survey on design and use of sections within the Ministry of the Flemish Community in Belgium. This study shows that PM use is positively inuenced by the ability to identify PIs, organizational size (e.g. larger organizations adopt more PIs than smaller organizations), decision-making authority and goal orientation. Budding (2004) nds that, opposite to insights from other contingency studies, external uncertainty does not inuence PM use in Dutch municipalities, whereas factors like a results-oriented organizational climate, a supportive attitude of superiors and creation of game spirit have an inuence. European AC researchers most frequently apply case/eld studies as methods of data collection. Their research is mainly based on organization theories (such as strategic management theory and management control theory) and neo-institutional theory. European PA researchers on PM impact Within this last group of articles 40 per cent address PM impact, which can be understood as the effect of PM on the accomplishment of goals, at the organizational, programme or sectoral level, but also including functional (e.g. learning) or dysfunctional (e.g. manipulation) effects of PM. As elucidated below, some articles aim to evaluate reform programmes of government, while others endeavour to enrich understanding of the impact of PM on organizational learning. Martin and Hartley (2000) evaluate Best Value (BV) adoption in English and Welsh local government at an early stage of development. They report a strong support for BV, although pilot authorities expressed a higher level of condence in and understanding of BV than non-pilot authorities. Using theories on accountability and based on a wide range of data sources, Boyne et al. (2002) review the BV project within Welsh municipalities. Their assessment was more critical, and they conclude that in general the BV pilot performance plans did not provide useful information to internal and external stakeholders. In a case study of partnerships within the NHS, Alcock (2004) hints at current performance management regimes using targets and linking performance indicators to ofcial programmes and agencies, which lead to information overload and dysfunctional consequences.

van Helden et al.: Public sector performance measurement

647

Hartley and Allison (2002) present an analysis of a network consisting of local authorities in the UK that participated in a Better Value Development Programme. Their study claries that participants valued the exchange and transfer of tacit knowledge, through sharing and comparing experiences. Sanderson (2002) reects on the impacts of NPM like reforms, in which performance measurement plays a main role. This article concludes that the evaluation capacity of organizations (including capacities for reection, dialogue and research) should be improved. PA researchers from Europe relatively frequently undertake surveys and case/eld studies. They often use applied theories, such as theories on change, accountability or benchmarking. Figure 1 links the most important PM stages for the groups of authors, as indicated above, to the corresponding PM topics and to the use of theories and research methods. DISCUSSION Our analysis demonstrates that PA and AC can be regarded to some extent as two different academic tribes involving distinct research objectives and identities (Becher and Trowler 2001). This can account for a certain specialization in studying a common domain, and contribute to knowledge accumulation on PSPM. According to our analysis a specic setting of specialization has developed in PSPM research: US PA researchers concentrate on issues of PM design, European AC researchers on PM use and European PA researchers on PM impacts. The different foci may benet PSPM research in some regard. However, a lack of mutual communication between the disciplines their fundamental research orientations are fairly similar can also be a hindering factor. Researchers in one discipline, for instance, may suggest novel themes or research approaches, the novelty of which would have been questioned if those researchers had been aware of relevant knowledge production in the other discipline. An indication for this lack of communication can be derived from our review. We searched for those authors who publish two or more articles, separately or as co-author with others. The outcome of this search reveals that authors mainly stick to their discipline this holds for fteen from eighteen authors which corroborates our expectation of non-overlapping academic circles. A lack of publications by the same authors in journals from different disciplines is, however, only a weak indication of the lack of communication between those disciplines. This issue requires further analysis of citations within and across the PA and AC disciplines. There are some examples that clarify the risks of a lack of communication between disciplines that study the same phenomenon, PSPM. Cavalluzzo and Ittner (2004) published a survey study in an accounting journal, which does not cite relevant preceding research in PA concerning survey type investigations (Julnes and Holzer 2001; Wang and Berman 2001). Furthermore, PA research could benet from

648

Public Management Review

Figure 1: Distinctive research patterns of groups of authors

knowledge about case-based accounting research using institutional theories (ter Bogt 2003; Modell 2004). Quite a number of articles in our review show that European researchers are often critical of some tendencies in contemporary public management reforms

van Helden et al.: Public sector performance measurement

649

(for example: ter Bogt and van Helden 2000; Bowerman et al. 2001; Modell 2001; Boyne et al. 2002; Sanderson 2002), whereas much research in the North-American tradition is more normative (Rubenstein et al. 2003; Folz 2004). Such research is, generally speaking, more supportive of public management reforms. This would indicate that as regards to attitudes of PSPM researchers on public management reforms the tribal division would also be associated with continents, not only with the disciplines of PA and AC. There is also little interest in PSPM research within the AC discipline in North America. The major interest in PSPM comes from PA. This underpins a point of concern that American AC researchers seem to ignore the public sector. The results of our review may be bounded by the journal selection and by our omission of analysis of books. It has been difcult to nd good international AC journals that publish public sector research. Our review also includes a very small number of American AC academics. We envisage several directions for future research. It would be of importance to conduct a more in-depth study of the research themes on PSPM. This assessment could subsequently relate to similarities and differences between disciplines and different geographical areas, for instance of the European or North-American perspectives. Despite these new directions for research, our review provides an overall picture of the most distinctive types of research on PSPM conducted by both AC and PA researchers from various continents. ACKNOWLEDGEMENT The authors acknowledge the constructive comments received when earlier versions of this article were presented at the Second Transatlantic Dialogue: A Performing Public Sector, Leuven, 13 June 2006, and at the Fourth EIASM International Conference on Accounting, Auditing and Management in Public Sector Reforms, Siena, 79 September 2006. The authors are indebted to Bob Scapens and two anonymous reviewers for their helpful comments on earlier versions of this article. NOTES
1 The list of articles in the review is available on request from the corresponding author. 2 Classication results of all articles in the review are available on request from the corresponding author.

REFERENCES
Alcock, P. (2004) Targets, Indicators and Milestones, Public Management Review, 6: 2 pp2117. Becher, T. and Trowler, P. A. (2001) Academic Tribes and Territories: Intellectual Enquiry and the Cultures of Disciplines (2nd edn), Buckingham: Open University Press. Bowerman, M., Ball, A. and Francis, G. (2001) Benchmarking as a Tool for the Modernisation of Local Government, Financial Accountability and Management, 17: 4 pp3219.

650

Public Management Review

Boyne, G., Gould-Williams, J., Law, J. and Walker, R. (2002) Plans, Performance Information and Accountability: The Case of Best Value, Public Administration. 80: 4 pp691710. Brignall, S. and Modell, S. (2000) An Institutional Perspective on Performance Measurement and Management in the New Public Sector, Management Accounting Research. 11: 3 pp281306. Budding, T. (2004) Accountability, Environmental Uncertainty and Government Performance: Evidence from Dutch Municipalities, Management Accounting Research. 15: 3 pp285304. Cavalluzzo, K. S. and Ittner, C. D. (2004) Implementing Performance Measurement Innovations: Evidence from Government, Accounting Organizations and Society. 29: 3/4 pp24367. De Bruijn, H. (2002) Managing Performance in the Public Sector, London: Routledge. Edwards, D. and Clayton Thomas, J. (2005) Developing a Municipal Performance-Measurement System: Reections on the Atlanta Dashboard, Public Administration Review. 65: 3 pp36976. Folz, D. F. (2004) Service Quality and Benchmarking the Performance of Municipal Services, Public Administration Review. 64: 2 pp20920. Hartley, J. and Allison, M. (2002) Good, Better, Best? Inter-Organizational Learning in a Network of Local Authorities, Public Management Review. 4: 1 pp10118. . (2005) What Does 25 Years of Experience Tell Us about the State of Performance Measurement in Johnsen, A Public Policy and Management?, Public Money and Management. 25: 1 pp1522. Julnes, P. deLancer, and Holzer, M. (2001) Promoting the Utilization of Performance Measures in Public Organizations: An Empirical Study of Factors Affecting Adoption and Implementation, Public Administration Review. 61: 6 pp693708. Kelly, J. M. (2005) The Dilemma of the Unsatised Customer in a Market Model of PA, Public Administration Review. 65: 1 pp7684. Kelly, J. M. and Swindell, D. (2002) A Multiple-Indicator Approach to Municipal Service Evaluation Correlating Performance Measurement and Citizen Satisfaction across Jurisdictions, Public Administration Review. 62: 5 pp61021. Likierman, A. (1993) Performance Indicators: 20 Early Lessons from Managerial Use, Public Money and Management. 13: 4 pp1522. Llewellyn, S. and Northcott, D. (2005) The Average Hospital, Accounting, Organizations and Society. 30: 6 pp55583. Lowe, A. and Locke, J. (2005) Perceptions of Journal Quality and Research Paradigm: Results of a Web-Based Survey of British Accounting Academics, Accounting, Organizations and Society. 30: 1 pp8198. Martin, S. and Hartley, J. (2000) Best Value for All: An Empirical Analysis of Local Governments Capacity to Implement Best Value Principles, Public Management Review. 2: 1 pp4356. Mayston, D. J. (1985) Non-Prot Performance Indicators in the Public Sector, Financial Accountability and Management. 1: 1 pp5174. Melkers, J. and Willoughby, K. (2005) Models of Performance-Measurement Use in Local Governments: Understanding Budgeting Communication and Lasting Effects, Public Administration Review. 65: 2 pp180 90. Modell, S. (2001) Performance Measurement and Institutional Processes: A Study of Managerial Responses to Public Sector Reform, Management Accounting Research. 12: 4 pp43764. (2004) Performance Measurement Myths in the Public Sector: A Research Note, Financial Accountability and Management. 20: 1 pp3955. Rubenstein, R., Schwartz, A. E. and Stiefel, L. (2003) Better than Raw: A Guide to Measuring Organizational Performance with Adjusted Performance Measures, Public Administration Review. 63: 5 pp60715. Sanderson, I. (2002) Performance Management, Evaluation and Learning in Modern Local Government, Public Administration. 79: 2 pp297313. Shields, M. D. (1997) Research in Management Accounting by North-Americans in the 1990s, Journal of Management Accounting Research. 9 pp361.

van Helden et al.: Public sector performance measurement

651

Ter Bogt, H. J. (2003) Performance Evaluation Styles in Governmental Organizations: How Do Professional Managers Facilitate Politicians Work?, Management Accounting Research. 14: 4 pp31132. (2004) Politicians in Search of Performance Information? Survey Research on Dutch Aldermens Use of Performance Information, Financial Accountability and Management. 20: 3 pp22152. Ter Bogt, H. J. and van Helden, G. J. (2000) Accounting Change in Dutch Government: Exploring the Gaps Between Expectations and Realizations, Management Accounting Research. 11: 2 pp26379. Van Dooren, W. (2005) What Makes Organisations Measure, Financial Accountability and Management. 21: 3 pp36383. Van Helden, G. J. (2005) Researching Public Sector Transformation: The Role of Management Accounting, Financial Accountability and Management. 21: 1 pp99133. Wang, X. and Berman, E. (2001) Hypotheses about Performance Measurement in Counties: Findings from a Survey, Journal of Public Administration Research and Theory. 11: 3 pp40328.