CASE STUDY: HUMAN RESOURCES ASSIGNMENT INSTRUCTIONS OVERVIEW In this Case Study, you will apply the Statesmanship model discussed in Module 1: Week 1 to a real, specific public administration context

© 2012 M.E. Sharpe, Inc. All rights reserved.

1530-9576/2012 $9.50 + 0.00.

DOI 10.2753/PMR1530-9576350306 489 HOw t O MEASuRE PublIc ADMINIStRA tION PERfORMANcE A conceptual Model with Applications for budgeting, Human Resources Management, and Open Government wOutER V AN DOOREN cHIARA DE cAluwE University of Antwerp ZSuZSANNA lONtI Organization for Economic Cooperation and Development AbstrA ct: The economic crisis provides some insights on the role of measurement systems. As shown by the ongoing discussion of credit rating agencies by political actors and in the news media, measurement is not a neutral device, but an active agent in societal processes. Comparative measurements of international public administration are as frail, technocratic, and overly aggregated as bond ratings, but nonetheless are increasingly used by journalists, aid organizations, foreign investors, and, indeed, rating agencies to hold governments accountable.

Better measurement is needed in public administration performance. This article builds on study reports from the OECD’s Government at a Glance project to address the issue of how to measure public administration performance. The fields of budgeting, human resources management, and open government illustrate both the potential and the challenge of such measurements.

Keywords: budgeting, human resources management, open government, public administration performance the economic crisis provides some insights on the role of measurement systems in society. Most notably, the role of credit rating agencies has been discussed by political actors and in the news media. 1 this discussion shows that measurement is not a neutral device, but an active agent in societal processes. As in the case of bond ratings, comparative measurement of international public administration seems frail, technocratic, and overly aggregated (Arndt, 2007; Arndt & Oman, 2006; Van de walle, 2006). Nonetheless, these indicators are increasingly being used by journalists, aid organizations, foreign investors, and, indeed, rating agen- cies for holding governments accountable. Hence, better measurement of public administration performance is needed. the discussion that follows builds on study 490 PPMR / March 2012 reports from the Organization for Economic cooperation and Development’s (OEcD’s) Government at a Glance project to address the issue of how to measure public administration performance. the fields of budgeting, human resources management, and open government are used to illustrate both the potential and the challenge of measuring public administration performance. the current economic crisis is also a crisis of measurement. 2 In a 2005 cartoon by Randy Glasbergen, a banker offers a couple an interest-only mortgage, bal- loon mortgage, reverse mortgage, upside-down mortgage, inside-out mortgage, loop-the-loop mortgage, and a spinning double-Axel mortgage with a triple-lutz (Mortgage cartoon Glasbergen, 2005). the cartoon appeared two years ahead of the collapse of Northern Rock in the uK and three years before the height of the crisis in 2008. However, Glasbergen and other, more systematic observers, such as Roubini and taleb, were well aware of problems in the housing mortgage market.

Measurement systems of both the public and private sectors failed to detect this undercurrent. the economic measurement models used by the national banks and the planning bureaus did not capture the trend, just as the credit agencies failed to adequately gauge the real value of the assets under scrutiny. today, national creditworthiness ratings are being discussed in response to notable downgrades of several eurozone countries along with Standard & Poor’s u.S. credit rating downgrade. Some critiques target the failure of measurement systems to reflect reality. An additional step is to seek the causes of the crisis in the measurement systems. the AAA ratings for complex collateralized debt obligations (cDOs), for instance, are blamed for having been at least a catalyst for the crisis. crotty (2009) asserts that the recent global financial boom and crisis might not have occurred if perverse incentives had not induced credit-rating agencies to give absurdly high ratings to illiquid, nontransparent, structured financial products. Interestingly, the same mechanism of bad ratings by the agencies before the crisis and overly strict ratings after the crisis has also been documented for the East Asian financial meltdown in 1997 (ferri, liu, & Stiglitz, 1999). Moreover, rating agencies’ power and lack of accountability to the public was also discussed before the crisis (Kerwer, 2005). for present purposes, there is no need for a thorough analysis of the ins and outs of the economic crisis. Yet the crisis does remind us that measurement systems are not neutral controls for policymakers in the cockpit of society. Measurement systems are themselves agents, and influence behavior through their workings.

when measurement systems are used to hold organizations to account, it should be kept in mind that they influence the behavior of account givers as well as ac- count holders. therefore, both a critical attitude toward existing indicators as well as continuous efforts to improve measurement seem warranted. this is also true concerning performance measurement for public administration. Existing indicators on public administration performance—which usually go Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 491 under the banner of “governance indicators”—have been subject to some criticism.

this is true even at the national level, but far more so for the purpose of interna- tional comparisons (Arndt, 2007; Van de walle, 2006). One of the main critiques of governance rankings targets their validity—do they measure what they claim to measure? Rankings such as the world bank governance indicators comprise a broad and unintelligible compilation of opinion surveys, risk assessments, and sector indicators (Arndt & Oman, 2006). typically, nonperceptual indicators on the system’s core functions are lacking. As a result, there is a risk of constructing an image of a country’s performance based on indicators that do not accurately reflect its performance. If these performance indicators are used for accountabil- ity purposes in a subsequent phase of governance, the consequences of invalid measurement become real. the present article does not offer further critique but, rather, proposes possible measurement improvements. the text is based on several research reports commis- sioned by the OEcD’s governance directorate, which is currently undertaking the large project called Government at a Glance. the focus is on doing measurement, rather than on further elaborating the functions and dysfunctions of measurement (for the latter, see, e.g., Hood, 2006; Radin, 2006). No subjective stance is taken on the utility or potentially negative effects of using performance measures. for public administration more than for fields such as education or health, an additional difficulty for measurement is the conceptual confusion on what public administration performance actually means. this article begins by exploring the conceptual foundation for indicators of public administration performance.

based on this foundation, it discusses possible performance indicators in three areas of public administration: budgeting, human resources management, and open government.

the conceptual Foundation of Public Administration Performance As suggested above, it is first necessary to sort out a number of definitional is- sues. the primary task is to define performance. there seems to be a common mainstream understanding of performance, but this definition is process-oriented and nonsubstantial. In order to measure public administration performance, a substantial definition of performance is required.

the MAinstreAM deFinition oF PerForMAnce Performance is usually defined in terms of the outcomes and outputs that follow from a public production process (Hatry, 1999). this model seems to provide the dominant vocabulary used by public administration researchers and practitioners when discussing performance. Although some terminological issues remain, and although some analysts will primarily emphasize the importance of contextual 492 PPMR / March 2012 factors, the main building blocks have become widely accepted foci of public administration theory and practice.

Outcomes are the result of activities that convert inputs to outputs. the transfor - mation of inputs, such as financial and human resources, to activities is mediated by the structure of government, cultural predispositions, and institutional and managerial arrangements. Outputs are the goods and services that public orga- nizations supply in response to demand. Outcomes are the consumption of the goods and services (intermediate outcomes) as well as the effects this consumption entails (final outcomes). the criterion for assessing outcome is added value (Moore, 1995). the added value of a private firm is the result of the aggregation of individual decisions to consume a service or good at a given price. A firm’s profit can be conceptualized as its outcome in society, since it is the sum of the values that individuals attach to a good or service, minus the costs of production. free-rider problems 3 and the positive and negative externalities of consumption, 4 however, mean that one cannot rely on individual consumption decisions for all goods and services (Musgrave, 1959). for public services, although the criterion of added value remains intact, the notion of public value replaces private value. In the absence of monetary profits, it is much more difficult for public organizations to assess outcomes. subst Antive APPro Aches to PerForMAnce A problem with the mainstream definition of performance is its nonsubstantial na- ture. the definition of performance as outputs and outcomes of public services does not tell us what these outputs and outcomes should be. Different ideologies will have different views about public services, such as whether or not to redistribute outputs, how many services to provide, and whether or not regulation is needed.

As such, the mainstream definition is a purely analytical concept. Different actors can define performance differently without invalidating its conceptual definition in terms of output and outcome (Van Dooren, bouckaert, & Halligan, 2010). Operationalizing public administration performance calls for a “thick” substan- tive approach to performance that includes a variety of public values. the tendency of measurement systems to develop tunnel vision, similar to the narrow focus of credit-rating agencies, could be countered by maintaining an open conceptual view on what defines public administration performance. In a seminal article on the New Public Management (NPM), Hood (1991) proposes a classification with three clusters of public values. He makes the point that NPM reforms almost exclu - sively stress public values of one type, often to the detriment of other values. the public-values literature expanded further in the following decade. Jorgensen and bozeman (2007), for instance, developed an inventory of more than 70 different values in the public-values universe. for reasons of parsimony, Hood’s three broad value groups are utilized, since they also seem to reflect general dimensions of Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 493 performance (Van Dooren, bouckaert, & Halligan, 2010). three broad but distinct substantial definitions of performance might be: • Product performance reflects success in matching resources to defined tasks. Government has to operate in an economical, efficient, and effective way. Product performance reflects what Piotrowski (2010) calls mission-based values.

• Procedural performance reflects success in keeping government fair and honest. Government has to pursue honesty, fairness, and mutuality through the preven- tion of distortion, inequity, bias, and abuse of office. Procedural performance is not mission-based (Piotrowski, 2010).

• Regime performance reflects success in keeping the public sector robust and resilient. Government has to operate even in adverse “worst-case” conditions and to adapt rapidly in response to crisis and change. Regime performance is another form of non-mission-based performance.

It is generally easier to measure product performance than process or regime performance (see Table 1). for production, resources are allocated to defined tasks and indicators are a means to assess goal attainment. this is not to say that measurement is unproblematic, but at least conceptually it is unambiguous.

Processes in production cycles are usually repeated, which allows for learning in measurement. Measuring levels of fairness and honesty becomes substantially more complicated. Such measures will predominantly be about the absence of fairness and honesty, measuring fraud, corruption, favoritism, and so forth. typi- cally, these activities are “under the radar.” they are, however, recurring events, which makes the development of indicators somewhat easier. Indicators for ro- bustness and resilience are even more complicated, since failures and worst-case conditions that really put a strain on robustness are uncommon. As an alternative to outcome measures, measures of capacity could be developed, but they are not performance indicators (Hall, 2008). the selected public administration dimensions discussed below touch upon two of the three approaches to public values. Human resources management and bud- t able 1. Approaches to Performance and consequences for development of indicators Product Procedure Regime How can performance be observed?Sufficiency (provision of services) (Mostly) deficiency (absence of fairness and honesty) Deficiency (system failure) what is the incidence of these observations? (Mostly) repeated (Mostly) repeated One-time Developing indicators feasible DifficultVery difficult Source: based on Van Dooren, bouckaert, & Halligan, 2010. 494 PPMR / March 2012 geting are typically cyclical, recurring activities. A human resources management department has to hire, evaluate, motivate, and terminate staff on a continuous basis.

timely and accurate budgets have to be developed by budget departments every year. Human resources management and budgeting activities have a good deal of product performance, and hence it can be expected that measuring performance on these dimensions is viable. the main “product” of a budget department is the budget. for a human resources department, hires are an important product. Open government, on the contrary, has a more procedural, nonmission focus, and hence more difficulties can be expected in developing indicators. for regime performance, it is not at all clear whether it is possible to develop performance indicators; and in consequence this dimension is left out of the scope of the present ar\ ticle. Public Administration Performance the discussion in this section defines public administration performance. the development of performance indicators for public administration requires an understanding of two defining features of the nature of public administration. first, public administration is about enabling rather than delivering. Public administration almost never provides final goods and services. Public administra- tion, however, is a precondition for the successful operation of other government departments. It is government for government rather than government for the citizens. that takes nothing away from its importance. Public service delivery is a chain of inputs and outputs. clearly, public administration arrangements are to be found earlier in the chain. Schools need to be staffed and financed before they can provide teaching. the chain of impact is schematically represented in figure 1. Public admin- istration processes, including typical horizontal functions such as financing and human resources management, are a precondition for functions performed by line departments and agencies. Public administration outputs are inputs for the functional processes. for instance, ethics training sessions are an output for an ethics division but an input for a social security department. the outcome of the training sessions is better awareness of ethics issues within the administration of social security. In the same way, the number of administered allowances is an output of the social security administration that is an input for societal processes.

If the allowance protects people from poverty, then it is a policy outcome. the step from a public administration process such as ethics to a policy outcome such as poverty in society is a rather big leap. It would be hard to demonstrate how integrity training has an impact on poverty statistics. However, by taking into consideration the intermediate processes in the chain, it might be possible to bridge the gap. Public administration processes should, in the first place, improve the quality of public administration and enable others to govern society in a more Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 495 direct way. Public administration performance should reflect this position.

A second typical feature of public administration is its crosscutting nature.

Precisely because it is an enabler, public administration has an impact on all other policy sectors. this also explains why it is so difficult to implement government- wide administrative policies (Verhoest, bouckaert, & Peters, 2007). Often, such policies are perceived to run counter to the vested interests and practices of the policy sectors. for measurement, the crosscutting nature of public administration complicates data collection and standardization. Measuring the enabling and crosscutting roles of public administration has two consequences. first, the outputs of public administration processes are the inputs for substantive processes in line departments and agencies. In order to identify output, it is necessary to ask what products and services public administration processes deliver to line agencies. Second, the outcomes of public administration primarily have to reflect its ability to facilitate policy sectors. to identify outcome, it is necessary to ask whether public administration processes succeed in enabling performance in other sectors.

MeAsuring Public AdMinistrA tion PerForMAnce:

b udgeting, huMAn resources MAn AgeMent, And oPen governMent the conceptual approach described above will now be applied to three important dimensions of public administration: budgeting, human resources management, and open government. Maturity of measurement differs for the three dimensions.

Measurement in budgeting is quite well developed, which allows for relatively robust and comparative indicators. Human resources management is also a dimen- sion in which several measurement efforts exist, but these are not yet developed for international comparative purposes. Open government is, for measurement Figure 1. A chained Approach to outcome 496 PPMR / March 2012 concerns, a field of experimentation. Some prospects for indicator development are discussed below. the implications for their use are considered in the discus- sion and also in the conclusions.

Budgeting A budget is defined as a comprehensive statement of government financial plans, including expenditures, revenues, deficit or surplus, and debt. the budget is the government’s main economic policy document, indicating how it plans to use public resources to meet policy goals (OEcD, 2006). budgeting activities follow a cycle of events or stages in making decisions about the budget, and implementing and assessing those decisions. the cycle usually has four stages: formulation, ap- proval, execution, and audit. what are the outcomes of these budgeting activities?

what constitutes successful and unsuccessful budgeting? budgeting has multiple objectives (Gray, Jenkins, & Segsworth, 2001; Sterck, 2007). three conventional budget functions are authorization (legal function), al- location (policy function), and management (management function). by approving the budget, the legislature authorizes the executive to spend money and to collect taxes. the question, then, is whether budgets succeed in informing executives about the decisions they have to make. More generally, the budget has to inform the general public about government’s spending and revenue collection inten- tions. A second objective of a budget is to allocate resources to policy fields while maintaining fiscal balance at the macrolevel. the question is whether the budget is successful in allocating the right amount of resources to public services and, at the same time, is successful in ensuring a structural balance between revenues and expenditures. A third function is management. Managers have to use budgets to control their organization. Performance indicators for budgeting could first look at the variance between projected (intended) and actual revenues and the variance between projected and actual expenditures (colonna & Puma, 2003). these measures give an indication of the quality of the estimates. If the variance becomes too high, the impact of the budget as a tool for policy, management, and legislative control erodes. At the microlevel, managers may no longer be able to plan spending and investment.

Moreover, overestimation of expenditures may lead to end-of-the-year spending— also known as “December fever” (Douglas & franklin, 2006; tarschys, 2002).

Managers typically spend all of their appropriated budgets, partly because they fear sanctions in the subsequent budget round. At the macrolevel, economic and monetary policy will be based on faulty assumptions (Rubin, 1997). the Govern- ment Performance Project (Maxwell School of citizenship and Public Affairs, 2002) uses the following measures; a. Revenue estimation accuracy = (actual revenue – estimated revenue)/estimated revenue Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 497 b. Expenditure estimation accuracy = (actual expenditure – estimated expenditure)/ estimated expenditure Expenditures and revenues can be over- or underestimated. cautious budgeting would suggest a higher tolerance for the underestimation of revenues and the over - estimation of expenditures. A budget in the public sector is also a tool for legisla- tive control over the executive. the legislature’s appropriations provide authority under law to the executive to spend public funds up to a set limit and for a specified purpose. In this respect, a budget reflects not only a projection but also an intention (Kristensen, Groszyk, & böhler, 2002). Overestimation of expenditures implies that appropriated budgets are not used. this may be the result of bad budgeting, but may also point to implementation problems and thus an infringement on the legislature’s decision to spend a given amount on a particular program. A second set of budget performance indicators reflects budgeting transparency.

blondal (2003) observes that the budget is the government’s principal policy docu- ment, wherein the government’s policy objectives are reconciled and implemented in concrete terms (i.e., the authorization function). budget transparency—openness about policy intentions, formulation, and implementation—is therefore at the core of good public administration and may even influence more general political indicators, such as voter turnout (benito & bastida, 2009). Researchers at international institutions and nonprofits, as well as a\ cademics, have attempted to develop indices of budget transparency that combine several indicators (Alt & lassen, 2006; De Renzio & Masud, 2011). According to blondal (2003), transparency has three essential elements. the first is the systematic and timely release of budgetary data. this is what is traditionally associated with bud- get transparency. budget transparency is seen as an output, that is, the release of budgetary data. the second element is an effective role for the legislature. It must be able to scrutinize the budget reports and independently review them. It must be able to debate and influence budget policy and be in a position to effectively hold the government to account. the third element is an effective role for civil society, through the media and nongovernmental organizations. citizens must be in a posi- tion to influence budget policy and must be in a position to hold the government to account. In many ways, it is a role similar to that of the legislature albeit only indirectly. the second and third elements discuss the roles of the legislature and society, respectively. Active scrutiny by the legislature and society may be the result of efforts to increase budget transparency, but also of characteristics of the polity or broader trends (Sterck, 2007). Fiscal stability may be a third budgeting performance indicator. As argued above, one of the key budget functions is allocation: to match spending priori- ties with estimated revenues. Many institutional arrangements contribute to this objective—fiscal rules, economic assumptions, expenditure frameworks, fiscal-risk 498 PPMR / March 2012 assessments, and so on. therefore, a balanced budget seems to be a valid indica- tor of the outcomes of these processes. there is, however, a delicate balance to maintain in practice as well as in the indicator set. Institutions for safeguarding balanced budgets, combined with the growth of entitlements and taxing limitations, can lead to an erosion of allocative efficiency (Marlowe, 2009). Audits close the budgeting cycle. they are expert examinations of legal and financial compliance or performance (Raaum & Morgan, 2001). Performance of compliance audits might, for instance, be a decrease in the number of corrections auditors have to put forward. the results of such performance measures, however, are usually hard to interpret. Does a higher number of infringements imply that auditors detect more mistakes or that they fail to help administrations to avoid mistakes? Several supreme audit offices have attempted to measure audit outcomes (for an overview, see lonsdale, wilkins, & ling, 2011). the uK National Audit Office measures the financial savings due to auditing work relative to the money invested in the auditing agency. It identified a savings of £582 million in 2007 as a result of audit work, which exceeded the 8:1 target by some £22 million (National Audit Office, 2007). the National Audit Office, however, stresses that many benefits are not financial, and thus not measured. the u.S. Government Accountability Office (GAO) reports financial benefits of $46 billion, which is a 94:1 ratio relative to every dollar invested in audit (2007). the GAO also measures nonfinancial benefits. It measures (1) how often agencies act on GAO information to improve services to the public, (2) when information the GAO provides to the congress results in statutory or regulatory changes, and (3) when core business processes improve at agencies and government-wide management reforms are advanced by the GAO’s work. beyond this, the GAO measures the percentage of past recommendations implemented. In fiscal year 2007, 82% of the 2003 recommendations had been implemented. A final outcome indicator from the GAO report relates to client satisfaction. It counts the number of congressional hearings at which the GAO testified. the underlying argu- ment is that congressional attention is an indication of responsiveness and impact. budgeting has been subject to substantial international standardization, in particular within the European union (Eurostat, 2001; united Nations, 1993). Hence, some relatively robust international comparative-performance indicators can be derived from the budgeting system. Most Eu countries seem to follow the Eurostat guidelines, with the notable exception of Greece, which has resorted to unprecedented creative accounting. the reaction of the European union seems to be to impose more control and more intensive measurement. 5 Again, an argument pro measurement. Human Resources Management One of the most prominent public administration issues is public personnel management. the OEcD (2007) identifies four dimensions of human resources management policy: (1) workforce planning and management, (2) management of staff performance, (3) flexibility and coherence of human resources manage- Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 499 ment rules across governments, and (4) core values. the latter were discussed above in the section dealing with the guiding principles of outcome indicators.

the first three dimensions are used as a point of departure for the definition of performance indicators. (1) the objective of good workforce planning and management is “to ensure an appropriately dimensioned, appropriately structured, and representative workforce, able to meet changing labor needs in the context of changing demands and rapid developments in the wider labor market” (OEcD, 2007). A number of indicators can be derived. the workforce needs to be appropriately dimensioned and structured, with the right size and education. At first sight, size and education are input indicators.

Yet it is the adverb “appropriately” that gives this objective an outcome flavor.

w ell-functioning human resources management arrangements should bring about a workforce that is adequately sized and educated in order to ensure undisrupted and financially sustainable service delivery. Indicators that reflect the appropriate- ness can be considered outcomes. An important outcome of successful human resources management policies in a competitive labor market is government’s being an employer of choice (Vandenabeele, 2008). A 2002 OE cD study assessed this issue by asking whether governments expect problems in recruitment and retention—now or soon. the study also asked whether there are critical skill shortages (OE cD, 2002). Duration of hiring is another potential indicator of workforce management.

t he u.S. Office of Personnel Management used this indicator to assess the human resources management performance of government agencies (2007). It defines the indicator as the agency’s percentage of hires within a 45-day timeframe. It uses a hiring time-frame model to uniformly define the indicator. the u.S. GAO uses hire rate and acceptance rate to assess the impact of hu- man resources management practices. the hire rate is the ratio of the number of people hired to the number the agency planned to hire as defined in its workforce plan. the acceptance rate is the ratio of the number of applicants accepting of- fers to the number of offers made. Acceptance rate is seen as a proxy for GAO’s attractiveness as an employer and an indicator of competitiveness in bringing in new talent (GAO, 2007). Many governments strive for a workforce that reflects society (Gualmini, 2008). Evidence of the impact of diversity on performance efficiency is diffuse (Andrews, boyne, Meier, O’ toole, & walker, 2005), but representation also reflects other public values, such as social mobility of minorities. Indicators on equal opportunities reflecting the gender balance and minority representation are thus potential outcome indicators. (2) the objective of good staff performance management is “to ensure a suitably empowered and highly motivated civil service that is flexible and collaborative 500 PPMR / March 2012 and provides services in a cost efficient manner” (OEcD, 2007). this outcome suggests a second set of performance indicators. first, consider staff satisfac- tion. A satisfied workforce could be seen as an outcome for a human resources department. Sick leave/absenteeism is another example. A reasonable measure of successful human resources management is a low absenteeism rate. External factors that need to be taken into account might include the age structure of the workforce (younger people are healthier) and the task environment (idea-based and relational tasks are probably less exhausting and more motivating than routine physical jobs) (Hackman & Oldham, 1980). A third example may be indicators on retention and turnover. Organizations invest in hiring and training people, and thus presumably want to retain them. High retention rates may reflect workplace attractiveness. clearly, an important external influence will be the condition of the general labor market and the resulting degree of private-sector competition for critical skills. (3) the objective of a good balance between human resources management arrangements is “to ensure that transaction costs in negotiating shared responsi- bilities between governments are minimized, and that an active labor market is supported for staff with distinctive public sector skills and competencies” (OE cD, 2007). A third set of indicators is implied in this outcome. An active labor market is related to workforce mobility. first, there may be a learning effect. Mobility broadens horizons, leads to the circulation of ideas, and in this way is an antidote against tunnel vision. Second, mobility is in some instances a strategy against cor - ruption and unethical behavior, especially in inspection services. the time spent in a particular sensitive position is too short to develop unsound relations with target groups (Abbink, 2004). third, mobility may be a matter of allocation. Some departments may have staff shortages while others have redundancies. Internal labor markets should sort out these imbalances. the kinds of indicators in this field are typically found in many larger orga- nizations, both private companies and public bureaucracies. Hence, there seems to be fertile ground for solid international and cross-sectoral comparison. the main problem for comparative purposes is that, unlike for budgeting, there are no internationally agreed upon standards and definitions.

Open Government A third field of public administration to be highlighted as a case of comparative measurement concerns the openness of government. Open government mostly re- lates to procedural performance, and therefore performance measurement becomes more difficult. It is a field characterized by measurement experimentation. the OEcD identifies four dimensions of accountability and openness (OEcD, 2001). Transparency is the ability of the government to ensure that its actions, and those who are responsible for its actions, are exposed to public scrutiny and Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 501 challenge. Accessibility in the public sector is achieved when the government can ensure everyone’s capacity to obtain information and to utilize services at any time, anywhere, and in a user-friendly manner. A responsive government is one that can and will react to new ideas, demands, and needs of citizens. Inclusiveness aims to ensure the broadest base of participation possible. the four dimensions of openness and accountability are assumed to build on each other (see Figure 2). transparent government is a precondition for accessible government. citizens need to know their government in order to access it, and citizens need to access government in order for government to respond to them.

Inclusiveness seems to be a different dimension, which assesses to what extent government is transparent for everyone, accessible for everyone, and responsive to everyone. the underlying logic of this scheme resembles Arnstein’s ladder of citizen participation (Arnstein, 1969): the higher on the ladder, the higher the level of participation. In this model, it can be argued, open and accountable governments are at the top of the pyramid. Transparency is a key tenet of contemporary performance and accountability relations (Dubnick, 2005; Piotrowski & Rosenbloom, 2002). freedom of in- formation laws are a key institutional arrangement for achieving transparency.

Performance indicators might reflect whether citizens (and companies and media) actually refer to these laws or whether they remain dead letters. the catch is that in very transparent public administration systems, there is no need to use information laws. Interpretation of performance information in context and through dialogue seems warranted (Moynihan, 2008). the publication of annual reports, performance data, strategic plans, and legis- lative timetables is a means of opening up government activity to the public eye.

Performance indicators could measure the extent to which target groups are actually using available information. A good definition of target groups is necessary. It is not realistic to expect ordinary citizens to read annual reports and strategic plans in their leisure time. It is realistic, however, to expect media and interest groups to Tr an sparen t gove rnme nt Acce ssible gove rnme nt Respon sive gove rnme nt Inclus ive gov ernm ent Figure 2. open and Accountable government: Four dimensions 502 PPMR / March 2012 scrutinize public documents. 6 these mediators play a crucial role in safeguarding transparency in democratic societies. Expert assessments are likely to be a good way to proceed in measuring the quality of these reports. Accessibility. countries mainly safeguard accessibility through administrative law, but ombudsmen, citizen or service charters, and policies to reduce red tape also have accessibility as an objective. It is very difficult to measure the direct impact of these arrangements. Is the number of complaints handled by the ombudsman an indication of problems with accessibility, or does it simply mean the ombudsman is better known? A more general indication of accessibility problems may be the length of the waiting list for services. the problem of this indicator is that waiting lists usually cannot be attributed to open government policies. Although the latter may help to put the issue on the agenda, waiting lists are more likely caused by lack of capacity or extraordinary demands on a service. Responsiveness is pursued through public consultation or active public par - ticipation throughout the policymaking process. According to several authors, responsiveness should be more than a mere client orientation. they stress the importance of citizen-government collaboration (Vigoda, 2002) and coproduction ( b ovaird, 2007), which reflect the quality and depth of the relationship rather than the power or the number of participants. Measuring the performance of partici- pation processes is thus a delicate matter. One possible indicator could measure the range of backgrounds and personal profiles of the participants. In an optimal scenario, participation is widespread and representative, but again, the interpreta- tion of the results is not straightforward. to conclude, the performance of arrangements aimed at an open and account- able government is especially hard to measure. Outcomes can only be observed within society—using public opinion data from citizens, businesses, and privileged observers such as interest groups. these data are perceptual and nonstandardized.

the standardized surveys that do exist, such as Eurobarometer and the world Values Survey, often face the problem of translating core concepts such as legitimacy and trust (MacQuarrie, 2008). the fallback option is to measure citizens’ use of open government arrangements under the premise that arrangements need to be used in order to be effective. Here, international comparison is plagued by differences in the institutional framework that arranges access to government. discussion and conclusion the present article began with the observation that the economic crisis is also a crisis of measurement in the ratings industry. Yet ratings are unlikely to disappear. More generally, measurement seems to be a key tenet of modern society (Porter, 1995; Scott, 1998). Performance indicators are increasingly used to make complex realities tangible and to hold actors accountable. the vast array of activities and performance Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 503 in hospitals, schools, universities, local governments, and the like are captured by a limited number of indicators in performance contracts, league tables, and star ratings (bevan & Hood, 2006; Moynihan, 2009; Radin, 2006). the story of the rating agencies also shows that the impact of these indicator regimes can be substantial. In order to ensure that accountability for performance is providing the right effects in government and society, the quality and diversity of indicators becomes crucial. this is also the case for public administration indicators. the quality of public administration indicators is often problematic, in particular when they are internation- ally comparative. Nonetheless, they are increasingly used, among other purposes, to inform funding, investment decisions, the rating of government bonds, and media coverage (united Nations Development Programme, 2004). As a result, the develop- ment of diverse and valid comparative indicators seems to be warranted. this article builds on OEcD initiatives with a conceptual discussion of indicators in the fields of budgeting, human resources management, and open government. the cases show that public administration performance measurement is at least conceptually conceivable for all three fields. there are differences between the cases, however. for budgeting, good comparative performance indicators can be developed. the high level of standardization and cyclical nature of budgeting contributes to the potential. f or human resources management, the development of performance indicators is also relatively straightforward, but international comparison may be more difficult due to the lack of standardization. for open T X D O L W \ R I P H D V X U H P H Q W X V H K L J K O R Z K L J K O R Z E X G J H W L Q J R S H Q J R Y + 5 0 F U H G L W U D W L Q J V J R Y H U Q D Q F H L Q G L F D W R U V Figure 3. Measurement Quality and use 504 PPMR / March 2012 government, the procedural character seems to be a hindrance to the development and interpretation of indicators. for all three fields, performance indicators will only make sense in context, suggesting that there is need for dialogue on indica- tors (Moynihan, 2008). Accountability for performance should not be confined to accountability for performance indicators. the article mainly deals with the validity and robustness of international compari- sons of public administration. Yet, as is argued above, the use of performance indicators is an equally important dimension. Although use was not the main object of the article, it ends with some reflections on how measurement quality and use may interact.

figure 3 combines measurement quality and its use and gives a rough assessment of where to locate the cases of budgeting, human resources management, and open government. It also plots the general governance indicators (for an overview, see Van de walle, 2006) and the credit ratings of cDOs that sparked the economic crisis.

International comparative data on open government and human resources man- agement tend to be of relatively low quality (somewhat better on quality for human resources management). this, however, is not problematic, because the low intensity of use leaves room for experimentation. budgeting has higher quality, but international comparisons of budget data are used quite intensively by international institutions. the case of low use and high quality is mainly a missed opportunity. the most problematic quadrant is high use of low-quality data. the credit ratings clearly were to be situated in this category, and several governance indicators also seem to head this way. use and quality of data are intertwined. Once data are used intensively and have an impact, they become institutionalized. At that point, it becomes difficult to improve quality, because changes in definitions and methods also touch upon interests and challenge routines. Efforts to alter GDP calculations—another inten- sively used indicator—are a case in point (Van Dooren, 2009). In other words, it is very difficult to move upward from the lower-right quadrant in figure 3. Another implication is that being in the lower-left quadrant should be cherished. this is the period in which the DNA of the performance indicators can be established through experimentation. for indicators of public administration performance, there is still room for experimentation. finally, the admittedly rough analysis of the quadrants may call for more longitudinal research on how indicator sets are developed over time, how invalid data are able to construct a reality, what determines the use of valid data, and where the indicator sets originate. notes 1. the Obama administration has been very critical of the quality of Standard & Poor’s ratings since the u.S. downgrade (“President Obama’s statement,” 2011). In the media, there have been some less critical accounts of the role of rating agencies (e.g., tasker, 2011), together with more critical and very critical perspectives (e.g., “credit-rating agencies,” 2011; Rushe, 2011). 2. the views expressed in this article are entirely those of the authors and do not reflect any opinion of the OEcD or its member governments. Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 505 3. A free rider is someone who enjoys the benefits of a public good without bearing the cost. the image of someone who uses public transport without paying is illustra\ tive.

4. Externalities are costs and benefits attributable to an activity that are not reflected in the price of the goods or services being produced. 5. this is mainly done through the establishment of the so-called European semester, a more frequent follow-up of national budgets by the Eu, with new incentives and sanctions for countries that do not provide qualitative budget data (Europa Press Release, 2011) 6. there is a parallel with private-sector annual reports. Only a selected audience of fi- nancial journalists, analysts, and interest groups have the time and competence to assess these reports. references Abbink, K. (2004). Staff rotation as an anti-corruption policy: An experimental study. European Journal of Political Economy, 20(4), 887–906.

Alt, J.E., & lassen, D.D. (2006). transparency, political polarization, and political budget cycles in OEcD countries. American Journal of Political Science, 50(3), 530–550.

Andrews, R., boyne, G.A., Meier, K.J., O’t oole, l.J., Jr., & walker, R.M. (2005). Rep- resentative bureaucracy, organizational strategy, and public service performance: An empirical analysis of English local government. Journal of Public Administration Research and Theory, 15(4), 489–504.

Arndt, c. (2007). The politics of governance ratings. Maastricht: Maastricht Graduate School of Governance.

Arndt, c., & Oman, c. (2006). Uses and abuses of governance indicators. Paris: Organiza- tion for Economic cooperation and Development.

Arnstein, S.R. (1969). A ladder of citizen participation. Journal of the American Planning Association, 35(4), 216–224.

benito, b., & bastida, f. (2009). budget transparency, fiscal performance, and political turnout: An international approach. Public Administration Review, 69(3), 403–417.

bevan, G., & Hood, c. (2006). what’s measured is what matters: targets and gaming in the british health care sector. Public Administration, 84(3), 517–538.

blondal, J. (2003). budget reform in OEcD member countries: common trends. OECD Journal on Budgeting, 2(4), 7–25.

bovaird, t. (2007). beyond engagement and participation: user and community coproduc- tion of public services. Public Administration Review, 67(5), 846–860.

colonna, A., & Puma, J. (Eds.). (2003). Paths to performance in state and local government:

A final assessment of the Maxwell School of Citizenship and Public Affairs. Syracuse, NY: campbell Public Affairs Institute.

credit-rating agencies: Judges with tenure. (2011). Economist, August 11. Available at www.economist.com/node/21525936/, accessed August 2011.

crotty, J. (2009). Structural causes of the global financial crisis: A critical assess- ment of the new financial architecture. Cambridge Journal of Economics, 33(4), 563–580.

De Renzio, P., & Masud, H. (2011). Measuring and promoting budget transparency: the open budget index as a research and advocacy tool. Governance, 24(3), 607–616.

Douglas, J.w ., & franklin, A.l. (2006). Putting the brakes on the rush to spend down end-of-year balances: carryover money in Oklahoma state agencies. Public Budgeting & Finance, 26(3), 46–64.

Dubnick, M. (2005). Accountability and the promise of performance: In search of mecha- nisms. Public Performance and Management Review, 28(3), 376–417.

Europa Press Release. (2011). European semester: A new architecture for the new EU 506 PPMR / March 2012 economic governance—Q&A. Available at http://europa.eu/rapid/pressReleasesAction.

do?reference=MEMO/11/14/, accessed August 2011.

Eurostat. (2001). Handbook on price and volume statistics in the national accounts. lux- embourg: European communities.

ferri, G., liu, l.G., & Stiglitz, J.E. (1999). the procyclical role of rating agencies: Evi- dence from the East Asian crisis. Economic Notes, 28(3), 335–355.

Gray, A., Jenkins, b., & Segsworth, b. (2001). Budgeting, auditing, and evaluation: Func- tions and integration in seven governments. New brunswick, NJ: transaction.

Government Accountability Office. (2007). Performance and accountability report, 2007.

w ashington, Dc.

Gualmini, E. (2008). Restructuring weberian bureaucracy: comparing managerial reforms in Europe and the united States. Public Administration, 86(1), 75–94.

Hackman, J., & Oldham, G.R. (1980). Work redesign. Reading, MA: Addison-w esley.

Hall, J.l. (2008). the forgotten regional organizations: creating capacity for economic development. Public Administration Review, 68(1), 110–125.

Hatry, H.P. (1999). Performance measurement: Getting results. washington, Dc: urban Institute Press.

Hood, c. (1991). A public management for all seasons. Public Administration, 69(1), 3–19.

Hood, c. (2006). Gaming in target world: the targets approach to managing british public services. Public Administration Review, 68(4), 515–521.

Jorgensen, t.b., & bozeman, b. (2007). Public values: An inventory. Administration & Society, 39(3), 354–381.

Kerwer, D. (2005). Holding global regulators accountable: the case of credit rating agen- cies. Governance, 18(3), 453–475.

Kristensen, J., Groszyk, w., & böhler, b. (2002). Outcome-focused management and budgeting. OECD Journal on Budgeting, 1(4), 7–35.

lonsdale, J., wilkins, P., & ling, t. (2011). Performance auditing: Contributing to ac- countability in democratic government. cheltenham, uK: Edward Elgar.

MacQuarrie, c. (2008). Putting the demos back in democracy: Declining citizen trust in government and what to do about it. Ottawa: School of Political Studies, university of Ottawa.

Marlowe, J. (2009). Public financial engineering and its discontents. Public Performance & Management Review, 32(4), 626–630.

Maxwell School of citizenship and Public Affairs. (2002). Paths to performance in state and local government: A final assessment from the Maxwell School of Citizenship and Public Affairs. Syracuse: Maxwell School of citizenship and Public Affairs.

Moore, M.H. (1995). Creating public value strategic management in government. cam- bridge: Harvard university Press.

Mortgage cartoon Glasbergen [image] (2005). Available at www.glasbergen.com/real- estate-cartoons/, accessed 2007.

Moynihan, D.P. (2008). The dynamics of performance management: Constructing informa- tion and reform. washington, Dc: Georgetown university Press.

Moynihan, D.P. (2009). through a glass, darkly. Public Performance & Management Review, 32(4), 592–603.

National Audit Office. (2007). Annual report, 2007. london.

OEcD. (2001). Citizens as partners: Information, consultation and participation in policy making. Paris.

OEcD. (2002). Public service as an employer of choice. Paris.

OEcD. (2006). OECD budget practices and procedures database phase II: Final glos- sary. Paris. Van Dooren, De caluwé, and lonti / MEASuRING PublIc ADMINIStRAtION PERfORMANcE 507 OEcD. (2007). OECD reviews of human resources management in government: Belgium.

Paris.

Office of Personnel Management. (2007). Performance and accountability report: Fiscal year 2007 (pp. 1–229). washington, Dc.

Piotrowski, S.J. (2010). An argument for fully incorporating nonmission-based values into public administration. In R. O’leary, D.M. Van Slyke, & S. Kim (Eds.), The future of public administration around the world: The Minnowbrook perspective (pp. 27–33).

w ashington, Dc: Georgetown university Press.

Piotrowski, S.J., & Rosenbloom, D.H. (2002). Nonmission-based values in results-oriented public management: the case of freedom of information. Public Administration Review, 62(6), 643–657.

Porter, t.M. (1995). Trust in numbers: The pursuit of objectivity in science and public life.

Princeton: Princeton university Press.

President Obama’s statement on credit downgrade (2011). washington, Dc: white House. Available at www.whitehouse.gov/photos-and-video/video/2011/08/08/ president-obamas-statement-credit-downgrade?page=6&v=accessibility/, accessed September 2011.

Raaum, R.b., & Morgan, S.l. (2001). Performance auditing: A measurement approach.

Altamonte Springs, fl: Institute of Internal Auditors.

Radin, b.A. (2006). Challenging the performance movement: Accountability, complexity, and democratic values. washington, Dc: Georgetown university Press.

Rubin, I.S. (1997). The politics of public budgeting: Getting and spending, borrowing and balancing. chatham, NJ: chatham House.

Rushe, D. (2011). How do you rate the rating agencies? Guardian, March 31. Available at www.guardian.co.uk/commentisfree/cifamerica/2011/mar/31/ratings-agencies-credit- crunch/, accessed August 2011.

Scott, J.c. (1998). Seeing like a state: How certain schemes to improve the human condi- tion have failed. New Haven: Yale university Press.

Sterck, M. (2007). the impact of performance budgeting on the role of the legislature: A four-country study. International Review of Administrative Sciences, 73(2), 189–203.

t arschys, D. (2002). time horizons in budgeting. OECD Journal on Budgeting, 2(2), 77–104.

t asker, P. (2011). How to make monkeys out of rating agencies. Financial Times, August 11. Available at www.ft.com/intl/cms/s/0/b8d489a8-c363-11e0-b163-00144feabdc0.

html#axzz1Ve5yc100/, accessed August 2011.

united Nations. (1993). Revised system of national accounts: SNA 1993. New York.

united Nations Development Programme. (2004). Sources for democratic governance indicators. Oslo.

Vandenabeele, w. (2008). Government calling: Public service motivation as an element in selecting government as an employer of choice. Public Administration, 86(4), 1089–1105.

Van de walle, S. (2006). the state of the world’s bureaucracies. Journal of Comparative Policy Analysis: Research and Practice, 8(4), 437–448.

Van Dooren, w. (2009). A politico-administrative agenda for progress in social measure- ment: Reforming the calculation of government’s contribution to GDP. Journal of Comparative Policy Analysis: Research and Practice, 11(3), 309–326.

Van Dooren, w., bouckaert, G., & Halligan, J. (2010). Performance management in the public sector. london: Routledge.

Verhoest, K., bouckaert, G., & Peters, b.G. (2007). Janus-faced reorganization: Specializa- tion and coordination in four OEcD countries in the period 1980–2005. International Review of Administrative Sciences, 73(3), 325. 508 PPMR / March 2012 Vigoda, E. (2002). from responsiveness to collaboration: Governance, citizens, and the next generation of public administration. Public Administration Review, 62(5), 527–540. Wouter Van Dooren is an assistant professor of public administration in the Department of Political Science, University of Antwerp, Belgium. He researches performance measurement and management and recently coauthored Performance Management in the Public Sector (Routledge, 2010).

Chiara De Caluwé is a researcher in the Department of Political Science, Uni- versity of Antwerp, Belgium. She is working on a Ph.D. dissertation concerning performance by public administration.

Zsuzsanna Lonti is the head of the statistics and indicators unit in the Budgeting and Public Expenditures Division of the OECD Governance Directorate, Paris.

She leads the Government at a Glance project.

Copyright of Public Performance & Management Review is the property of M.E. Sharpe Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.