Article Critique Essay
A Systems Approach to Measuring Return on Investment for HRD Interventions Greg G. Wang, Zhengxia Dou, Ning Li This study contributes to the limited methodological literature on HRD program evaluation and measurement.
The study explores an inter- disciplinary approach for return on investment (ROI) measurement in human resource development (HRD) research and practices.
On the basis of a comprehensive review and analysis of relevant studies in economics, industrial-organizational psychology, financial control, and HRD fields, we developed a systems approach to quantitatively measure ROI for HRD programs. The ROI concept for HRD field was defined, and a theoretical systems framework was developed.
The applicability of using statistical and mathematical operations to determine ROI and isolate non-HRD program impacts is discussed.
Application scenarios are presented to demonstrate the utility of the systems approach in real-world ROI measurement for HRD interventions.
Return on investment (ROt) in human resource development (HRD) has been a hot issue pursued by many HRD researchers, practitioners, and organizations during the past decade (Philhps, 1997b). Current approaches to ROI mea- surement in HRD are rooted in and center around an accounting model devel- oped by the DuPont company in 1919 for decentralized financial control (Koontz, O'Donnell, and Weihrich, 1984; Nikbakht and Groppelli, 1990).
HRD intervention is a process much more complex than accounting because the former involves dynamic human behaviors. Benefits derived from HRD intervention often tangle with the impact of other organizational variables.
Note: The authors are grateful to seven anonymous reviewers for their many constructive comments and suggestions on earlier versions of this article. We also wish to thank the twelve hundred members of ROInet, an Internet forum dedicated to this subject, especially Dean Spitzer, Fred Nickols, Phillip Rutherford, Terje Tonsberg, and Matt Barney, for their many thought-provoking online discussions over the past two years.
HUMAN RESOURCE DEVELOPMENT QUARTERLY, vol. 13, no. 2, Summer 2002 Copyright ® 2002 Wiley Periodicals, Inc. Wang, Dou, Li Thus ROI measurement for HRD programs requires identifying the HRD ben- efits and separating them from other impacts if the ROI measurements are to be accurate (Wang, 2000).
The purpose of this study is to explore a systems approach to measuring ROI benefits by integrating theories and methodologies developed in other dis- ciplines, such as economics as well as industrial-organizational (I/O) psychol- ogy, with HRD research and practices. We first conduct extensive reviews and analyses of studies in economics, I/O psychology, and the human resource-related area on training ROI, with a thorough examination of the evo- lution of ROI approaches in the HRD field. We then define an ROI concept explicitly for HRD interventions, construct a framework as the theoretical foun- dation for a systems approach, and discuss operationalizing the approach.
Application scenarios are subsequently presented to demonstrate the utility of the systems approach to ROI measurement in the HRD field.
Economic Studies on Training ROI Economic analyses of training ROI can be broadly categorized into macro and micro studies. The macro-level studies deal with the relationship between training and national economic performance. A general conclusion of such studies is that training improves labor quality, which in turn counts as one of the most important factors contributing to economic growth (Sturm, 1993).
At the micro level, economic studies measure the impact of training on com- pany productivity and profitability This article relates to the micro level of ROI studies. Readers interested in macro-level studies may find more information in the reviews by Sturm (1993) and Mincer (1994).
At the micro level, economic studies on training ROI can be traced to the 1950s, when several economists coined the term human capital and originated studies on industrial training.^ As a pioneer of study on the subject. Mincer pinpoints the core issues for empirical economic studies on human capital, or "capital formation in people," as "(1) How large is the allocation of resources to the training process? (2) What is the rate of return on this form of invest- ment?" (1962, p. 50). As is apparent, the first issue relates to identifying train- ing cost; the second is the ROI measurement that interests HRD professionals today Becker, a Nobel laureate, developed a theoretical model to examine investment in human capital and its marginal productivity (Becker, 1962), in which human capital was categorized as schooling and on-the-job training.^ Early research on training ROI on the part of economists focuses on the causality of company training and company performance (Becker, 1962; Ben-Porath, 1967). Critical issues being explored include whether company training leads to enhanced profitability, or higher profitability fosters a com- pany's investment in training; and how productivity and training investment infiuence each other.^ The general conclusion of the human capital theory is that a company providing training programs is to forgo current profit-earning A Systems Approach to Measuring ROI 205^ opportunity for future profitability (Becker, 1962, 1964; Ben-Porath, 1967; Ehrenberg and Smith, 1994; Feuer, Glick, and Desai, 1987; Kaufman, 1994).
The effect of training on productivity growth is approximately five times as much as would be generated by compensation incentives (Barron, Berger, and Black, 1993; Bishop, 1991).
Empirical ROI studies by economists were brought to a new level when the Congress required evaluation of training programs sponsored by the U.S.
government. These training programs are a series of large-scale job training initiatives designed to reduce unemployment and help disadvantaged social groups enter the labor market. The programs are the Manpower Development Training Act (MDTA) in the 1960s, the Comprehensive Employment Training Act (CETA) in the 1970s, the Job Training Partnership Act QTPA) in the 1980s, and more recently the Workforce Investment Act (WIA) of 1998. Relevant ROI evaluation studies by economists started in the early 1970s with CETA pro- grams (Maynard, 1994; O'Neill, 1973; Perry, 1975). Various models and tech- niques were developed for experimental (Heckman, Hotz, and Dabos, 1987), nonexperimental (Heckman and Hotz, 1989), and quasi-experimental (Moffitt, 1991) measurements. The net impact of government training programs, the validity and reliability of ROI results, and the sensitivity of various method- ologies were thoroughly scrutinized (Dickinson, Johnson, and West, 1987; Eraker and Maynard, 1987; Heckman, Hotz, and Dabos, 1987; Moffitt, 1991).
Eor more than forty years, economists have brought to research on train- ing ROI a full spectrum of theories, approaches, and techniques, covering ROI measurement design (Eriedlander and Robins, 1991; Hotz, 1992), sampling methods (Heckman, 1979; Heckman and Hotz, 1989), statistical modeling (Eraker and Maynard, 1987; LaLonde, 1986;), bottom-line result analysis (LaLonde and Maynard, 1987; Perry, 1975), and measurement accuracy analysis (Barron, Berger, and Black, 1997; LaLonde, 1986). Integrating the rich studies in economics with existing ROI efforts in the HRD field may create an impetus to advance ROI research and practices in the HRD field.
Existing economic studies of training ROI are not vvdthout limitation when considering their applicability to the HRD field. Eirst, economists are mainly interested in issues involving societal resource allocation, determination of wages and employment in the labor market, and so on (Kaufman, 1994). As a result, relevant economic studies of ROI measurement rarely offer pre- program decision-making recommendations or postprogram feedback for HRD intervention, which, by contrast, are the highlight of ROI measurement for HRD programs. Eurthermore, economists generally lack in-depth knowl- edge and expertise in HRD and performance improvement. Thus their recommendations derived from research rest heavily on generic policy impli- cations instead of specifics sought by organizations. Clearly, it is to HRD researchers and practitioners' advantage to present ROI measurement and information on program improvement and decision making at the level of the organization. Wang, Dou, Li Utility Analysis in Industrial-Organizational Psychology Utility analysis (UA), developed in the field of I/O psychology, is another fam- ily of theories relevant to ROI measurement in HRD. The concept of utility originated from studies in the economics of consumer behavior (Stigler, 1950), which was integrated with psychological and psychometric theories for ana- lyzing cost, benefit, and return of various organizational programs. Hence, util- ity analysis bridges economics theory with organizational-measurement reality (Boudreau, 1991).
Perhaps because of the nature of the subjects, only a number of UA stud- ies in I/O psychology specifically focused on the business impact of training; the majority are about economic gain or loss from other HR-related programs.
Brogden and Taylor (1950) proposed a cost accounting concept for measuring the dollar value of employee selection, turnover, training, and on-the-job per- formance. Doppelt and Bennett (1953) explicitly examined the relationship between the effectiveness of selection testing and training cost. The significance of Doppelt and Bennett's study for HRD measurement is that the authors rec- ognized the complexities of training measurement and discussed issues of direct and indirect training costs, which are frequently measured by today's HRD professionals.
Although early UA research centered on the utility of employee selection testing (Taylor and Russell, 1939), recent UA studies are focused heavily on decision options regarding enhancing the productivity of human resource man- agement programs, such as selection testing, recruitment, staffing, and com- pensation (Boudreau, 1991). Among various UA approaches developed by I/O psychologists (Brogden, 1946, 1949; Cronbach and Gleser, 1965; Naylor and Shine, 1965; Raju, Burke, and Normand, 1990; Taylor and Russell, 1939), the Brogden-Cronbach-Gleser (BCG) model has particular significance for ROI measurement in the HRD field.
Employing a dollar criterion, Brogden (1949) used a psychometric concept SDy (or (jy), denoting the standard deviation of the dollar value of performance, to measure the economic utility of a program. Brogden's approach was later refined by Cronbach and Gleser (1965); thus the BCG model. When applied to training UA study, the BCG model was expressed in this equation (Schmidt, Hunter, and Pearlman, 1982):
AU = TNd, SDy - NC (equation 1) where AU is the net dollar value of the training; T the duration of the training effect on performance; N the number of trainees; d, the true difference in per- formance between the average trained and untrained employee, in SD units; SDy the standard deviation of performance in dollars of the untrained group; and C the cost of training per trainee. Equation 1 was used for a hypotheti- cal computer programmer training in a control group setting to estimate the A Systems Approach to Measuring ROI 207 dollar impact, or the utility, of the training (Schmidt, Hunter, and Pearlman, 1982).
It is clear that if AU in equation 1 can be measured accurately, then ROI for a training program, in the form of the accounting equation, becomes ROI = AU/(total cost of training) X 100%. Therefore, training ROI measure- ment would be a matter of simple mathematical derivation. Unfortunately, the approach to measuring the key term, SD^ in equation 1, has not yet been satis- factorily resolved. The conventional approach to obtaining SD> is subjective estimation of the trainee's or supervisor's perception of the dollar value of relevant performance, which is typically classified in three percentiles: the 15th, or low-performer; 50th, or average performer; and 85th, or superior per- former (Schmidt, Hunter, McKenzie, and Muldrow, 1979; Mathieu and Leonard, 1987; Cascio, 2000).
There has been much criticism of this "judgment-based" approach (Latham and Whyte, 1994; Raju, Burke, and Normand, 1990). Cascio (2000) noted that the estimation process "lacks face validity because the components of each (rater's) estimates are unknown and unverifiable" (p. 231). As a result, the estimations considerably overstate actual returns "because researchers have tended to make simplifying assumptions with regard to certain variables, and to omit others that add to an already complex mix of factors" (Cascio, 1992, p. 31); "the result has been to produce astronomical estimated payoffs for valid selection programs" (p. 34). Thanks to the unsolved problem in esti- mating SD^, Latham and Whyte (1994) reported in a recent study that man- agers, when making decisions regarding human resources, did not consider the result of UA studies even when impressive evidence on dollar returns was presented to them. Clearly, if HR managers are to gain the respect of this bottom-line approach, they need to use dollar figures in a more credible fash- ion (Wolf, 1991).
Recognizing the challenges in estimating SDy, Raju, Burke, and Normand (1990) proposed an alternative model (the RBN model) for UA measurement to circumvent subjective estimation of SD>..
The RBN model is mathematically comparable to the BCG model, the major difference being in the parameters that are estimated. Instead of measuring judgment-based SDy in dollars, the RBN model estimates a multiphcative constant in a hnear regression frame- work. Although seemingly an improvement from the BCC model, applicabil- ity of the RBN model has been questioned in a recent empirical study (Law and Myors, 1999).
UA approaches have yet to be adopted by HRD practitioners for ROl- related measurement. One reason may be that the nature and functions of HRD programs differ substantially from those of such 1/0-related interventions as selection testing, turnover and layoff, and performance appraisal. For instance, Schmidt, Hunter, and Pearlman (1982) discussed the differences between selection and HRD interventions. Furthermore, extensive use of psy- chological and psychometric concepts in 1/0 psychology may be another 208 Wang, Dou, Li obstacle preventing HRD practitioners from using UA models. Additionally lack of acceptance of UA method by HRD practitioners may be due to the subjective nature of the approach (Phillips, 1997a).
ROI Measurement iti the HRD Field In contrast to the rigorous methodologies developed by economists and psy- chologists for training measurement and evaluation, ROI in the HRD arena is still in its infancy with largely descriptive rather than quantitative features. ROI analysis in the HRD field can be traced back to Kirkpatrick's four-level train- ing evaluation model (1959), which divides training evaluation into four levels:
reaction, learning, application, and business impact. Of these, only the fourth level is relevant to ROI measurement. Besides some general evaluation guide- lines (Kirkpatrick, 1959, 1967, 1977, 1998), the model does not offer specific techniques for quantitative evaluation of training programs, even at the fourth level. As has been pointed out by several authors (Alliger and Janak, 1989; Bobko and Russell, 1991; Holton, 1996), Kirkpatrick's model is more a tax- onomy or classification scheme than a methodology for training evaluations.
Perhaps the greatest contribution of the model, as well as the reason it gained popularity in the HRD community lies in the fact that it created a common vocabulary for HRD practitioners to communicate their evaluation efforts.
In his First Leisurely Theorem, Gilbert (1978) states that human compe- tence is a function of worthy performance expressed as the ratio of valuable accomplishments to costly behavior. Although not intended specifically for ROI measurement, the theorem is one of the first rigorous efforts to model human competence in the HRD field. Current ROI models used in the HRD field still mirror Gilbert's influence (Brandenburg, 1982; Bakken and Bernstein, 1982; Fitz-enz, 1984; Swanson and Gradous, 1988).
The approach used by an organization for training evaluation and mea- surement may also depend on the organizational nature and function. To give one example, cost effectiveness of training usually is the core measurement for U.S.
military training programs (Browning, Ryan, Scott, and Smode, 1977; Allbee and Semple, 1980). Thode and Walker (1983) proposed a model for identifying the structure of training cost for a P-3 fieet readiness squadron (FRS) aircrew training system measurement. For government agencies and nonprofit institutions, ROI measurement often focuses on the effectiveness of accomplishing a specific mission rather than monetary value.
It was not until the early 1980s that the term return on investment was used explicitly in training evaluations (see, for instance, Bakken and Bernstein, 1982).
Meanwhile, a wave of interest in ROI was developing in the HRD field.
The move may have reflected widespread downsizing in the business world at the time, as well as the need for training and HRD justification accordingly Since then, ROI for HRD programs has evolved from Kirkpatrick's descriptive model to a more quantitative measurement, represented by a simplified A Systems Approach to Measuring ROI 209 accounting equation. This accounting-type equation, as a variation of DuPont's financial control model,'^ has been used by many in conjunction with benefit- cost (B/C) analyses (Bakken and Bernstein, 1982; Brandenburg, 1982; Fitz-enz, 1984; Fusch, 2001).
The ROI and B/C^ equations were expressed as:
ROI = (total benefit - total cost)/total cost X 100% (equation 2) B/C = total benefit/total cost (equation 3) Seemingly simple and easy-to-use, equations 2 and 3 suffer the same problem that psychologists encountered in estimating SDy in UA analysis (equation 1). To identify the term of benefit for the numerator in the equations, Phillips (1997a, b) used an approach similar to the judgment-based estimation employed by 1/0 psychologists: asking the program participants or their super- visors to assign dollar values to the results of a specific training program. The only difference is that it added a step, requesting the rater's confidence level, ranging from 0 to 100 percent, toward assigned dollar value (Phillips, 1997a).
Obviously, the critiques made by I/O psychologists (Cascio, 1992, 2000; Wolf, 1991) on SDy estimation noted earlier apply equally to Phillips's approach here.
This is because many indicators of HRD programs do not lend themselves to monetary valuation, and the basis for attributing the observed benefit to an individual program or participant is rather vague or nonexistent.
In the financial vv'orld, many researchers and practitioners have criticized the accounting ROI equation for its lack of applicability in financial measure- ment (Dearden, 1969; Searby, 1975; Stewart and Stern, 1991). In the mean- time, researchers and practitioners are searching for alternatives to measure organizational financial perfonnance (Myers, 1996). One recent example is the development of the economic value added (EVA) concept (Ehrbar, 1998; Stew- art and Stern, 1991) and its application in measuring a range of company financial performance. Likewise, the challenge presented by the accounting approach in the HRD area is substantial.
If we consider the four-level framework as a description of HRD program evaluation and measurement, the application of the ROI formula may be viewed as an attempt to address how it can be done. It is certainly a worth- while effort toward resolving the puzzle of HRD evaluation and measurement, although it has an inherited weakness from its financial counterpart. Unlike the four-level framework, which serves as a classification scheme (Alliger and Janak, 1989; Bobko and Russell, 1991; Holton, 1996) and communication tool, the ROI equation and its applications should not be considered an exten- sion of the four-level classification, that is, a fifth-level evaluation (Phillips, 1997b). In fact, it is inappropriate to parallel a specific approach or applica- tion of a formula with a classification scheme. Our analysis, however, by no means undermines the significant role played by the ROI equation and its 210 Wang, Dou, Li applications. Perhaps one of the greatest contributions made by Phillips's work (1994, 1997a, 1997b) is that it has substantially increased awareness of the importance of evaluation and ROI measurement in the HRD practitioner com- munity and at the upper management level. It has also created a more quanti- tative HRD program measurement environment than the descriptive classification scheme.
Along with the typical accounting ROI model for after-the-fact measure- ment, forecasting models for before-tbe-fact estimation can be important for HRD investment decision making. An approach known as forecasting finan- cial benefits (FFB) bas been proposed (Geroy and Swanson, 1984; Swanson and Gradous, 1988; Swanson, 1992). Using a formula similar to Gilbert's First Leisurely Theorem (1978), tbe FFB approach predicts tbe benefits to be accrued according to tbe type of investment to be made. For a given HRD pro- gram, FFB calculates benefit by subtracting cost from performance value.
Benefit is the financial contribution for tbe work to be produced; performance value is tbe financial worth of the work that will be generated through tbe HRD program. Evidently, obtaining performance value can be difficult because it is affected by many intervening factors. The FFB approach bas received limited attention in HRD research and practices Qacobs, Jones, and Neil, 1992).
The Dilemma of ROI Measurement Increasing business need for measuring intellectual capital and HRD interven- tion in botb the private and public sectors bas created an ROI surge in tbe HRD community in recent years. However, existing ROI approaches are inad- equate to meet the needs of HRD practices. Tbe dilemma can be summarized in at least tbree aspects.
First, current accounting equations for ROI analysis only deal vidtb param- eters having dollar values. Equations 2 and 3 bave no solution unless numeric dollar values are explicitly assigned to the parameters. For intangible measures such as customer or employee satisfaction or ROI measurement for an organi- zation whose business objectives are not necessarily measured in dollar values (government agency military, or otber nonprofit organization), tbe current ROI approach is inapplicable. Despite the strong need for intangible measurement in these organizations Q. Yan, personal communication, Apr. 16, 2000), prac- tical approaches for such measurements are seldom discussed in tbe HRD lit- erature.
Second, business outcome can often be attributed to HRD programs plus many other concurrently intervening variables. However, tbe current ROI approach fails to separate true HRD program impact from other intervening variables. Although some general guidelines bave been discussed (Phillips, 1997a, 1997b), lack of rigorous methodology for isolating HRD impact and separating it from otber variables remains a major obstacle for furtber ROI researcb and practices. In some cases, a control-group approach may be a A Systems Approach to Measuring ROI remedy Yet in many cases tbe control-group approach may not be applicable.
(For a detailed review, classification, and application of the control group approach, see Wang, 2002.) Furthermore, tbe current judgment-based dollar value estimation for "total benefit" to obtain tbe numerator for the accounting ROI equations is rather ironic in tbe HRD field. By definition, HRD as a profession is intent on improv- ing competency-based performance (Gilbert, 1978). Yet tbe participants who are asked to assign a dollar value, along witb a confidence percentage, to an HRD program may or may not be qualified to do so. Tbe ROI results obtained in this fashion can be not only subjective but also misleading. Such ROI results, if serving top management in making HRD-related investment deci- sions, may jeopardize tbe long-term credibility of HRD intervention.
The diverse situation in HRD interventions for many types of organiza- tions and the dilemma in existing ROI measurement approacbes bave led some practitioners to search for a nonquantitative approach to circumventing tbe ROI issue. Return on expectations (McLinden and Trocbim, 1998) is one example of sucb an alternative. As bas been noted in several studies (Holton, 1996; Kaufman and Keller, 1994), if HRD is to grow as a discipline and a pro- fession it is vital to develop a rigorous and comprehensive evaluation and mea- surement system. In tbis study, we attempt to apply interdisciplinary theory and metbodology to ROI measurement for tbe HRD field. We first clarify tbe ROI concept with a definition for tbe field. A systems framework is establisbed as the tbeoretical foundation, and we tben operationalize tbe model to HRD settings. Two scenarios tbat are familiar to many in HRD are presented to demonstrate tbe applicability of tbe systems approach. Finally, recommenda- tions for future study are discussed.
Return on Investment: A Definition for the HRD Field From tbe preceding discussions and analyses, it is clear that ROI measurement in HRD may be approached by integrating interdisciplinary efforts witb HRD realities and cbaracteristics. Tbe field of HRD itself is interdisciplinary, after all (Rotbwell and Sredl, 1992; Swanson and Holton, 2001). Investment and returns in tbe HRD field differ significantly from tbose in tbe accounting and financial world in several aspects; bence ROI measurement for HRD is more tban simply an accounting issue.
First, HRD investment is not an investment in fixed assets. Ratber, it involves a learning process witb subsequent behavior cbange and performance improvement (McLagan, 1989; Rotbwell and Sredl, 1992). How mucb one can leam and bow successfully one may apply tbe learned skills on tbe job depends on many variables: past learning experience, education background, work experience, and tbe like. Second, tbe impact of learning application on final business results often interacts witb factors at organizational and environmental levels, sucb as organization culture and market conditions (Wang, 2000). 212 Wang, Dou, Li Tbird, the benefit of HRD intervention may take on a value otber tban mone- tary Tbe purpose of some HRD programs is to cbange participant bebavior, wbich can rarely be measured in dollars. Many studies have sbown tbat bebav- ior cbange may or may not lead to cbange in productivity (Pritcbard, 1992).
Gonsidering the characteristics of ROI in tbe HRD field, we redefine return on investment for an HRD program as any economic retums, monetary or non- monetary, tbat are accrued tbrough investing in the HRD interventions. Mon- etary returns refer to those that can be measured and expressed in dollar values; nonmonetary returns apply to any other returns that have economic impact on the business results but may not be explicitly expressed in dollar values. Fxamples of nonmonetary return are sucb parameters as customer satisfaction and employee job satisfaction.
Glearly tbe ROI concept defined here is not confined within the account- ing ROI equations. Rather, it extends HRD measurement efforts to a much broader range, regardless of tbe nature of tbe organization (for-profit or non- profit, government or private institute) or tbe form of program outcome.
We now establish an analytical framework on tbe basis of tbis explicit definition.
An Analytical Framework Tbe general purpose of an HRD intervention is to produce a competent and qualified employee to perform an assigned job and contribute to the organi- zation's business outcomes. Tbus HRD interventions ought to be examined witbin tbe overall organization's production system, witb three components:
input, process, and output (Bertaianffy, 1968).
First, consider HRD as a subsystem witbin a production system (Figure 1) Tbe input may include employees, and HRD investment in terms of dollar value, time, and otber effort, equipment, and materials.
Process refers to spe- cific HRD interventions, sucb as a training event, a performance improvement Figure 1. A System View of HRD Interventions and the Potential Impact Production System HRD Subsystem Input Process Output 1 1 Y All other inputs Input 1 J —>• Process —*1 Output A Systems Approach to Measuring ROI intervention, or an e-learning initiative. Output is usually reflected by enbanced performance on the part of the employees involved. To realize tbe value of an HRD program, tbe output of tbe HRD subsystem must enter tbe production system as an element of input. It is the relationship between the HRD subsys- tem and tbe production system, and tbe interaction of the two, that creates challenges in ROI measurement. In otber words, a good output from the HRD subsystem alone, now as part of the inputs in the larger production system, is a necessary-but-not-sufficient condition to produce tbe final output, because the output from the HRD subsystem is now only a component of tbe produc- tion inputs. Undoubtedly tbe final business result depends not only on tbis particular input but also on otber system inputs and the effectiveness of busi- ness processes. Tbe key to a comprebensive ROI measurement for an HRD program is to identify and measure tbe impact of the HRD subsystem on tbe output of tbe production system.
An analogy may be made between the concepts illustrated in Figure 1 and Kirkpatrick's four-level model (1959, 1983): level-one evaluation is equivalent to assessment of the input for the HRD subsystem (instructor, material, equip- ment, and so forth), level two for the process of the HRD subsystem (bow mucb learning takes place during a learning intervention process), and level three eval- uates the output of tbe HRD subsystem (behavioral change as a result of an HRD intervention). In the meantime, level four measures business impact derived from tbe HRD subsystem at tbe level of tbe organization's production system.
Tbe unresolved issue with Kirkpatrick's four-level model, and the accounting ROI approach as well, is how to identify, isolate, and measure the impact gen- erated by tbe HRD subsystem on tbe output of tbe production system.
Given the multiple factors involved in the measurement process, we now describe an analytical framework for measuring ROI of an HRD intervention in a production system scenario. Assuming a production system with Y as its products or services; and considering input parameters of employees (L), capital asset (K), a predetermined organizational structure (0), and a given economic environment (p), the production function can be described as:
Y = j(L, K,0,. . .\ p; e) (equation 4) wbere (e) is a term representing measurement error.
In equation 4, an HRD intervention affects the production function through variable L (employees). Within the HRD subsystem, a functional rela- tionship between employees (L) and HRD intervention (Hrd) and learning or intervention tecbnology (A) can be described as:
L = g{Hrd, A) (equation 5) Togetber, equations 4 and 5 state tbat, to achieve a certain organization objective (Y), the system requires tbat its employees maintain a set of required Wang, Dou, Li competencies through appropriate HRD intervention (Hrd) and learning tech- nology (A), while the other inputs, sucb as capital (K) and organizational design and its structure (0), must be functionally appropriate under the given economic environment ((3). Enhancement at the output level (Y) may involve any one or a combination of several input components: enbanced employee performance (L) derived from HRD intervention (Hrd) or intervention tecb- nology (A); or cbange in organization effectiveness (0), capital investment (K), or economic environment (p). The error term (e) in equation 4 offers a quan- titative measure of the accuracy of the measurement process.
Combining equations 4 and 5, we bave:
Y=h (Hrd, A,K,O,...,^, e) (equation 6) Equation 6 defines the output variable in tbe production system as a func- tion of various inputs, including HRD subsystem and other production factors and variables.
It must be noted that equations 4 through 6 are generic in nature; the indi- vidual parameters may represent an array of variables in a specific business setting or HRD subsystem. The distinction between tbe system construction in equation 6 and the judgment-based approaches we discussed earlier is tbat many organizational variables, such as capital investment, number of employ- ees, investment in HRD, and otber objective variables, enter into tbe frame- work to support an ROI measurement tbat is objective in contrast to subjectively assigning dollar values. Furtbermore, various statistical testing tools can be effectively used to test wbetber a selected variable belongs to tbe specific system function identified.
Operationalization of the Framework Tbe system functions represented by equations 4 tbrough 6 can be opera- tionalized through quantitative procedures, including both linear and nonlin- ear analyses. Tbe dependent variable Y measures the bottom-line results on wbicb an HRD program bas an impact. It may or may not be in monetary units and can be continuous or discrete, depending on tbe measures cbosen. Con- tinuous data may be total revenue or sales, profitability, employee turnover rate, cost, or time variables (tbese are bard data in an organization). Discrete data may comprise variables sucb as rating for job satisfaction, productivity index, customer satisfaction or complaints, or any variable measuring business results in tbe Likert rating scbeme in an organization (these data may appear subjective but bave relative stability).
For the variables on the right side of the equations, investment or cost data for the HRD program can be included, togetber with other identified objective variables sucb as individual variables affecting tbe dependent variable (Y). Var- ious data on HRD investment or cost, learning tecbnology data or index, and A Systems Approach to Measuring ROI bours spent in a particular program, may be used for tbe terms on tbe rigbt- band side of equation 6. To make the approach amenable to organization real- ity, an organizational survey (sucb as 360-degree feedback or management competency assessment) or a culture or opinion survey may also be combined with the objective data. However, tbe results and interpretations must be sub- ject to statistical tests.
Econometric or statistical model specification processes are available to identify underlying functional form and estimate tbe coefficients of tbe vari- ables in tbese equations (Judge and otbers, 1988). Tbe functional form may be linear, log linear, quadratic, or nonlinear, depending on specific system rela- tionsbips. For example, Douglas (1934) used a log transformation to model a nonhnear function in economics research. The resulting function, known as the Cobb-Douglas production function, identified tbe contributions of indi- vidual input to tbe growtb of tbe U.S. economy.
Witb identified functional form and estimated coefficients using appro- priate quantitative processes (for example, multiple regression), it is now possible to measure tbe net impact of an HRD program on output variable Y with other relevant variables taken into account. For continuous data, the impact of HRD intervention on the system can be identified through the par- tial regression coefficients (Gujarati, 1988; Kacbigan, 1986; Neter, Wasserman, and Kutner, 1990; Nicbolson, 1990), namely, taking tbe partial derivative of tbe output variable (Y) with respect to tbe HRD variables (Hrd), wbile account- ing for tbe influence of all otber systems variables. For equation (6), assuming HRD programs bave no impact on environmental variable (P)—tbat is, OY/ap)OP/aHrd) = 0—we bave:
JY__dhJl^dh_d^d}i_dK_dh__dO_ (equation 7) dHrd " dL dHrd dA dHrd dK dHrd dO dHrd wbere dY/dHrd measures tbe cbange in the mean value of the output variable (Y), per unit change in HRD investment (Hrd), accounting for the influence of all other variables. In otber words, dY/dHrd is the "net" effect of a unit cbange in Hrd on the mean value of Y, net A, K, and 0. In the case of measuring ROI for discrete data on variable Y, models witb limited dependent variables can be constructed: linear probability models, logistic models, and probit modeis (Maddala, 1983). Interpretation of these models is similar to tbose for contin- uous variables.
Tbe ROI approach under tbis framework can be used to measure tbe net effect of HRD variables tbat make a contribution to overall business outcomes.
Examples of sucb HRD variables are investment in e-learning, instructor-led training (including technical, soft skills, or management development), invest- ment in quality initiatives, or organization development. Many economists have demonstrated tbe feasibility of estimating tbe impact of individual vari- ables on tbe basis of a similar analytical framework. Mincer (1994) used a Wang, Dou, Li semilog wage function and estimated the impact of training-related variables on tbe participant's posttraining wage cbange. Parsons (1990) used a mathe- matical derivation to investigate general training investment decisions and training market efficiencies. Using a logistic model, Wang (1997) measured employer training decisions on the basis of employee characteristics in tbe U.S.
private sector.
Application Scenarios To illustrate tbe need for, and use of, tbe systems approacb, we present two hypothetical scenarios that may be familiar to many HRD professionals. Tbe first scenario is to measure monetary return witb sales revenue as a dependent variable. Tbe second scenario is to measure nonmonetary return—employee job satisfaction—that is due to HRD intervention. For simplicity, we assume the functional relationships are linear, though any nonlinear functions will not change the operation and interpretation ofthe scenario.^ Scenario One: Training Impact on Sales Volume. Suppose company A intends to measure the contribution of customer service training and sales training to its sales volume growth. Applying the systems approach, we first identify tbe production system, expressed as:
Y, =/(Cr,_,, Ns,_,, Ts,_j, Adv,_j; p) (equation 8) wbere Y, is a vector of output or sales volume for time period t, Cr,_j is a vector of customer accounts in time period t-1, which is prior to period t; Ns,_j is a vector of the business development representatives in period t-1; Ts,_j is a vector of dollar value invested in sales training in period t-1; Adv,_j represents tbe company's expenditure in advertisement in t-1; and P is a market condi- tion during time period t that is out of the company's control but needs to be considered in tbe system. The time lag treatment (t, t-1) acknowledges that tbe impact of training is not instant but takes a certain time to be effective.
Notice tbat Cr,_j is an output of a subsystem, expressed as Cr,-j = g (Nc,_2, Tc,_2, Cn,_2, • . .) (equation 9) wbere Nc,_2 is a vector of customer service representatives, Tc,_2 is a vector of tbe investment in customer service training, and Cn,_2 is a vector of customer complaints. (There migbt be other variables affecting tbe outcome that should be included in the subsystem under otber circumstances or for anotber organization.) Combining equations 8 and 9, we bave:
Y, = h (Nc,_2, Tc,_2, Cn,_2, Ns,_j, Ts,_,, Adv,_j; p) (equation 10) A Systems Approach to Measuring ROI 217 Fquation 10 states tbat a change in sales volume is a function of all tbe variables specified on tbe right side. With tbe system functions established, quarterly data may be collected for eacb variable in tbe given time periods.'' Conceivably, sales volume Y and customer complaints Cn will move in oppos- ing directions, meaning, increasing customer complaints may be associated with decreasing customer accounts and less sales, implying that the coefficient for Cn would be a negative value. Sucb knowledge of the system is useful in confirming the analysis results.
With quarterly data randomly generated for this exercise, equation 10 resulted in tbis form (t ratio in parentheses):
Y = 49023.43 + 12524.3 Nc + 19.3 Tc - 125.4 Cn + 26147.3 Ns + 25.5 Ts -I- 37.4 Adv (19.6) (5.7) (4.8) (-2.4) (4.3) (3.6) (4.1) R2 = 0.81 d = 1.91 With all the coefficients being significant and bearing tbe expected posi- tive or negative signs, tbe two variables sales training (Ts) and customer service training (Tc) are now tbe central interest. Taking the partial derivative of Y with respect to Tc and Ts, we have:
= $19.3 and -^ = $25.5 die The results indicate that, accounting for tbe influence of all otber variables specified, tbe direct effect on the sales volume is $19.30 for each dollar invested in customer service, and $25.50 for each dollar spent in sales training.
As demonstrated, tbis systems approacb uses actual data available in an organization and requires no subjective judgment or guesswork. Tbe approacb may also be used for forecasting future training ROI if appropriate data are identified for the relevant variables. It should be noted that this exercise is intended to demonstrate the applicability of tbe systems approacb; tecbniques on time series data treatment and analysis are beyond our scope bere. Also, HRD practitioners may furtber explore why the two types of training invest- ment resulted in differing return by examining and comparing tbe content, delivery, or on-tbe-job support of the two intervention types so that informa- tion for program improvement can be obtained.
Scenario Two:
Training Impact on Employee Satisfaction. Suppose com- pany B attempts to determine tbe impact of a series of HRD programs on employee job satisfaction, a nonmonetary economic return. Tbis measurement can be conducted using cross-sectional data obtained tbrougb employee sur- veys.
A similar system equation can be constructed:
Y =/(xj, X2, X3, X4, . . .) (equation 11) wbere Yis a vector of employee perceived ratings on job satisfaction, Xj is a vector of training hours (or dollar value) per employee received from technical Wang, Dou, Li training in a given period, X2 is a vector of training hours (or dollar value) in soft skill training, Xj is a vector of management or leadership development pro- grams, and X4 is a vector of employee perceived ratings on organization effec- tiveness. Since tbe scenario is only relevant to the HRD subsystem, related data can be obtained tbrougb training records and employee surveys. The objective nature of the system approach is not jeopardized by including a seemingly sub- jective variable, X4, because tbe otber variables are based on objective data, and more important, statistical procedures enable one to gauge the relevance and accuracy of the functional form.
With a regression procedure applied to randomly generated cross-sectional data, we obtained these results (for consistent measurement units—training time in ten hours, job satisfaction and organization effectiveness ratings from 0 to 9) with t ratio in parentheses:
Y = 1.35 + 0.62xj + 0.71x2 + 0.76x3 + 0.82x4 (20.1) (5.2) (6.1) (4.4) (3.8) R^ = 0.76 As witb scenario one, tbe results can be interpreted to sbow tbat tbe net effect of every unit of technical training (every ten hours) increases the employee job satisfaction rating by .62 points, with all other variables accounted for. Likevidse, tbe employee job satisfaction rating increases by 0.82 points if the perceived organizational effectiveness rating increases 1.00 point.
Other coefficients can be interpreted similarly It is also obvious tbat training time may be replaced witb investment in dollar value if one cbooses to do so.
Discussion The two scenarios have demonstrated the applicability of tbe systems approach constructed. We now extend our discussion to the following issues.
Some Advantages ofthe Systems Approach. The systems approach dis- cussed in this study is not to equate accounting for variance with causahty Rather, it demonstrates use of a systems framework for establishing quantita- tive relationships between business production outcome and all relevant vari- ables tbat conventional ROI accounting equations do not encompass. Unlike current usage of accounting ROI equations, the systems approacb largely depends on data tbat are objective and obtainable from organizational data- bases, or in some cases provided by respondents wbo are competent to make estimates on the basis of job-related experience (job satisfaction rating). Sub- jectivity is largely avoided and guesstimating is minimized.
Also, as demonstrated in scenario two, nonmonetary benefit derived from HRD intervention can be now evaluated tbrough tbe systems approach, which would be otberwise difficult to quantify by the conventional ROI accounting approach. More important, through the systems approach it is now possible to A Systems Approach to Measuring ROI 219 objectively separate HRD tmpact from non-HRD variables witb statistical confidence so tbat tbe "true" HRD program contributions can be identified. As demonstrated in tbe two scenarios, this is done by imposing tbe results trom tbe systems approacb on a partial derivative operation with respect to eacb sys- tems variable. Furtbermore, tbe systems approacb can be used to forecast ROI for future HRD investment on tbe basis of an organization's past HRD invest- ment practice. In short, tbe systems approacb can be used to measure the ROI impact of an HRD program witb known statistical precision.
Comparison with UA Models. Tbe systems approacb presented m tbis study for ROI measurement in tbe HRD field differs substantially from UA models especially tbe RBN model proposed by Raju, Burke, and Normand (1990) in tbe 1/0 arena. Although botb models involve regression analysis, tbere are tbree fundamental differences. First, tbe systems approacb takes into account botb tbe HRD subsystem and the overall production system, while tbe RBN approach (as well as tbe BCG model) by construction does not require a multisystem framework because it operates in a case-by-case settmg. Second, tbe systems approacb can measure combined business impact for multiple HRD programs, whereas tbe RBN model measures only an individual HR pro- gram Tbird tbe UA model is developed explicitly as a linear form (Raju, Burke and Normand, 1990); this systems approach may take many functional forms, as production functions and tbe HRD subsystem functions vary signit- icantly with organizations.
Limitations and Future Research Directions. Organizations sometimes need to conduct ROI measurement of a single program intervention, such as a leadersbip development program. The systems approacb may be inadequate for sucb a measurement. Tbis is because a sbort-term program (one-day team- building training event), wben examined in a larger system, may bave neghgi- ble impact on overall business outcomes. Anotber limitation, also a cballenge to tbe HRD practitioner, is tbat the systems approach requires certain skills m system thinking and quantitative analysis. HRD as a profession is notoriously lacking in quantitative approacbes, as Pbilhps lamented (1994, p. 20)—"tbe concept of statistical power is largely ignored" in existing ROI measurement, and tbis problem "does not receive enough attention witb practitioners. Addi- tionally, tbe fact tbat some organizations are systemically poor at capturing HRD and otber operational data, or are reluctant to sbare tbeir numerics with internal or external groups, may pose some constraint in applying tbis approacb Still some individuals in an organization may cboose not to obtain or acknowledge tbe true value of HRD intervention from various concerns (organization politics, the turf issue, and so on).
Furtber empirical studies are needed to apply tbe systems approacb m real-world HRD evaluation and measurement practices. Sucb "field" studies would not only demonstrate tbe advantages of the systems approacb but also belp HRD researcbers and practitioners better understand the nature. 220 Wang, Dou, Li processes, and functions ofthe HRD subsystem, its relationsbip witb the pro- duction system, and the evaluation and measurement processes Moreover researcb on practical methods that can be effectively used to measure ROI for individual HRD programs is needed. Such methods may be desirable to many held practitioners and appeal to top management in many organizations To address this issue, Wang (2002) has proposed a separate framework in a recent study Conclusion , Built on relevant studies in economics, financial accounting, and industrial- organizational psychology, along with conventional HRD approaches for train- mg evaluation and ROI measurements, a systems approacb is proposed in this study to measure ROI for performance improvement intervention in the HRD held. We acknowledge the fact that business outcome is often the result of mul- tiple input variables (including HRD interventions and non-HRD factors)- the system approach treats HRD interventions as a subsystem within the frame- work of the entire production system.
The impact of the HRD subsystem on business outcomes is tben examined and evaluated witb full consideration of the production system. Through appropriate mathematical operations the fair share of the HRD intervention contributing to the business outcome can be Identified and isolated. Tbis system approach is the means for comprehensive ROI measurement of HRD intervention; judgment-based subjective measure- ment can be avoided substantially Also, in contrast to the conventional accounting-type approach, which is incapable of evaluating nonmonetarv ROI the system approach is equally applicable to both monetary and nonmonetar^ ana yses and thus is useful for various types of organizations (private or pub- lic, for-profit or nonprofit). Two application scenarios are presented to dernon- strate tbe applicability of tbe system approacb, one measuring tbe dollar-value contnbution to sales volume from HRD intervention, the other measuring the degree of contribution to employee job satisfaction derived from relevant train- mg activity.
Notes 1.
We use ROI fi' IH '^'°l^^°f **= P^P" ^hen referring to studies in both economics and the ^^'^^ *7gh the phrase was never used by economists in thei ltd k PP en referring to studies in both economics and 2 S f 7g the phrase was never used by economists in their related work T 7 °™ ^"""'"' '° "''° practitio hb f 7 y sts in their related work T 7 °™ ^"""'"' '° "''° practitioners, on-the-job training here and in dml d r ^':°"°7'^ 1"-^^-^. ^^fe- to any trammg sponsored by the'comp;ny Fo" detailed discussion of the terminology differences in the two fields, see Wang (1997) 3.
The same issues were addressed in a recent ASTD study (Koehle 2000) 4.
DuPont first developed the ROI measurement for financial'control of decentralized busmess activities in 1919.
The formula is expressed as: uecentranzea (net income/sales) X (sales/total assets) = net income/total assets A Systems Approach to Measuring ROI It was intended to measure the combined effects of profit margin and total assets turnover.
5.
Because of the similar numerical terms involved in the ROI and B/C equations, HRD practitioners deem the two equations equivalent, although this may not be the case in financial accounting terms.
6. For concern about the consequences of nonnormal distribution in the data analysis process. Judge and others (1988) offer thorough discussion.
7.
It is also possible to combine perception data with so-called hard data of survey instruments in this case. However, using such data requires developing multiple surveys at different points of time for the same subject, to avoid same-method constraints.
References AUbee, K. E., & Sempie, C. A. (1980). Aircrew training device lije-cyde cost and worth oJownership.
(CIIG TR-3041E). Westlake Village, CA: Canyon Research Group.
Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later.
Personnel Psychology, 42 (2), 331-342.
Bakken, D., & Bernstein, A. L. (1982). A systematic approach to evaluation. Training and Development Journal, 36 (8), 44-51.
Barron, J. M., Berger, M. C, & Black, D. D. (1993). Do workers pay for on-the-job training?
(Work- ing paper no. E-169-93). Lexington: University of Kentucky Barron, J. M., Berger, M. C, & Black, D. D. (1997). How well do we measure training? Journal of Labor Economics, 15, 507-528.
Becker, G. (1962). Investment in human capital: A theoretical analysis.
Journal o/Political Economy, 70 (suppl. 5), part 2, S9-S49.
Becker, G. (1964). Human capital: A theoretical and empirical analysis, with special reference to education. New York: Columbia University Press.
Ben-Porath, Y.
(1967). The production of human capital and the life cycle of earnings. Journal oj Political Economy, 75, 352-365.
Bertaianffy, L. (1968).
General system theory:
Foundations, development, and applications.
New York:
Braziller.
Bishop, J. (1991). On the job training of new hires. In D. Stern &J. M. Rizen (Eds.), Market failure in training?
New economic analysis and evidence on training oj adult employees.
New York:
Springer-Verlag.
Bobko, P, cSt Russell, C. (1991). A review of the role of taxonomies in human resources management. Human Resource Management Review, 4, 293-316.
Boudreau, J. W. (1991). Utility analysis for decisions in human resource management. In M. D.
Dunnette & L.
M. Hough (Eds.), Handbook of Industrial and Organizational Psychology (2nd ed., vol.
2, pp. 621-745). Palo Alto, CA: Consulting Psychologists Press.
Brandenburg, D. (1982). Training evaluation: What's the current status?
Training and Development Journal, 36, 14-19.
Brogden, H. E. (1946). On the interpretation of the correlation coefficient as a measure ot predictive efficiency Journal of Educational Psychology, 37, 64-76.
Brogden, H. E. (1949). When testing pays off.
Personnel Psychologji, 2, 171-183.
Brogden, H. E., & Taylor, E. K. (1950). The dollar criterion: Applying the cost accounting con- cept to criterion construction. Personnel Psychology, 3, 133-154.
Browning, R. E, Ryan, L. E., Scott, P G., & Smode, A. E (1977).
Training effectiveness evaluation of device 2F 87E, P-3C operational flight trainer (TAEG report no. 42.). Orlando: Training Analy- sis and Evaluation Group.
Cascio, W E (1992). Assessing the utility of selection decisions: Theoretical and practical considerations. In N. Schmitt & W C. Borman (Eds.), Personnel selection in organizations.
San Francisco: Jossey Bass. Wang, Dou, Li Cascio, W. E (2000). Costing human resources:
The financial impact of behavior in ormnizations (4th ed.). Cincinnati: South-Western.
Cronbach, O. L., & Gleser, G. G. (1965).
Psychological tests and personnel decisions. (2nd ed ) Urbana: University of Illinois Press.
Dearden, J. (1969). The case against ROI control. Harvard Business Review, 47 124-134 Dickinson, K. P.Johnson, T. R., & West, R. W (1987). An analysis of the sensitivity of quasi- expenmental net impact estimates of CETA programs. Evaluation Review, 11 452-472 Doppelt, J. E., & Bennett, G. K. (1953). Reducing the cost of training sat'isfactory workers by using tests.
Personnel Psychology, 6, 1-8.
Douglas, P H. (1934). The theory of wages. New York: Crowell-Collier & Macmillan Ehrbar, A. (1998). EVA: The real key to creating wealth. New York- Wiley Ehrenberg, R., & Smith, R. (1994). Modern labor economics:
theory and public policy.
(5th ed ) New York: HarperCollins.
Eeuer, M., Glick, H., & Desai, A. (1987). Is firm-sponsored education viable? Journal of Economic Behavior and Organization, 8, 121-136.
Eitz-enz, J. (1984). How to measure human resources management. New York- McGraw-Hill Eraker, T., & Maynard, R. (1987). The adequacy of comparison group designs for evaluations of employment-related programs. Joumdo/Human Resources 22 194-227 Eriedlander, D & Robins, P (1991). Estimating the effects of employment and training programs- Assessment of some nonexperimental techniques.
New York: Manpower Demonstration Research Eusch, G. E. (2001). What happens when the ROI model does not fit? Performance Improvement Quarterly, 14:4, 60-76.
Geroy, G., & Swanson, R. (1984). Forecasting training costs and benefits in industry Epsilon Pi Tau, 10, 15-19. '^ Gilbert, T (1978). Human competence:
Engineering worthy performance. New York- McGraw-Hill Gujarati, D. N. (1988).
Basic econometrics.
(2nd ed.). New York: McGraw-Hill.
Heckman, J.J. (1979). Sample selection bias as a specification error. Econometrica 47 286-293 Heckman, J. J., & Hotz, V J. (1989). Choosing among alternative nonexperimental methods for estimating the impact of social programs: The case of manpower training. Joumal of American Statistical Association, 84 (408), 862-874.
Heckman, I J., Hotz, V J., & Dabos, M. (1987). Do we need experimental data to evaluate the impact oi manpower training on earnings? Evaluation Review, 11, 395-427.
Holton, E. E (1996). The flawed four-level evaluation model Human Resource Development Quarterly, 7, 5-21. '^ Hotz, V J. (1992). Designing and evaluation of the Job Training Partnership Act. In C. E Manski & I. Garfinkei (Eds.), Evaluating welfare and training programs. Cambridge MA- Harvard University Press.
Jacobs, R.
L., Jones, M.
J., & Neil, S. (1992). A case study in forecasting the financial benefits of unstructured and structured on-the-job training. Human Resource Development Quarterly, 3, Judge, G., Hill, R., Griffiths, W, Lutkepohl, H., & Lee, T (1988). Introduction to the theory and practice of econometrics.
(2nd ed.). New York: Wiley Kachigan, S. K. (1986). Statistical analysis: An interdisciplinary introduction to univariate and multivariate methods.
New York: Radius Press.
Kaufman, B. E. (1994). The economics of labor markets. (4th ed.). Eort Worth- Dryden Press Kaufman, R., & Keller, J. M. (1994). Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly, 5, 371-380.
Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal of the American Society for Training and Development, 13, 3-9.
Kirkpatrick, D. (1967). Evaluation of training. In R. Craig & L. Bittel (Eds ) Trainim and development handbook.
New York: McGraw-Hill.
Kirkpatrick, D. (1977). Evaluating training programs: Evidence vs.
proof.
Training & Development Journal, 31, 9-12. '^ A Systems Approach to Measuring ROI 223 Kirkpatrick, D.
(1983). Four steps to measuring training effectiveness.
Personnel Administrator, 28, 19-25.
Kirkpatrick, D.
(1998). Evaluating training programs:
The four levels.
(2nd ed.). San Francisco:
Berrett-Koehier.
Koehle, D. A.
(2000). Investing in workforce training improves financial success. Training & Development, 54, 73-74.
Koontz, H., O'Donnell, C, & Weihrich, H.
(1984). Management.
(8th ed.). New York:
McGraw-Hill.
LaLonde, R.
(1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76, 604-620.
LaLonde, R., & Maynard, R. A.
(1987).
How precise are evaluations of employment and training programs? Evidence from a field experiment. Evaluation Review, 11, 428-451.
Latham, G. E, & Whyte, G.
(1994).
The futility of utility analysis.
Personnel psychology, 47, 31-46.
Law, K. S., & Myors, B. (1999).
A modification of Raju, Burke, and Normand's (1990) new model for utility analysis. Asia Padficjoumal of Human Resources 37, 39-51.
Maddala, G. S.
(1983). Limited-dependent and qualitative variables in econometrics.
New York:
Gambridge University Press.
Mathieu, J. E., & Leonard, R.
L., Jr. (1987). Applying utility concepts to a training program in supervisory skills:
A time-based approach. Academy of Management Journal, 30, 316-335.
Maynard, R.
(1994). Methods for evaluating employment and training programs: Lessons from the U.S. experience.
In R.
McNabb & K.
Whitfield (Eds.), The market for training:
International perspectives on theory, methodology and policy Brookfield, VT: Ashgate.
McLagan, E (1989). The models.
Alexandria, VA:
American Society for Training and Development.
McLinden, D., & Trochim, W (1998). Getting to parallel: Assessing the return on expectations of training. Performance Improvement Journal, 37, 21-25.
Mincer, J. (1962). On-the-job training: Gosts, returns, and some implications. Journal o/Political Economy, 70, part 2, S50-S79.
Mincer, J.
(1994). Investment in U.S.
education and training: Levels and changes since the 1960s.
(Working paper no.
4844.). Gambridge, MA: National Bureau of Economic Research.
Moffitt, R.
(1991). Program evaluation with nonexperimental data. Evaluation Review, 15, 291-314.
Myers, R.
(1996). Metric wars.
CEO, 12, 41-50.
Naylor, J.
G., & Shine, L. G.
(1965).
A table for determining the increase in mean criterion score obtained by using a selection device. Journal of Industrial PsycholoQi, 3, 33-42.
Neter, J., Wasserman, W, & Kutner, M. H.
(1990). Applied linear statistical models: Regression, analysis of variance, and experimental designs. Boston: Irwin.
Nicholson, W (1990).
Microeconomic theory:
Basic principles and extensions.
(5th ed.). Eort Worth:
Dryden Eress.
Nikbakht, E., & Groppelli, A.
(1990).
Einance.
(2nded.). Barron's Educational Series.
O'Neill, D.
(1973). The federal government and manpower Washington, D.G.: American Enterprise Institute.
Parsons, D.
(1990).
The firm's decision to training.
In R.
Ehrenberg (Ed.), Research in Labor Economics, vol.
2.
Greenwich, GT:
JAI Press.
Perry, G.
(1975).
The impact of government manpower programs. Philadelphia: University of Pennsylvania.
Phillips, J.
(Ed.). (1994).
In action:
Measuring return on investment, vol.
1.
Alexandria, VA: ASTD.
Phillips, J.
(1997a).
Handbook of training evaluation and measurement methods.
(3rd ed.). Houston:
Gulf.' Phillips, J.
(1997b). Return on investment in training and performance improvement programs.
Houston:
Gulf.
Eritchard, R. D.
(1992). Organizational productivity In M. D.
Dunnette & L. M.
Hough (Eds.), Handbook of industrial and organizational psychology.
(2nd ed.).
Ealo Alto, GA:
Gonsulting Psychologists Press. Wang, Dou, Li Raju, N. S., Burke, M. J., & Normand, J. (1990). A new approach for utility analysis loumal of Applied Psychology, 75, 3-12.
Rothwell, W, & Sredl, H. (1992). The ASTD reference guide to pwjessional human resource development roles and competencies.
(2nd ed.). Amherst, MA: HRD Press.
Schmidt, E L., Hunter, J. E., & Pearlman, K. (1982). Assessing the economic impact of person- nel programs on workforce productivity.
Personnel Psychology, 35, 333-347.
Schmidt, E L., Hunter, J. E., McKenzie, R. C, & Muldrow, T. W (1979). Impact of valid selection procedures on work-force productivity Journal of Applied Psychology, 64, 609-626.
Searby, E W (1975). Return to return on investment. Harvard Business Review, 53, 113-119.
Stewart, G. B., & Stern, J. M. (1991). The quest for value:
the EVA management guide.
New York:
Harper Business.
Stigler, G. J. (1950). The development of utility theory. Journal of Political Economy 59 part 1 307-327; 59 part 2, 373-396.
Sturm, R. (1993). How do education and training affect a country's economic performance?
A literature survey Santa Monica, CA: RAND.
Swanson, R. A. (1992). Demonstrating financial benefits to clients. In H. D. Stolovitch & E. J. Keeps (Eds.), Handbook of human performance technology.
San Francisco: Jossey-Bass Swanson, R. A., & Gradous, D. B. (1988).
Forecasting financial benefits of human resource development. San Erancisco: Jossey-Bass.
Swanson, R. A., & Holton, E. E (2001). Eoundations of human resource development.
San Erancisco: Berrett-Koehier.
Taylor, H. C, & Russell, J. T (1939). The relationship of validity coefficients to the practical effectiveness of tests in selection. Journal of Applied Psychology, 23, 565-578.
Thode, W E, & Walker, R. A. (1983). A model for comparing training costs in a complex military training system. (NPRDC technical report EA 015809.). San Diego: Navy Personnel Research and Development Center.
Wang, G. (1997). Private sector training in the United States:
Behaviors and characteristics.
Unpub- lished doctoral dissertation, Pennsylvania State University, State College.
Wang, G. (2000, May). Training economics:
An alternative approach to measuring ROI for HRD programs. American Society for Training and Development (ASTD) International Conference Dallas.
Wang, G. (2002). Control group methods for HPT program evaluation and measurement.
Performance Improvement Quarterly, 15(2), forthcoming.
Wolf, M. G. (1991). Review of costing human resources: The financial impact of behavior in organizations.
Personnel Psychology, 45, 881-882.
Yan, J. (2000, Apr. 16). Personal contact. Pan American Health Organization, World Health Organization.
Greg G.
Wang is president oJPerformTek LLC, West Chester, Pennsylvania.
Zhengxia Dou is assistant professor of agriculture systems at the University of Pennsylvania.
Ning Li is an instructional designer at Bank of America, Charlotte, North Carolina.