Case Study 3: Monitoring and Controlling Project DeliverablesRead the following articles: •“Agencies Need to Improve the Implementation and Use of Earned Value Techniques to Help Manage Major System A
1 Earned Value Management at NASA: An Integrated, Lightweight Solution Peter Putz 1, David A. Maluf2, David G. Bell1, Mohana M. Gurram1, Jennifer Hsu3, Hem il N. Patel 3, Keith J. Swanson2 1Universities Space Res earch Association NASA Ames Research Center Moffett Field, CA 94035 650-604-2137 [email protected] [email protected] [email protected] 2NASA Ames Research Center Moffett Field, CA 94035 [email protected] [email protected] 3QSS Group Inc NASA Ames Research Center Moffett Field, CA 94035 [email protected] [email protected] Abstract 12— This paper describes a fresh approach to Earned Value Management (EVM) at the U.S. National Aeronautics and Space Administration (NASA). The goal of this approach is to provide a lightweight tool that allows project managers to apply earned value performance measu rem ents with minimal effo rt in term s of data entry , and without the need to learn the highly specialized jargon that mystifies many EVM solutions. The presented technical and managerial solution addresses the practical challenges of applying EVM in the messy realm of project management. An empirical case study involving five projects at the NASA Ames Research Cen ter illu strates the challenges of creating a consistent performance measurement baseline under the constraints of schedule, budget, and labor requirements, and of matching actual costs with budgeted costs on the level of granularity needed. The case study also highlights the benefits of using the implemented EVM solution in terms of data quality and time savings. The paper concludes with general recommendations for the design and application of EVM tools with the focus on ease of use. TABLE OF CONTENTS 1. INTRODUCTION ......................................................1 2. CREATING A PERFORMANCE BASELINE...............2 3. REPORTING MONTHLY PROGRESS.......................4 4. CUSTOMIZ ING EVM REPORT FORMATS..............5 5. LESSONS LEARNED ................................................6 6. CON CLUSION S.......................................................7 REFERENCES.............................................................7 BIOGRAPHY...............................................................8 1. INTRODUCTION Earned Value Management (EVM) is a management technique that allows project managers to determine the true cost and schedule variance between plan and material work accom plished of a given project at any tim e. More 1 1-4244-0525-4/07/$20.00 ©2007 IEEE. 2 IEEEAC paper #1282, Version 2, Updated December 11, 2006 importantly it allows management to predict the total costs at completion and the date of completion. A key benefit of EVM is that it serves as an early-warning system against cost overruns and schedule delays. This is the main reason why NASA and other government agencies— most prominently the Department of Defense— require projects to produce EVM reports on a regular basis (NPR 7120.5C [1]). Project managers reacted with skepticism when, in 2004, NASA’s newly founded Exploration Systems Mission Directorate (ESMD) mandated that even relatively sm all projects with a budget from 1 to 10 million dollars had to account for their project performance in the form of a highly specific monthly EVM report. They did not understand the details of the EVM method and, more severely, they were not provided with the necessary tools to integrate their schedule and financial data. After attending multiple-day- long EVM training sessions most of them ended with either creating their own set of Excel spreadsheets to manually calculate the required metrics or struggling with oversized and rigid software packages. These helped in getting the core EVM metrics, but project managers were still not able to automatically produce the customized report formats required by ESMD. For some reason the practical difficulties of a highly specialized language paired with a lack of adequate tool seem to keep haunting EVM. Fleming and Koppelman [2] complain: “What started out originally as a simple concept on the factory floor has evolved into a sort of vocational cultist confederation in which one must be specifically trained to use a foreign language in order to be a member of the team” (p. 73). Fleming and Koppelman continue on a positive note: “There is nothing difficult or complicated about the earned value concept. It does not require highly trained people to grasp the fundamentals. In fact, many people use the concept in thei r daily routines and are not even aware they are employing earned value” (p. 73). This paper describes a novel solution to creating EVM reports in a way that is as easy to understand as the earned 2 value concept itself. After ESMD issued the mandatory EVM requirement, some project managers found that the existing NASA Program Management Tool (PMT) [3], which they were using for project planning and reporting, captured already most of the financial and schedule data necessary for EVM reporting. What was missing was the report itself and some minor changes in the data input templates. Given this assessment, PMT was used to implement a full set of EVM reporting capabilities. PMT is a NASA-developed program management tool suite that supports program and project managers in all their essential activities, like creating and monitoring annual task plans, analyzing variances between budget plans and actual costs, creating periodic status reports on technical, schedule, budget, and management status, identifying program risks and tracking mitigation strategies, creating aggregated program dashboard views and other customized reports for single subprojects or the entire program. The PMT software architecture is built around two distinguishing features: (1) The main user interfa ces are sta ndard busin ess documents lik e spreadsheets, presentation slides, and text documents. The use of standard business document templates not only enables automated exchange of information between machines, but also supports natural and often ad-hoc on-line and off-line workflows between humans gathering data or making decisions. (2) The backend is a ‘sc hema-le ss’ XML 3 database, which enables easy data integration and query-based document composition. At the same time it eliminates the need for database administration by automating the integration of information that have diverse schemas. The following graphical representation illustrates the typical PMT document workflow: Drag and Drop Email Web Upload Decompose to XML User Store in XML Database Enter Context / Content Query contentontext> <É> ÉÉ> QueryresultinXML Compose from XML 3 Extensible Markup Lanaguage – http://www.w3.org/XML/ Figure 1 – PMT Document W orkflow PMT is an ideal software platform for EVM reporting for the following reasons: (1) Seamless integration of heterogeneous and distributed information: EVM reporting requires the integration of a variety of data which are partly user entered and partly retrieved from existing databases like the financial system, scheduling tools, risk management systems and so on. The XML database allows adding new data elements without the need of changing any predefined schema, in fact, without even touching the database at all. (2) Automatic composition of analyses and reports: EVM reports are often multi-page documents covering not only core EVM metrics, but also graphical representations of the work breakdown structure, the project master schedule, risk and mitigation status and financial performance. PMT has the capability to autom atically produce com prehensiv e slid e decks or multi-page spreadsheet documents. The document composition is built upon an advanced XML query module. (3) Easy communication of complex information among diverse subject matter experts and stakeholders: The gathering of financial, schedule, and various other project status data is usually an effort involving the collaboration of multiple subject matter experts. With PMT, data entry and report tem plates can be accessed, distributed, archived in a variety of ways. As standard business documents they can be down-loaded to a local desktop for off-line access, the can be distributed as email-a tta chments (e.g. to resource analysts who do not even need to aware that a particular template is a PMT input form), and they can be archived in third- party document repositories. Given those features, PMT seem ed to be an ideal software platform for the im plem enta tion of an Earned Value Management system. The following sections discuss in detail, how users can setup and use PMT for EVM reporting, including empirical results on the system use in practice. 2. CREATING A PERFORMANCE BASELINE A consistent project baseline is an indispensable prerequisite for any kind of project performance assessment. For Earned Value Management a performance baseline has to include a work breakdown structure (WBS) for the entire project; a master schedule of work packages with budgeted costs for each work package; and a time-phased budget plan for each WBS element. 3 Figure 2 – Building a Performance Baseline Creating a work breakdown structure The work breakdown structure defines the scope of a given project. A WBS is a hierarchical structure of sub-projects, sub-tasks, or sub-products. Graphical representations of a work breakdown structure look very much like an organizational chart, even though its elements are not organizational units like groups, divisions, directorates etc., but sub-tasks. In PMT the WBS is entered into a configuration spreadsheet in form of a flat list. Figure 3 – Defining a W BS in PMT Figure 3 shows the data elements for the WBS structure. Noteworth y are the columns “Level”, “PMT WBS #”, and “BW WBS #”. “Level” indicates the hierarchical positio n of a particular WBS elem ent, “PMT WBS #” is a string of characters chosen by a project manager to systematically nam e the WBS elem ents, and “BW WBS #” links each WBS element to NASA’s financial system. Fortunately the accounts in NASA’s financial system— or Business Warehouse (BW)— are structured as a work breakdown structure including all current space missions and programs. In most instances this makes it very easy to link a PMT WBS elem ent to an account in the financial system . We will describe in section 4 below how this connection between PMT and the financial system enables an autom atic calculation of actual costs for each WBS elem ent. The users upload the configuration spreadsheet to the PMT server and the system autom atically produces three data entry spreadsheets per WBS element: (1) a Taskplan for entering schedule and budget information; (2) a Monthly Report for entering the progress accomplished on each work package; and (3) a Budget Report for analyzing cost plans versus actual costs and for entering subjective cost estimates. Entering schedule information Schedule information for work packages—which are called deliverables in PMT— is entered into the Taskplan spreadsheets. Breaking down the total project schedule into work pack ages per WBS elem ent facilitates the distrib uted collaboration between sub-project managers who are accountable for their on-time completion. Figure 4 – Scheduling and Costing W ork Packages Entering time-phased budget information The Taskplan includes a spreadsheet called “Budget Section” for entering the time-phased budget plan (or phasing plan). This again is done on the level of each WBS element. The phasing plan section is custom designed to match NASA’s accounting system. A phasing plan is typically: (1) broken down into full cost elements (civil service labor, civil service travel, procurement, etc.) (2) entered for multip le fiscal years (3) broken down into the various NASA Centers involved in the sub-project. 4 In the EVM jargon the phasing plan total is called “planned value” or “Budgeted Costs for Work Scheduled (BCWS)”. Validating the performance baseline In the context of Earned Value Management it is crucial that the schedule information and the time-phased budget information are integrated into a consistent performance plan. The following graphic shows the connection between schedule and budget data. In the (messy) reality of project management it is not a trivial task to create a consistent performance baseline. Dependent on the particular project environment the concrete steps for this planning process may vary substantially. Some projects might start by planning in terms of work hours, others in terms of dollar-values; some projects might optimize their schedule around predefined delivery deadlines and assign the required workforce afterwards, others might optimize the schedule for a given lev el of wo rkforce utilizatio n and move the wo rk pack ages around to accommodate that. Many other variants are feasible, too. Figure 5 – Creating a Consistent Performance Baseline Given the variety of project planning methods, we found it not helpful to provide the project managers with a one-size- fits-all planning tool within PMT. Rather, we left it to the individual project managers to use tools of their own choice. However, once the final schedule and phasing plan information is entered, PMT provides a baseline validation tool. The baseline validation tool is a spreadsheet that calculates the Schedule Performance Index (SPI) for all WBS elements of a given work breakdown structure under the assu mptio n of the follo wing best-c ase-sc enario : (1) all work packages are fin ish ed on tim e; (2) for all periods, the actual costs are equal to the planned costs. Under these assumptions the SPI and the Cost Performance Index (CPI) are always 1.00 for consistent performance baselines4. However, if schedule and budget information do not match up, the SPI is either bigger or smaller then 1.00. In such a case a project manager needs to go back and revise the plan. Section 5 below will describe our lessons learned this regard. 3. REPORTING M ONTHLY PROGRESS Earned Value Management is nothing more than comparing for a given point in time the planned value (BCWS) with two other numbers: the earned value (BCWP) and the actual costs (ACW P)5. With PMT the actu al costs are loaded autom atically from NASA’s financial system (see above) and do not require any manual data input into PMT. Figure 6 – Entering Percentage Complete The Earned Value (BCW P) is a dollar value representing the material progress made on a particular work package. 4 In fact, under assumption (2)—budgeted value equals actual cost—CPI and SPI are identical. 5 BCWS stands for Budgeted Cost for Work Scheduled, BCWP for Budgeted Cost for Work Performed, and ACWP for Actual Cost for Work Performed. 5 For each given WBS element it is the sum of the budgeted costs per work pack age multip lied by the percen tag e complete. As pointed out above, in PMT the budgeted cost per work package is entered into the schedule section of the Taskplan. Hence the only missing data point is the percentage complete per work package. This is done in a spreadsheet entitled “Monthly Report”. Figure 6 shows that the individual work packages are grap hically depicted as tim elin es. The percentages com plete are updated once a month by the project manager and entered as numbers directly under the timeline for a given work package. In the same graphical display start and end dates for milestones can be moved if necessary. No te that after the perfo rm ance baseline is created, at a minimum all a project manager needs to do for EVM reporting is to access the Monthly Report and update the percen tag e complete! In the case that the final EVM report contains additional information besides the core EVM metrics it can also be entered into the “Monthly Report” spreadsheet. E.g., in the EVM report for NASA’s Exploration Systems Mission Directorate required entering project risks. 4. CUSTOMIZING EVM REPORT FORMATS Figure 7 shows how PMT integrates the performance baseline data and the monthly progress data in order to calculate the core EVM metrics. 6 Figure 7 – Calculation of Core EVM metrics The system is built such that an EVM report can be run against any individual WBS element. By default PMT calcu lates the core EVM metrics for the selected WBS element (e.g. the level 1 project) and for the next level down. This level is often called Control Account Plan (CAP), which is simply the management control point where the performance measurement has to take place. PMT dynamically rolls up all lower level schedule, budget and cost inform atio n to the CAPs. Ma ny organizatio ns follo w widely accepted standard formats like Contract Performance Report, which was originally mandated for Department of Defense (DoD) acquisition contracts. However, in some other cases, executive management wants to see the EVM metrics in the broader project context including financial reports, risk reports and other status updates. This is a challenge for project managers since it requires time consuming and error-prone manual data manipulation. For this reason PMT was designed to support the production of highly customized EVM reports. The XML database and the XML query protocol used in PMT enable the composition of customized reports with a minimum of software coding efforts. 5. LESSONS LEARNED The EVM reporting capabilities were im plem ented in 2004 when NASA’s newly-founded Exploration Systems Mission Directorate mandated earned value project management even for relatively small sized research projects. The software development was done in close cooperation with five ESMD project managers who provided valuable feedback and requirem ents for a system that is easy to use in a NASA environment. Inconsisten t Baselin es It turned out that challenge number one was the formulation of a consistent perform ance baseline. In a sm all case study we analyzed the originally submitted baselines that were produced with a variety of ad-hoc tools— mostly self- created spreadsheets. We applied a best-case scenar io— assu ming that a) all work packages were delivered exactly on schedule and that b) the actual costs equal the budgeted costs in any point in time— and calculated the predicted cost and schedule indexes. For a consistent baseline SPI and CPI necessarily need to be 1.00 for all months. However, if the baseline is inconsistent, that is, the schedule for earning value and the time-phased budget do not match— SPI and CPI would be either higher or lower then 1.00. In accordance with the ESMD EVM format we coded the SPI/CPI numbers with green (0.9 ! index ! 1.1), yellow (1.1 ! index ! 1.2 or 0.8 ! index ! 0.9), and red (index > 1.2 or index < 0.8) color, indicating the amount of the variance from the performance baselin e. The follo wing table shows the results: Figure 8 – Case Study Results The results indicate that the perform ance baselines were highly inconsistent. Even under the best case scenario of the project being on budget and on schedule at all times: (1) not a single project would achieve green each month, meaning not a single project had a consistent baseline (2) 31.7% of all reported months were in the red or yellow (3) one particular performance baseline had even 58% of reported months in the red or yellow at project level. These results proved the urgent need to implement a performance validation tool that project managers could use before they submitted the final plan. Need for Subjective Cost Estimates When we showed our EVM pilot implementation to the NASA project managers, their immediate question was: Where do you get the actual costs from ? When they learned that we im ported the costs directly fro m NASA’s fin ancial system, they responded: “Then we can’t use it”. As it turned out the project managers had good reasons for their objection. In fact, under certain circumstances the financial data out of the accounting system are problematic for performance measurement. For example when contractors complete a job it takes up to a few weeks until the contractor receipt is received, reviewed, paid and entered into the accounting system . Other exam ples are arbitrarily timed assessments of organizational overhead costs or costs charged to the wrong WBS element within a project. In those cases the accounting system is not “wrong” but the time lag in the accounting data is too great for someone who wants to do project performance measurement. Therefore, 7 project managers need a way to “adjust” the accounting data. With PMT we took the approach to allow project managers to enter 'estim ated costs' replacing the system of record data. However, for reasons of data transparency it is essential that the EVM system keeps the numbers of the accounting system and the ‘subjective’ cost estim ates logically and visually strictly separated. Figure 9 – Entering Estimated Costs The figure above shows the PMT Budget Report. The actual costs out of NASA’s accounting system are depicted as thick vertical bars. The subjective cost estim ates are added as thin vertical lines on top of the bars. 6. CONCLUSIONS The pilot implementation of the described EVM solution at the NASA Am es Research Center was highly successful in terms of time savings and in terms of reliability and consistency of reports across very different projects. Before project managers started to use the PMT EVM module it took them between one and two weeks to gather the relevant schedule and cost data from all the sub-projects, to manually calculate the EVM metrics, to create the required charts, and to integrate everything into the final slide deck. With PMT this time span came down to one to two days and instead of data gathering and manual data manipulation the project managers could focus on variance analysis and on creatin g actio ns where necessa ry. We feel that our experiences with the presented approach are encouraging. The positive results are based on the following factors, which can be seen as general recommendations for EVM tools: (1) Use standard business documents as the main user interface. Project managers live and breathe spreadsheets and slid e sets. A tool that uses standard business docum ents significantly increases user acceptance, reduces the need for training, and allows for complex off-line workflows. (2) Provide the capability to automatically produce custo m report form ats. Senior management wants to see high-level project status information in a custom ized, easy to read form at. Since those sta ndards change frequently an EVM tool should provide the flex ibility to use any output form at req uired . (3) Minimize the need for manual data entry. Project managers do not lik e to enter inform atio n twice. A seamless integration with existent financial and scheduling systems is highly recommended. (4) Provide a baseline validation tool. There are tw o ways to support project managers in creating a performance measurem ent baseline. One is to provide a set of predefined input templates that produce automatically a consistent baseline. The other way is to have a baseline validation tool. In a highly heterogeneous project environment it might be more successful to leave it to the project managers to choose their own tools and processes for creating a performance baselin e. Ho wev er, to ensure that the performance baseline is self-consistent, a baseline validation tool has proved highly valuable. (5) Provide a means for entering “cost estimates”. As shown above corporate accounting systems do not provide the right costs information for performance reporting in all cases. It is frustrating for project managers to explain performance variances that are merely undesired accounting artifacts. One approach to avoid that is to “allow” project managers to enter “cost estim ates” based on their accurate knowledge of project activities.REFERENCES [1] NASA Program and Project Management Processes and Requirements NPR 7120.5C, Web site: http://www.hq.nasa.gov/office/codeq/doctree/71205.htm [2] W. Quentin Fleming and Joel M. Koppelman, “The Essence and Evolution of Earned Value,” Transactions of AACE International, 73–79, 1994. [3] Bell, David G., et al. “The NASA Program Management Tool: A New Vision in Business Intelligence,” 2006 IEEE Aerospace Conference Proceedings, March 4-11, 2006. 8 BIOGRAPHY Peter Putz is a management scientist with the Research Institute for Advanced Computer Science (RIACS) at the NASA Ames Research Center. Previously he was a member of research staff with the Xerox Palo Alto Research Center (which is now PARC Inc.) wh ere he was doing research on learning and knowledge sharing strategies together with an interdisciplinary group of social scientists in the Knowledge, Interaction and Practice Area. Peter received his Ph.D. for the Johannes Kepler University Linz, Austria. There he was an assistant professor with the Department of Business Information Systems and the Department of Management for more then ten years. David A. Maluf leads the NASA Advanced Exploration Network laboratory (AEN), a laboratory consisting of 20+ staff with an average of 12 projects/year. He has over 70 technical publications in journals and conference proceedings, over hundreds of presentations at international conferences and is an inventor of numerous patents. He has taught courses on system engineering and databases, and has written two books. He received his PhD fro m Mc Gill University and conducted post-d octoral research in information integration at Stanford University. David G. Bell is Director and Senior Scientist at the Research Institute for Advanced Computer Science, located at the NASA Ames Research Center. Prior to working at NASA, David worked for ten years at the Xerox Palo Alto Research Center, and previously held an appointment at MIT where he led a research program in the Center for Innovation in Product Development. David is co-inventor of multip le patent and patent-pending information system technologies, including XML query technologies related to NETMARK and the NASA Program Management Tool, extensible blog technology called Sparrow Web, and distributed knowledge management software called Eureka. David received his Ph.D. from Cornell University with a dissertation on the dynamics of product development processes. Mohana M. Gurram is a computer scientist for Universities Space Research Association at NASA Ames Research Center. He has worked with Mars Exploration Rover mission and has been presented with awards like TIGR, Honors Award at NASA. His area of interest is Data Visualization especially context-sensitive data. He earned his Masters in Computer & Information Science. Jennifer Hsu is a Systems Analyst with QSS Group Inc.. She is a member of the Program Management Tool Development Team at Advanced Exploration Networks Laboratory, NASA Ames Research Center. She also worked on Mars Exploration Rover Mission’s Rover Activity Planner, MAPGEN project. She received her Ph.D. degree in Biochemistry from Cornell University. Hem il N. Patel is a computer scientist with the QSS Group at the NASA Ames Research Center. Previously he worked on the Aviation Data Integration System (ADIS) where he received numerous awards including NASA's Space Act Board Award. Hemil has a Master's Degree in Electrical Engineering from University of North Carolina at Charlotte. Keith Swanson is a computer scientist in the Advanced Exploration Networks laboratory of the Intelligent Systems Division at NASA Ames Research Center. He has over 20 years of technology management and development experience in the areas of system health management, planning and scheduling, and knowledge-based systems. Keith has a Master's degree in Computer Science from Stanford University and a Master's Degree in Engineering from UC Berkeley.
