When Earned Value Doesn't Help

Earned Value Methodology Definition

Earned value is the key element in the earned value methodology (EVM). Earned value can be thought of as a measure of the value of physical progress in a project. It integrates three elements to quantify the value achieved by a project: 1) budgeted cost of work scheduled (BCWS) or the budgets of the activities planned to be completed; 2) actual cost of work performed (ACWP), or the real cost of the work charged against the completed activities; and 3) budgeted cost of work performed (BCWP), or the planned cost of the activities that were actually completed. BCWP is itself often referred to as "earned value."

BCWP or earned value may be most easily defined as the sum of the budgets for the work that is complete. Therefore the earned value for a completed project is equal to the total budget. For activities not yet begun, the earned value is zero. In order to measure the performance of activities in progress, you must come up with a system of measurement that includes objective judgments. The Earned Value Management System (EVMS) guidelines promulgated by the Under Secretary of Defense for Acquisition, Technology, and Logistics give a number of alternative methods for measuring the earned value of an activity in progress. These methods are often called earned value methodologies or performance measurement techniques (PMT). The basic theory behind all of the methodologies is to multiply the budget by a percentage complete to get the earned value. Most projects contain at least some work that is regarded as inherently immeasurable. An example of this is the work performed by a project manager or a quality control inspector. This type of task is sometimes referred to as "level of effort" since its earned value is assumed to be the same as the amount budgeted. Basically, as long as the task is performed, then the value is earned.

Baseline Management

Management of the project baseline is critical. Fundamentally senior management is most interested in the cost of progress toward desired products and the probability of meeting schedule targets. In order to measure progress there must be some ruler to measure against. Whether called a plan or a baseline, the estimated cost of the estimated amount of work is the ruler. If only one thing can be managed, the baseline is by far the most important. Is the baseline established and complete? Are changes managed? Cost and schedule problems can usually be traced to a baseline that is ill-defined, or out of date. EVM integrates cost, schedule, and technical performance into a single metric so managers can make effective comparisons. By using the same unit of measure for both physical progress and cost, EVM helps project managers compare planned and completed work. EVM requires project managers to clearly define the scope of the project before beginning work; allocate appropriate resources; objectively measure work accomplished; identify problem areas early; realistically project cost to complete; and limit the inclination of clients or superiors to add work without increasing the project budget.

Baseline Development

Defining the project baseline is a comprehensive planning process. The planning process requires breaking the entire statement of work down into discrete, credibly estimable tasks and then a bottom-up plan be created. Dependencies must be identified for each task to ensure that the tasks are scheduled appropriately. Resources (people, facilities, equipment, even travel funds) are associated with each task and the dependency-based schedule reviewed to assure that resources are not overcommitted (i.e. that individuals are not expected to work more than full-time (neglecting overtime) and facilities are not scheduled for more than their capacity). This allows measurement to take place during the entire period of performance, from 1 percent to 100 percent of the project's lifecycle. The quantification of management cost and schedule reserve is part of baseline development. The means for determining a project's management reserve is to assess the risk associated with aggregations of tasks. The risk assessments are used to derive a desired reserve for cost and for schedule at the project level. While there are stochastic mechanisms for the calculation of management reserve in the program management literature, and such methods have been used on particularly large (billion dollar) DoD programs, the standard of practice is to use management judgment to determine the appropriate total reserve. Since almost all programs do not have adequate provision for either cost or schedule reserve there is little to be gained by engaging in the more deterministic approach. In fact, empirical methods for determining management reserve based on the difference between original estimates and actual costs for historical projects of similar nature will lead to reserve determinations as valid as any other method. The completed baseline flows into scheduling systems to provide for work being carried out in the proper order. It also flows into budgeting systems to provide adequate financial resources at the appropriate time and accounting systems to identify the proper categories (charge accounts or numbers) for the accumulation of costs. It flows into procurement systems for the purchase of equipment, sub-contracted services, and logistics support. Finally, the baseline flows into work measurement systems used to determine earned value.

Baseline Maintenance

A nightmare for project managers in general and software project managers in particular is "extras" thrown at them by the customer (or the customer's customer). Of course, revised requirements are supposed to be renegotiated and reflected by a revised project baseline that includes a new completion date and changed cost. However, many times the "requirements creep" seems so trivial that project managers forego the perfect practice and merely adjust their funding reserve to account for the change. For many situations, the effort required to re-baseline the project and negotiate the change is far greater than the amount of reserve lost. Despite advising customers that changes are being accrued and that the right is reserved to negotiate them once it is apparent the effort to do so is worthwhile in practice the effective baseline for product performance changes incrementally and the contract baseline for cost and schedule does not.

Earned Value in Practice

"Not everything that can be counted counts, and not everything that counts can be counted."

An earned value management system is not a reporting system, contract administration, cost analysis, accounting, or a contractor's task management system. It is a measure of the value of physical progress in a project and as such adds additional effort to the work of managing a project. Beyond the additional effort of an EVMS, care must be taken to avoid hindering the contractor's ability to use its organic management systems. Earned value presupposes that the "signal" of actual progress overwhelms the "noise" of error. Errors in EVMSes stem from estimating error, measurement error, and baseline error. Estimating error is the deviation of the cost and schedule resource estimate in the baseline from the ideal; it is the result of extrapolation, scaling, imprecise analogy, and assessment limits. Measurement error is the difference between the measured percentage complete and the actual percentage complete; it is the result of person-to-person communication, limits of understanding, and human optimism. Baseline error is the difference between the program plan and the program as executed where those differences are in response to evolving needs, changing external constraints, requirements creep, and customer direction.

Earned value has a "processable signal-to-noise-ratio" (provides meaningful results) when the implicit assumption that the total error in earned value reporting is significantly less than the actual measurement. Since most thresholds for initiating corrective action are exceeded for deviations from the plan of less than an order of magnitude of the baseline, i.e.

BCWP - BCWS	< 10%
BCWS

It follows that the error component should be less than two orders of magnitude of the baseline.

Estimating error is static once the baseline is defined and in place. In fact, one of the functions of an EVMS is to identify the existence of significant estimating error. Constant deviation of reported earned value metrics of CPI or SPI from 1.0 (above or below) indicates systemic estimating error.

Measurement error exists in the results reported each period and is an inescapable reality. Research has explored many alternatives to reduce the measurement error component without real success. Earned value progress measurement methods such as 0-100, 50-50, and 20-60-20 succeed only converting a portion of the statistical error to a fixed bias at the expense of motivating project performers to execute tasks in a suboptimal fashion driven by a desire to "make their numbers."

Baseline error accrues in a quantum fashion. Each time the end goals of the project move incrementally, another error term is added.

Earned value is effective because the contributing estimating and measurement errors are fundamentally Gaussian in nature, neglecting any fixed bias in estimating which would soon become apparent as noted above. For a program with an accurate estimate and rigid baseline management, measurement error is the significant contributor to errors in reported progress. In accordance with the theory of large numbers as applied to a Gaussian distribution of error terms, for a large enough number of tasks in a program as described, the reported results of an EVMS will be accurate.

Statistics reminds us that the level of confidence of a probabilistic calculation for a standard deviation is only 66 per cent. Quite a large number of tasks are required in order to attain true accuracy of earned value reporting as a project proceeds even without significant contributions from baseline error.

For purposes of discussion, given an accurate earned value calculation what is the benefit thereof? Earned value does not facilitate a prospective view of the program. Certainly there are calculations to determine the "to complete" performance indices, but those calculations assume that the plan to reach satisfactory completion will not and cannot change. They do not reflect risk retirement or acceptance, and they do not account for the allocation of management reserve. In fact both EVMS and cost/schedule control system criteria (C/SCSC) methodologies continue to struggle with the appropriate accounting for and reporting of management reserve. In neither system is management reserve segregated into quantifiable management elements.

Define Work Scope

Using a work breakdown structure (WBS) to organize the project baseline data, 100 per cent of the scope of work must be defined based on the statement of work and the element specifications. The baseline is a prerequisite for development of a cost-to-complete, definitive assessment of performance, and earned value reporting.

Realistically no work effort can be defined with absolute precision; some intelligent assumptions are necessary to quantify the work with sufficient confidence that the defined effort can be planned, scheduled, and estimated with some degree of certainty. Revisiting the baseline manages residual risk associated with those assumptions in addition to providing a means of responding to externally driven baseline changes.

This scope definition must and would be established and maintained whether or not earned value reporting is required.

Integrated Bottom-Up Plan

Critical processes, including defined work scope, schedule, and estimated resources, are combined into an integrated bottom-up plan of individual tasks. The ordering of tasks based on their interdependencies and the availability of required resources (principally staff) is necessary for assuring that the plan is realistically achievable, and for assessing progress against the baseline plan. For earned value, this schedule work constitutes the planned value, or BCWS. While the portion of the planned value that is accomplished becomes the earned value (a retrospective measure), qualitative assessment of the work yet to be completed and the capability and availability of the assigned staff (prospective management) is what leads to project success.

The detailed plan must and would be established and maintained whether or not earned value reporting is required.

Task Assignments for Performance

Each of the defined tasks must be assigned to individual staff members for performance. A senior individual is responsible for performance of the groups of tasks that lead to verifiable products; that person provides the assessment of per cent complete for earned value reporting. More importantly, they make the tactical judgments regarding ongoing work to continue progress and provide technical and management guidance to the other assigned staff that lead to completion of the specified products.

Task assignments and accountability of element leads must and would be established and maintained whether or not earned value reporting is required.

Measure Performance against Schedule

The project's schedule performance is measured regularly against the planned master project schedule that is the time-domain portion of the baseline. The schedule constitutes the view of scope used for daily, weekly, and monthly guidance of work effort. On a monthly basis (coincident with the publication of project status reports by company cost accounting), the progress accomplished is compared to the progress scheduled, and adjustments made as necessary to staffing, priorities, and technical approach. For earned value reporting, the additional step is taken of assigning value to the progress accomplished for comparison with the distributed value of work scheduled in the baseline; the difference is the schedule variance.

A negative schedule variance reported by an EVMS means that the value of the work accomplished does not match the value of the work scheduled, i.e., the project is falling behind in its scheduled work. If the late task is on the critical path, or if the task carries a high risk to the project, efforts can be made to get the late task back on schedule. However, the earned value reporting for cost and schedule is contingent on the availability of accounting reports that are available approximately ten days after the close of the monthly accounting period (at best--some companies run a full month behind the accounting period). Accordingly earned value metrics are calculated for work two to six weeks earlier. Daily and weekly communication between staff and element leads, and between element leads and project management, address schedule and technical performance and initiate corrective action as needed. Earned value data therefore does not contribute to issue identification and in fact reflect issues after they have been responded to.

Performance must and will be measured against schedule by element leads and project management whether or not earned value reporting is required.

Measure Cost Efficiency against the Costs Incurred

Cost performance must be measured regularly. Project status reports show period and cumulative costs for each cost account (charge number) and CLIN. Although available after the point of issue identification, the accumulated cost information over time will provide useful trend information that will, when compared to schedule performance, indicate where there are staffing or procurement issues.

Despite the position of earned value management advocates that displays of costs incurred as a function of time do not provide for the separate evaluation of cost and schedule performance, experience on small teams in small organizations is that the expenditure curve synthesizes total project status very effectively. This is due at least in part to: regular and detailed insight into schedule performance independent of incurred cost, small team sizes, high and stable productivity rates, and assurance within the staff of adequate pending work.

For earned value reporting, data must be extracted from the baseline to determine the originally estimated value of the work actually performed for comparison with the actual cost of the work performed to develop earned value cost metrics. Not only are the earned value cost metrics not timely, they do not reflect investments associated with risk management that may reduce the probability of program cost growth despite near term costs.

Accumulated costs must and will be accounted for and compared to the baseline expenditure plan whether or not earned value reporting is required.

Forecast Final Costs Based on Performance

In association with a regular (annual) baseline review, the work remaining is reassessed and a forecast produced of final cost requirements based on its performance against the plan. Earned value methods forecast the estimate at completion (EAC) using linear extrapolation, assuming variously that one and only one of productivity and schedule is fixed. management considers the dimensions of cost, schedule, technical performance, and risk in conceiving the modifications necessary to the baseline plan in order to meet technical and programmatic requirements within the available budget. Earned value can provide retrospective insight into the implications of progress against the baseline plan. Technical and personnel management have greater value to guiding the program to successful completion.

An EAC must and will be developed in association with each baseline review to provide strategic insight into management needs and to reallocate the planned staff assignments for the ensuing increment whether or not earned value reporting is required.

Manage Remaining Work

Management and element leads must continuously manage the project's remaining work. At any time, the results achieved to date, good or bad, are "sunk costs." Any improvements in performance must come from future work—tasks ahead of the latest status date. Daily and weekly schedule-driven status assessments provide effective and timely fora for guidance and direction to staff. In the absence of such a management approach, earned value would measure the cost and schedule performance achieved to date and areas requiring management attention would be identified. Due to the reality of systemic latency in the reporting and earned value calculations, earned value reporting is redundant in this regard and often misleading since it lags the functional management process in place.

Management and technical leads must proactively manage remaining work using daily and weekly status discussions whether or not earned value reporting is required.

Manage Baseline Changes

Every individual assigned to the program must strive to accomplish the required technical performance for each delivery and installation. As a practical matter that has resulted in cumulatively substantial additions to the scope of work over that conceived in the baseline plan without concomitant adjustments in the budget and schedule requirements.

Classical program management tenets, earned value scope management methodologies, and theory of constraints guidelines mandate rigid control of the project baseline. No additional work or other effort should be accepted without negotiation of a change to the baseline, generally requiring greater budgets and longer schedules.

Unfortunately, the adjustments in work flow to accommodate the needs of the customer or the customer's customer may perturb the earned value reporting metrics, causing additional burden to perform the requisite calculations that do not reflect the real status, or the commitment to deliver the required capabilities within budget and schedule.

Conclusion

Earned value methodologies certainly have their place. Appropriately applied to large programs with many tasks, hierarchical organizations, and large budgets they add value by identifying issues and problem areas earlier than would otherwise be the case. A program with many tasks can count on the theory of large numbers to result in the error terms of earned value metrics remaining small relative to deviation action thresholds. A program with a large, hierarchical organization can use earned value methods and reporting to facilitate communication of performance status to senior management where corrective action may be taken.

Large budgets accommodate the relatively fixed cost of an infrastructure to implement an EVMS efficiently and attain positive cost/benefit ratios. These characterizations may not apply to small teams in small organizations with relatively few tasks to execute. A small project team with a flat organizational structure in a company that itself has an extraordinarily flat structure will find that communication is convenient and frequent. In such an organization, all infrastructure tasks are "other duties as assigned" for technical staff, drawing energy and time from tasks that directly contribute to development of required products and services.