|
|
|  |
 |
Better Software Home > In This Issue > Featured Article
|
 | |
November 2007 | |
 |
|
The Measure Of A Management System
by Jim Highsmith
In the six years since the Agile Manifesto was published, a wide variety of organizations have implemented agile project management and software development practices. However, far fewer have taken the step vital to unleashing the power of agile methods—namely, changing their performance management systems. All too often, an agile team's success is determined by traditional measures, or worse, the team's adaptability and agility is restricted by measuring and rewarding the wrong things.
I once worked with a project manager whose team's results were considered by the organization to be a failure. He had been given two hours to create his “project plan," and of course, when the actual work was more extensive than he had forecast, the schedule slipped—but the expectations didn't. Subsequent independent analysis of this project put it in the “above average” category when compared to industry norms for productivity index and scheduled response. This is one example of using the wrong metric to measure team performance.
In order to achieve truly agile, innovative organizations, a change in our approach to performance management systems is necessary. The adaptive performance management system (APMS) is one such system. APMS is divided into two parts: the project performance management system, which focuses on outcomes that generate customer value, and the team performance management system, which focuses on informational metrics that teams can use to improve.
Change is defined in the American Heritage Dictionary of the English Language, Fourth Edition as "to cause to be different; to give a completely different form or appearance to." Adapt means "to make suitable to or fit for a specific use or situation." There is no goal inherent in change; as the bumper sticker says, “Stuff Happens." Adaptation, on the other hand, is directed toward a goal. Change is mindless; adaptation is mindful. Adaptation can be considered a mindful response to change. It is responding to change in order to meet an established goal within certain constraints or boundaries. Adaptation is a natural response to our environment. Our business environment changes constantly, and so we should measure our response to those changes—our adaptability—to meet our goals.
Our traditional project management and budgeting performance management systems were designed to measure conformance to plan (or budget), not adaptability. These systems need to change—not just at the project management level but throughout the organization.
Adaptive Performance—Outcomes and Outputs
In a recent Business-IT Strategies Resource Center Executive Update, Cutter Consortium senior consultant Helen Pukszta writes:
I recently asked a colleague whether he would prefer to deliver a project somewhat late and over budget but rich with business benefits or one that was on time and under budget but of scant value to the business. He thought it was a tough call, and then went for the on-time scenario. Delivering on time and within budget are part of his IT department's performance metrics. Chasing after the elusive business value, over which he thought he had little control anyway, is not.
If we are to benefit from agile methods and become truly agile, innovative organizations, then, as this story indicates, we will have to alter our performance management systems. Having a "system" that leads managers and others into valuing “conformance to plan" while delivering "scant business value" will seriously impede agility—and ultimately, success—in both projects and the entire enterprise.
APMS addresses this need for a new approach by looking at two "systems": one set of measurements oriented to outcomes—the business value created—and another set of measurements oriented to outputs—productivity, cost, and so on. The difference between this and traditional measurement systems is that, for outcomes, the fierce link to quickly outdated plans or budgets is broken and the output measures are geared toward learning and improving by looking at relative performance across teams rather than performance against a plan. Two key objectives of APMS are:
1. To focus any enterprise group (team, department, division, or company) on a set of desired strategic or tactical outcomes (effectiveness)
2. To encourage those groups to perform at high measures of output (efficiency)
The Predictability Paradox
Measurement issues are complex because the business community wants its organizations not only mobile, agile, flexible, and adaptive but also predictable and reliable. There seems to be a core disconnect here; pick one: agility or predictability. In fact, companies need both, and herein lies the dilemma. The business world requires agility to be successful, but many companies are mired in predictability in organizational structure, management style, and performance measurement systems. This could be called the "predictability paradox."
It is a fantasy to act as if we can continue to measure performance in traditional ways and also have flexible, adaptive organizations. This doesn't mean we abandon measurement or completely replace existing systems, but it does mean that we have to adjust our thinking about what constitutes "good performance." An example is how we traditionally think about measuring project management performance using the "iron triangle" of scope, schedule, and cost. Success is defined as having a project meet all its requirements and delivered by the planned time, within the budgeted cost. In many companies, requirements are "fixed" and project managers are forced to be maniacal about cost and budget—often to the detriment of delivering value.
When management has rigid expectations, project teams are actually discouraged from being flexible. The narrower the tolerance to variation, the greater the restrictions on adaptive behavior. The wider the tolerance to variation, the fewer the restrictions on adaptive behavior—within some limits. Success, particularly today, is a function of a team's ability to react to change, not its ability to plan and follow that plan. How teams are measured affects their adaptiveness. Adaptation—rather than adherence—brings success.
Measurement Concepts
In looking for concepts on which to base an adaptive management system, two have proven to be particularly useful: the concepts described by Jeremy Hope and Robin Fraser in Beyond Budgeting and Rob Austin's views on performance measurement in organizations described in Measuring and Managing Performance in Organizations.
Budgeting Systems in Organizations
In Beyond Budgeting, Hope and Fraser outline a measurement system and management style that fit with an agile enterprise. Although Hope and Fraser discuss issues beyond budgeting, they start with the issues surrounding traditional budgeting systems in organizations:
Budgets have since been hijacked by a generation of financial engineers who have used them as remote control devices to "manageby the numbers." They have turned budgets into fixed performance contracts that force managers at all levels to commit to delivering specified financial outcomes, even though many of the variables underpinning those outcomes are
beyond their control.
I would argue that the "generation of financial engineers" has been paralleled by a generation of project administrators who feed on the same ideas and have created a culture of conformance-to-plan performance measurement. Both financial and project management measurement systems suffer from the same problems:
- They are cumbersome and expensive. We spend time and money on excruciatingly detailed plans based on a fuzzy understanding of the project and significant uncertainty about the future.
- They are out of kilter with the competitive environment. These fixed budgets or plans are out of kilter with the competitive environment because things change and assumptions are invalidated.
- "Gaming the numbers" is rampant. In fixed-performance contracts, budgeting and estimating are often hog-tied by politics, and their relationship to real performance remains tenuous at best.
The solution to these problems with budgeting, according to Hope and Fraser, is to abandon the traditional budgeting process and create a different kind of performance management system. This system is based on key performance indicators (KPIs) that establish strategic and tactical outcomes desired by companies and relative performance indicators that measure performance against internal and external benchmarks rather than using fixed budgets or plans. To improve team and individual performance, flawed fixedperformance contracts are replaced by relative improvement contracts that promote self-regulation within organizational units and project teams.
Measuring Performance in Organizations
Performance measurement and management has proven to be much more difficult than people expected. Anyone undertaking the design of a performance management system would do well to read Rob Austin's Measuring and Managing Performance in Organizations, a sobering look into why measurement systems can go awry. Measurement "systems" are difficult, according to Austin, because "unlike mechanisms and organisms, organizations have subcomponents that realize they are being measured." In his introduction, Austin states that "if there is a single message that comes from this book, it is that trust, honesty, and good intentions are more efficient in many social contexts than verification, guile, and self-interest." The intentions of the managers who use the measurement system are what ultimately determine its veracity.
Many organizations have dysfunctional measurement systems. In the beginning, a measurement system tends to improve results because people don't really understand the system. However, under pressure to "improve,' people are forced to subvert good intentions in order to meet the measurement goal. Since there is always a disconnect between the desired outcome and the metrics used to achieve the outcome, over time the "measured performance trends upward; true performance declines sharply," according to Austin.
I recently worked with a company whose managers were compensated according to CMMI levels achieved. This is a clear example of measuring not output, but input or process. (This should be interpreted not as a problem with the CMMI, but of its use.) Measuring and compensating staff for process improvement—if not combined with outcome measures that are understood to be more important—is a clear road to dysfunctional behavior. While a focus on process improvement measures may improve efficiency, it will distract people from measuring effectiveness (outcomes); so asthe measured performance trends upward (CMMI levels), true performance (outcomes) declines sharply.
Some project management metrics used today have made this trip from functional to dysfunctional. Earned value analysis (EVA) is one such dysfunctional metric. Its name indicates that it measures value, but the calculation combines schedule and cost into a single performance number completely unrelated to value! Project teams striving to achieve EVA targets have a disincentive to adapt in order to deliver valuable business results.
APMS Design Guidelines
Drawing from the ideas of Hope, Fraser, and Austin, the following design guidelines were developed for APMS:
- Build the measurement system on a foundation of trust, honesty, and an intent to increase organizational value.
- Place the most emphasis on measuring outcomes, even if the metrics are not easy to obtain or completely precise.
- Create outcome metrics whose constraints are as broad as possible (tolerance for variations) in order to encourage adaptation.
- Create output informational metrics that support people's innate, internal, motivational needs, and provide them with aggregate, rather than individual, measures of overall performance.
The keys to a successful transition to an agile project, organization, or enterprise are focusing on customer value rather than schedule, building collaborative project communities based on trust and honesty, and learning from good feedback systems. Our measurement systems must support these focal points. The measurement system must ensure that the connection between business drivers (profit, ROI) and project deliverables is transparent
In order to do so, ask two questions. The first question is always "Is the project community delivering a continuous stream of value to the customer?" Ultimately, outcomes can only be determined after the project delivers "running, tested features" and the business impact is assessed. However, the agile focus on feature value, whether qualitative or quantitative, gets teams closer to outcomes than scope and schedule.
The second question is then focused on constraints: "Did the project community deliver within acceptable cost, schedule, and defect constraints (not estimates)?" Outcome metrics are intended to guide decision making—to define the desired outcomes and then trust people to find the best way to produce those outcomes. Plans are used to guide project teams but not to encase them in straitjackets. Teams will strive to deliver to the plan, but delivering the desired outcomes will take precedence.
Constraints
Outcomes are some measure of business value. But businesses require more than outcomes themselves; they require that outcomes be generated within certain constraints in order to make them financially viable.
At a high level, we need to understand constraints in three areas: scope, schedule, and cost. The measurement-design guidelines give us as few narrow constraints as possible (to encourage adaptation); therefore, only one of these three characteristics should be a must constraint, and the others should be ant constraints. For example, if the project's objective relates to a governmental requirement that must be met by some date, then the must constraint is that schedule. My definition of a constraint is "a measured characteristic of a project that if projected to exceed its target would subject the project to immediate cancellation or major revision requiring sponsor approval." Constraints should be broad in order to encourage flexibility and experimental design—and thereby innovation.
When looking at performance metrics in this way, we should not forget that value (outcome) is not independent of cost and schedule. For example, a project's viability (ROI) might depend on delivering six high-level capabilities, constrained by a June delivery date (must) and a $250,000 cost (want).
Measuring Outcomes and Constraints
The three key questions to be answered by outcome metrics are:
1. Are we consistently delivering chunks of functionality that are useful to the customer team?
2. Have we delivered that functionality within the constraints set by the customer team and sponsor?
3. Has the project community adapted its plans, activities, and strategies to deliver within the project's constraints?
Project status historically has been shown in Gantt charts that emphasize activities and schedule. Agile projects need new types of status reports, like the "parking lot" diagram in figure 1 that emphasizes value first (each box represents features or stories that have been completely implemented and acceptance tested) and schedule second. While the real outcome is value, the assignment of cost to stories can be difficult and timeconsuming. If teams are dedicated to linking relative value to stories during the project, then parking lot diagrams (or similar story burn-up charts) reflect outcomes much more directly than Gantt charts. If we want managers and others to think differently about performance reporting, then we need to change the reporting mechanisms to reflect the most important characteristic—the outcomes.
Output Performance Metrics
Managing people is a key ingredient of project management that includes coordination, motivation, conflict management, collaboration, and more. We want teams to increase their productivity. We want teams to deliver faster. We want teams to deliver low-defect products. However, we won't achieve these goals by measuring conformance to plan; we will achieve them by measuring these characteristics directly and comparing progress against external and internal benchmarks.
Measuring team performance against plan often leads to poor results. If a team is assigned to a project where the plan is heavily padded or buffered and it succeeds, is it then a high-performance team? If a team is assigned to a project whose plan is completely and utterly unreasonable and it fails to achieve the plan, is it then a low-performance team? Wouldn't comparing team performance against realistic internal and external benchmarks be better? We want teams to improve their performance, and, in an agile environment, we encourage teams to work on improving their own performance. Management's job is to provide "informational" metrics so that teams can evaluate their own performance and work on improving outputs.
One of the simple metrics I use with teams during retrospectives is to ask the question "From both a performance (stories delivered) and a team-effectiveness perspective, are you doing the best you possibly can?" This is very different frommeasuring against a plan. I find most teams are honest in their evaluation of themselves and what they need to do to improve.
A single article doesn't provide space to examine the details of a quantitative output-measurement system, but there are approaches—schedule, productivity, cost, etc. (for example, Lawrence H. Putnam and Ware Myers's Five Core Metrics: The Intelligence Behind Successful Software Management)—that allow a team to:
- Compare its progress over time against itself—Are we getting better?
- Compare performance against other internal teams—How do we compare with others in our organization?
- Compare performance against an external measure of like industries or projects—How do we compare with other companies?
One of the clear advantages of measuring relative team performance rather than how well a team performed against a plan is that there are no upper limits to improvement (target). Since everything is relative, good teams will strive to get better, not just try to meet a target. Teams that only have to meet a fixed target tend to manipulate that target to their advantage and then slack off once the target is reached. Relative performance keeps constant internal pressure on teams to improve.
Conclusion
Measurement systems are crucial to creating an agile enterprise, particularly for those projects that implement new products, services, or processes that will create your future company. If you are creating the future, an extra month or a few extra dollars will be insignificant. If you don't deliver value or you don't deliver innovation, that will be significant. We cannot continue to ask teams to be innovative, flexible, and adapt to changing competitive conditions and then measure their performance by forcing them into narrow expectation alleys. We have to be as innovative with our measurement systems as we are with our methodologies.
In conclusion, I want to return to one of the guidelines for APMS design: "Build the measurement system on a foundation of trust, honesty, and an intent to increase organizational value." Those who have studied the agile movement understand that these agile, adaptive approaches are really social and management style movements. In my book Adaptive Software Development, I outline the differences between what is typically known as a command-and-control style of management and what I call a leadership-collaboration style. Extreme Programming (XP), Crystal, and Scrum all profess similar kinds of social changes—oriented toward improving the human dimensions of our workplaces and thus improving performance.
If we want adaptive organizations, we have to think long and hard about what success and performance mean and align our measurements of success and performance to the values of the management style we choose.We have to build our measurement systems on the right intentions—to illuminate, not punish; to learn, not repeat; to adapt, not resist change. If we want to build adaptive organizations, then we must abandon fixed-performance contracts for a measurement system that aligns with these new intentions. Hopefully, APMS is a step in the right direction. {end}
Jim Highsmith, considered a leader of the agile movement, consults with companies worldwide on agile practices. He is the authorof Agile Project Management; Agile Software Development Ecosystems; and Adaptive Software Development, which won the Jolt Award. He is co-author of the Agile Manifesto and the Declaration of Interdependence. |
| | |
