A couple of weeks ago, I received an email from a ScrumMaster friend of mine. A group of managers at her company had determined they needed a metric to measure project efficiency, so they came up with the idea of comparing the number of forecasted story points for a time frame with the number of story points that were actually delivered by a project team. They felt the ratio of actual story points to forecasted story points for all the sprints in a given time frame would demonstrate whether the project is delivering to its capacity for work according to business priorities. The managers thought the measure would indicate whether business value was being delivered according to forecasted timelines, and it could be used for guiding continuous improvement efforts.
Here's how I responded:
“I don't like it at all.
It's measuring output, not outcome. I'd be much more interested in whether the team is delivering value, and in some cases you may be able to provide more value by delivering fewer story points. Of course, the problem here is that value can sometimes be very difficult to measure, so people like to fall back to measures like this, which in all reality is just a distorted "earned-value" calculation.
It's measuring the wrong thing—this is really telling you how "accurate" estimates are, which I think is the wrong thing to look at. See my first point.
The team did not come up with the measure, so they most likely were not able to weigh in with what it actually means.
It will drive bad behavior. "Oh, we're being measured on how many points we get done. Okay, we'll inflate our point estimates, and we'll do easy stuff that we know we'll be able to get done." The story points lose their usefulness as a tool to help the team predict.
What value does this metric bring? What will you do with this information? What are you trying to accomplish with this measure?”
My friend replied that she agreed with my assessment and appreciated the backing. She also asked if I had any ideas on measuring business value. She mentioned that she had heard about assigning value points to stories but wasn't sure she liked that, either.
My response was not as direct as the one above, but I did point out that starting with stories and then trying to derive value was, in effect, doing things backward. Ultimately, delivery teams should be concerned with helping their stakeholders solve their problems (i.e., delivering value). In the case of software development efforts, a delivery team should also strive to solve those problems with as little code as possible. The more code they write to solve the problem, the higher the likelihood that today's solution will be well on its way to generating tomorrow's problems that will be solved by—you guessed it—more code. It's this vicious cycle that generates the bowls of spaghetti commonly known as "legacy systems." Establishing a backlog of stories that the delivery team will deliver before determining their value will inevitably drive teams to produce more than they need to. Even if you assign relative value scores to stories with some stories getting higher scores than others, the team will more than likely assign "value points" to everything, even those stories that really don't accomplish anything. This is a concept described as feature injection.