|
|
|  |
 |
Better Software Home > In This Issue > Featured Article
|
 | |
April 2008 | |
 |
|
Incremental and Iterative Development
by Alistair Cockburn
People still get wrapped around the axle trying to understand the difference between incremental and iterative development. The Unified Process authors in the 1990s didn’t help by indiscriminately calling everything iterative development. The two are different and must be managed differently. Successful teams do both at the same time, usually without thinking about it. Then someone starts thinking about it and does one without the other. Bad news follows.
Current agile practice doesn't seem to make it important that the users get to see what’s being built and have a chance to change it. Iteration lengths are so short that the programmers barely have time to program up the basics before the end of the iteration, so there's no time for the users to show up and say, "No, that’s not actually what I want."
I heard one Extreme Programming (XP) customer ask others in a multi-project exchange:
Is it just me, or do you also have to get it right the first time? On our project, if I request a change, they tell me that story will go to the back of the queue and the project timeline will be delayed. As a result, I have to get it right the first time. Isn’t this against the very spirit of agile development?
This person is right on all counts—it happens, it runs against the spirit of agile development, and it is bad business.
This problem is the opposite of what we wrestled with back in the 1990s. Back then, we struggled to get people to do incremental development. Getting people to break work into small-ish pieces and build a piece at a time was all we aspired to. Many people understood that it was good policy to show the results to users several times, and so, oddly enough, we sometimes saw iterative but not incremental development!
If you create smaller pieces but don’t show them to real users and incorporate their responses, then you revert back to the project failures of the 1980s, when users complained that what got delivered to them didn’t satisfy their needs.
How Did We Get into this Mess?
Back in 1991, my assignment with the IBM Consulting Group was to investigate
development methods for object-technology projects. In my research, I ran into two terms--incremental development and iterative development.
I read articles and interviewed people to learn what they meant by these two phrases and learned that they refer to distinct staging and scheduling strategies. They are choices made by the project team on how to slice, integrate, and revise the work. We learned that:
Incremental development is a staging and scheduling strategy in which the various parts of the system are developed at different times or rates and integrated as they are completed. The alternative is to develop the entire system with a big bang integration at the end.
Iterative development is a rework scheduling strategy in which time is set aside to revise and improve parts of the system. The alternative development is to get it right the first time (or at least declare that it is right!).
Please notice that these strategies don't imply, require, or preclude each other. It is possible to work incrementally without iterating (the product manager's complaint at the start of this article), to iterate without incrementing, or to do both (the recommended strategy).
In the next section, we'll look at the definitions more closely along with how to manage the differences, but first: How did we get into the situation where the two became so confused?
Back in the 1990s, the authors of the Unified Process—perhaps so excited to be able to revisit the requirements and design portion of the process at all--decided to call everything "iterations" and "iterative development," erasing the line between the two. At that moment, the difference in the two strategies got lost. Since then, projects have failed by doing one without the other. You need both, and you need to manage them differently.
Incremental Development
Incremental development is easier to explain and to manage, so let's start there.
First, let's get comfortable with the "Validation V" shown in figure 1. Read the figure from right to left, to see it as the dependency diagram it really is: We can't ship our system until it's been integrated, tested, and validated. It can't be tested until there’s code, and we can't design and code it until we've decided what to build.
Figure 1: The Validation V is a fact of life.
Some people see this diagram and immediately shout, "Waterfall!" (holding up the sign of the hex, or whatever). However, it is a simple fact of life, and we will take advantage of it in learning how to work with both incremental and iterative development.
Figure 2: The first increment delivers one slice of functionality through the whole system.
You can see the Validation V in action in figures 2 through 4, illustrating incremental growth of the system—multiple passes through the Validation V for each new section of the system to be built.
Figure 3: The second increment delivers another slice of functionality through the whole system.
In figures 2 through 4, the columns represent different features we wish to implement; the rows represent the different
layers: user interface, application, middleware, and database.
Figure 4: The third increment delivers the rest of the system.
Figure 2 shows that the first feature is completely available after the first increment, figure 3 shows that the second feature is completely available after the second increment, and figure 4 shows that the last feature is completely available after the third increment.
The figures don't show that the application or the database might have been refactored during the second or third increments. Those are iterative (rework) issues. They show the perfect incremental situation, in which additional features are added without having to revise the pre-existing features. It is rare that we get to work in such a pure incremental fashion in software development, which is why we also need to understand and manage the iterative strategy.
Incremental development is a very successful staging strategy that has been used as long as software has been built. The tragedy is that people forget to leave time to check for mistakes in understanding from the requirements gathering
periods, and they forget to leave time to refine their designs. For these, they need iterative development.
Iterative Development
Iterative development is all about allocating time to improve the features requested, the user interface, and the performance characteristics.
Iterative development also follows the Validation V. However, instead of releasing the system at the end of the V, we examine it from various angles, particularly the three listed above.
Figure 5: The Validation V for iterative development.
Figure 5 shows the minor adjustment to figure 1 needed to obtain iterative development.
Figures 6 through 8 depict iterative development. These figures show iteration (revision) happening within an increment.
Other strategies are possible, but this is the most common.
Figure 6: The first iteration delivers enough of feature 1 to evaluate what is correct and what needs revision.
Figure 7: After the second iteration, some revised parts still need improvement.
Figure 8: The third iteration produces the final and stable feature.
In constructing an iterative plan, two different strategies (and their blends) are possible. Each will work or backfire depending on the situation at hand. The art of iterative development is to guess well which strategy applies best at any given moment. The first strategy, shown in figures 6 through 8, is to develop everything as completely and correctly as possible the first time. The second strategy, illustrated in figures 9 through 11, is to develop as little as possible until things stabilize.
The idea in the first strategy is: If we can get it mostly correct the first time, it will take the least amount of time to revise just the few parts needing correction, and then the whole thing can be released. Musicians use this strategy when laying down mixes, patching only the faulty bars.
The idea in the second strategy is to expose early the places most likely to need rework, without having gone too far in the wrong direction.
To illustrate the second strategy, let’s look at the art world, starting with Michelangelo's painting of the Sistine Chapel [1].
Michelangelo knew that he would be perched on a scaffold each day, staring up at a small section of the vast ceiling, mixing his paints in wet plaster and trying to get a section complete before the plaster dried (his use of incremental development!). He knew that he would not have time to paint a section of (for example) a sun-lit body, climb down from the scaffold to see how his lighting angles looked, and climb back up to the scaffold and repaint the body. This would be a bad use of time and energy--that is, a poor iterative development strategy.
He made a full-scale, rough-draft "cartoon" of the whole ceiling, so that he would know what to put in each section
of the ceiling. He also made full-sized sculptures of certain human torsos, which he lit so that they would mimic the lighting from the windows in the Sistine Chapel. He sketched the lit torso, and then carried the sketch up the scaffold,
so he could paint the plaster correctly
the first time.
This was iterative development involving throw-away prototypes, a version of the second strategy.
To show more complete use of the second iterative strategy, I must invent an imaginary progression of work resulting
in Leonardo da Vinci’s Mona Lisa. Jeff Patton started this comparison [2]; I am modifying his progression to more clearly demonstrate the energy-saving value of the early iterative strategy, and to show the rework done.
Leonardo, experienced in the way patrons change their minds, first draws only a sketch, as shown in figure 9. He asks the patron for his thoughts.
The patron comments (we imagine): "Oh, no! She can't be looking right! That’s not her good side. She has to be looking left!"
Happy only to have done a sketch, Leonardo proceeds and blocks out the figure and the background—see figure 10—so the shape, the colors, and the background can all be seen. He is, perhaps, one-third of the way done by time and cost, so this is a good time to get more feedback.
The patron looks at the painting so far and complains again: "You can’t make her head that big! Shrink it to her body size!"
Leonardo, assessing that all the major obstacles have been met (and probably also thinking to himself that he doesn't want to give the patron any more leeway in how it looks in the end), finishes the painting without showing it again, see figure 11.
We can imagine that the patron grumbles a bit at the very end, but looks at what he has spent and says, "I'd rather have her eyes bigger ... but let’s call it done."
This second strategy—"Do the least work before getting feedback"—is a popular one in the software world, because
of the frequency of changes and the cost of those changes. However, either of the two strategies is valid.
Managing Incremental and Iterative Development Combined
The two fit together quite naturally in what I call "VW Staging"[3], which can also be thought of as "cycle unrolling"[
4]. Figure 12 illustrates. In the figure, the letter "S" means that the system ships at that point; the letter "E" means that the system only gets examined.
Figure 12: Integrating incremental and iterative development.
The figure shows that it is common to revise one subset of functionality several times before considering it good enough to ship. This is quite normal, but otherwise difficult to show on the schedule. Figure 12 shows it in an easy-to-read manner, and one that maps easily to a sponsor's calendar.
Of the two, iterative is the harder one Lisato manage, because the question always arises as to how many times to iterate and how much to allow to change. It’s not that you won't do rework if you don't schedule it--you would just like to have a way to forecast and allocate that time in advance.
On Project Winifred (see the sidebar), we developed a useful heuristic, which I recommend to this day: Allocate two revisions for every piece of functionality.
For the first revision, allocate one-third of the original development time. For the second, allocate half of that. These numbers are, of course, heuristic—a starting guess—but they are better than nothing. Tune the numbers for your team over time, as you gain experience.
These days, agile projects like the one at the start of the article make the mistake of not allowing for user review and correction. Let’s revisit that project, which we'll call Project Laddie.
In Project Laddie, development cycles were two weeks long. Their incremental development strategy was standard XP and Scrum: Put the user stories into a list, work from the top down, and show the customer what has been built at the end of the iteration.
Their iterative strategy was to have the customer say whether to put the completed-but-not-satisfactory story back on the top of the work queue, to put it somewhere down into the work queue, or to accept it. This is normal XP/Scrum behavior.
What I learned from talking to the customers on this project was that this is not as good a strategy as I had once thought. Very few customers (or product owners) are really strong enough to say, "Return it to the front of the queue—don't work on anything new until it's satisfactory." Instead, they feel the pressure to deliver so strongly that they are torn whether to deliver on time something that is not right, or to delay the project to make it better.
What these customers were missing, and I have since come to appreciate, is that they should have several chances to view and correct the growing piece of software as it is being developed. In other words, what I am coming to call the "user's bill of rights" is that they automatically get two chances to correct the work before it gets pushed either out to the users or back into the queue[5].
In contrast to Project Laddie, Project Winifred—a forty-five-person, multi-technology project—successfully incorporated
both increments and iterations.
I was pleased, recently, to run across an organization using Scrum across the company that applied this exact strategy, but within their one-month sprints. This company showed that they understood the use of both increments and iterations, and were pleased to get the benefit of both.
The Punch Line
Remember:
- The word incremental fundamentally means add onto.
- The word iterate fundamentally means re-do.
Your process, your feature set, and your product quality all need improvement. Use an incremental staging and scheduling strategy (and reflect between increments!) to improve your process and the feature set. Use the iterative, rework-scheduling strategy to improve the quality. {end}
References
[1] Zipser, Karl. "Drawing, Sculpture,
and the Sistine Chapel Ceiling."
[2] Patton, Jeff. "The Neglected
Practice of Iteration." StickyMinds.com. December 2007.
[3] Cockburn, Alistair. "Using VW
staging to clarify spiral development." Humans and Technology Technical Report HaT TR 1997.02.
[4] Cockburn, Alistair. "Unrolling
cyclic development processes." Humans and Technology Technical Report HaT TR 2004.03.
[5] Cockburn, Alistair. "Three cards for user rights," Humans and Technology Technical Report HaT TR 2007.03.
Alistair Cockburn is one of the world authorities on software development, having helped to craft the Manifesto for Agile Software Development and the project management Declaration of Interdependence. He created the Crystal family of ultralight methodologies to capture patterns of successful project teams. Many of his materials are available online at
www.alistair.cockburn.us. |
| | |
