Software development is currently being "driven". This article finds existing X Driven Development approaches wanting because they focus on too narrow an aspect of development and, primarily, because they are grounded in the wrong philosophy of what exactly software development "is". An alternative ( theorY Driven Development - YDD), addressing the "essential difficulty" of development is proposed. How YDD represents an evolutionary step for Agile is argued.

A Plethora of D's
At minimum, software developers are confronted with TDD, DDD, MDD, RDD, LDD, FDD, PDD, and SDD. The middle D in all acronyms represents the word, "driven." The final D might be Development or Design, but both words tend to be used interchangeably. The initial letter stands for Test, Domain, Model, Responsibility, Lean, Feature, Pattern, and Service, respectively.

According to Bob Martin, Responsibility Driven Design is the grandmother of DD as it appeared as the title of a book–by Rebecca Wirfs-Brock - in the 1990's.

Each of the DD's promotes one aspect of development to a position of privilege. It becomes THE thing that must be right if you are to have any hope of delivering software.

• If you fail to understand the domain, your software will always be an abrasive intrusion.
• If you modularize your software on any basis other than a "service" you will miss out on reuse and cool Web technologies.
• If you don't get the distribution of responsibilities across a set of objects correct you will fail to realize the benefits of OO and create overly coupled and brittle software.
• If development is not predicated on automated testing you will be forever mired in a swamp of un-fixable code.

Overstatement? Sure! But, each example does contain the essence of the claim made by each approach.

The DD's do not make the claim of exclusivity: only privilege. They do claim to be able to work together.

There is certainly value in adopting any or all of the DD's (except perhaps Model Driven, the latest incarnation of automatic programming) - each does address a pragmatic issue that makes software development difficult. Each offers a starting point and a perspective that quickly leads to improved software development. Agile development is as powerful as it is, in part, because of ‘D' adoptions.

Unfortunately, none of the ‘D's directly confronts the Essential Difficulty of software as described by Fred Brooks in the well known paper, "No Silver Bullet."

Normal 0 false false false MicrosoftInternetExplorer4

The essence of a software entity is a construct of interlocking concepts ...

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. [Brooks 1986]

If agility is to realize its full potential it must incorporate practices and principles that facilitate the development, refinement, and sharing of conceptual constructs.

Fortunately, agility already incorporates such principles and practices and is primarily in need of a contextualizing framework to fully understand and exploit them.

YDD is intended to provide that kind of framework, but it is first necessary to explore a bit what is meant by "conceptual construct" and "theory."

Conceptual Construct = Theory
A software entity is a complicated thing. It is embedded in, and must satisfy the demands of, a complicated deterministic system (the hardware) and a complex system consisting of cultures, applications, users, laws, environments, and change. According to Brooks, "Software entities are more complex for their size than perhaps any other human construct."

To have a conceptual construct of the software you are to build means having a mental "model" of all the interlocking and interacting parts - both the parts in the machine and those in the world-at-large.

The idea of a mental model is problematic. As Brooks notes: "Software is invisible and unvisualizable." Moreover,

"As soon as we attempt to diagram software structure, we find it to constitute not one, but several, general directed graphs superimposed one upon another. ... The graphs are usually not even planar, much less hierarchical ..."

Even a deep understanding of mathematics and physics will not help.

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

Much of the complexity that [the software developer] must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform. These differ from interface to interface, and from time to time, not because of necessity but only because they were designed by different people, rather than by God.

It is no wonder that Brooks considers software development to be hard. Given all the obstacles to the formation of adequate conceptual constructs–and the absolute need for such a construct - what reason do we have that it is even possible?

We do have some existence proofs–software entities that clearly are grounded in a well-formed conceptual construct. We also have existence proofs that both individuals and groups can formulate and share conceptual constructs of the same order of complexity as any software entity. I speak, of course, of things like culture and cosmology.

As humans we have internal conceptual constructs of the universe, our place in it, and of all the interactions and consequences of those interactions that take place in it. We can successfully apply this mental model even though it would be impossible to articulate it, model it, or reduce it to a set of formal rules.

We also have several prominent developers who hold that the establishment of a conceptual construct is the essential act required to successfully develop software. Peter Naur ("Programming as Theory Building"), for example, holds that software development IS conceptual construct creation–what he calls theory building.

"I shall use the word programming to denote the whole activity of design and implementation of programmed solutions. ... the activity of matching some significant part and aspect of an activity in the real world to the formal symbol manipulation that can be done by a program running on a computer.

... If it is granted that programming must involve, as the essential part, a building up of the programmer's knowledge, the next issue is to characterize that knowledge–the programmer's knowledge should be regarded as a theory, Very briefly, a person who has or possesses a theory in this sense knows how to do certain things and in addition can support the actual doing with explanations, justifications, and answers to queries about the activity of concern. [Naur 1985]

Theory Building and Agility
"Theory building" is also known as "design" and design has always been a bit problematic for agile developers. The first problem is the traditional belief that design is something done "up-front;" before any actual work (testing and coding) can take place. But this need not be the case. As Kent Beck pointed out, "XP does include design, you design a little, code a little, and repeat."

Another problem (for all developers, not just agilists) is the fact that design is not a subject most are comfortable with. Design is not part of typical computer science curricula and design requires knowledge of far more than the structure and modeling of programs.

A third problem arises from the fact: software development is a team activity. This means that the entire team must, somehow, share the same theory!

Agile already has the means and the tools to address these difficulties.

Starting with a kind of philosophical foundation. Alistair Cockburn included Naur's essay on theory building as an appendix in his book, Agile Software Development. Cockburn also noted how his understanding of agile principles, values and practices were colored by his reading of Naur. He was not alone, most of the other agile innovators had similar insights, they just remained implicit.

Numerous agile practices directly support the creation of theory, of conceptualization. Conceptualization is theory building; and it begins with telling stories about the world, the things in the world, what they do, why they do it, what other things they associate with and why and when they form those associations. This is the product backlog.

Metaphor is our most powerful tool as we seek to describe and understand what we do not know in terms of what we do know. Kent Beck once advocated the System Metaphor as a formal practice but has generalized the role of metaphor throughout the development process - even down to metaphor as a principle for proper naming of classes and methods.

Story cards are an "invitation to a conversation" and evocative memory of conversations had. On-site Customer insures a continuing evolution of theory as it is reformulated in terms of what is built - theory, like working software, is developed incrementally. At each stage the existing theory is evaluated and revised in terms of new understandings embedded in what we have built.

Most of the practices also support the development of consensual team theory building. Daily stand-ups, big visible charts, Collective Code Ownership, wall-to-wall white boards, communal work space, Whole Team–all of these ensure that everyone involved shares the same theory.

Reinhard Keil-Slawik is the author of a chapter in Software Development as Reality Construction, edited by Christiane Floyd. In that chapter he shows how "artifacts," like models on whiteboards, function as a kind of "external memory" for the humans working together developing software. Not only is this memory external–it is shared. Not only is it shared, it is integrated to the humans that create and use that external memory. External memory is not documentation and it cannot be passed to others as a means of sharing the theory. This fact is also noted and documented by Naur, and accounts for the difficulties encountered when developers are asked to alter or maintain the work of others.

The foundations for a theory driven approach to software development are clearly present in agile as it is practiced today. For YDD to become a reality those practices must be rearticulated stressing how they contribute to theory development and how they directly address what Brooks' called "The Essential Difficulty" of software development.

Existing practices must be augmented by techniques, heuristics, and practices that stress incorporate diverse subjects (e.g. the ethnography of ‘thick description' from anthropology, metaphor-based reasoning, and naturalistic philosophy) into the family of agile values, principles, and practices.

YDD will also need to address and overcome several obstacles.

Obstacles to Realizing YDD
There are two major obstacles that need to be addressed by YDD. Both transcend Agile and apply to all approaches to developing software.

One: almost all software developers today already operate on the basis of a "default theory." Unfortunately this theory is wrong and counterproductive. Briefly stated, developers, especially those trained in classic computer science programs, use the implicit theory, "the world is a computer."

Things learned from mathematics and the physical sciences allowed, and continue to allow, us to make advances in the computers and the basics of computing. However, as Brooks noted, those same tools do little to help us understand the essential complexity of the applications and systems that employ computers. This is the primary reason I objected to Model Driven Development earlier. MDD (or MDA) insists on projecting the same formalisms onto the world at large as are used inside the computer, and this is a fundamental mistake. It is nevertheless a prevalent, though tacit, presupposition of most software developers.

Two: Software development is not a production process!

Naur wrote his paper on theory in large response to the prevailing use of the production metaphor of software that justified all the effort in the areas of project management and the devising of ever more formal and repeatable development methods.

Management would like to believe that software development can be organized and controlled like any other manufacturing process. They would like to believe that developers are interchangeable commodities to be acquired at the lowest possible cost from anywhere in our "flat world."

Agile appeases management with practices like estimates, planning games, and burn-down charts. These practices have a dual role: helping us develop our theory of the world, how our software works, and what we know and can do on the one hand; and giving management the illusion of control over a rational process. Unfortunately, most of the time it is the latter role that is stressed.

The adoption of Lean by the Agile community is also a double-edged sword. Lean values are deeply compatible with agility. Too many lean practices are misinterpreted as techniques for improving production.

Conclusion
Any approach to software development that fails to address Brooks' essential difficulties, and THE essential difficulty of the conceptual construct, cannot offer more than a band-aid improvement over the current state-of-the-practice.

Agile values, principles, and most of its practices (plus Lean values) provide the most promising foundation to date for tackling the essential difficulties. As practiced, however, Agile/Lean is failing to realize its potential. What we have today is an excellent first step - but we need more. YDD is a way to embellish, enhance, and refactor Agile into a much more powerful (yet simpler and more elegant) approach.

This short article is a "call to arms," an initial argument for needing YDD. It is not a full exposition, as the approach is still under development. You can expect to see a full treatment of this subject in the near future. You can also be a part of the effort to establish this approach. Contact the author for more information.

---

About the Author
Dave West has been a professional software developer for too many years to count, in roles ranging from programmer to Director of IT.For the past fifteen years he has split his time between consulting and academia.Currently Professor of Business, Innovation, and Technology at the College of Santa Fe: formerly at New Mexico Highlands University and the University of St. Thomas.His own education was an eclectic mix that included Asian Philosophy (undergraduate at Macalester College), Computer Science and Cultural Anthropology (graduate degrees at University of Wisconsin-Madison) and a Ph.D. in Cognitive Anthropology (also at UW-M).He is the author of Object Thinking (MS Press Professional, 2002) and is nearing completion of a second book on “reinventing software development.”Any expertise he possesses is focused in the areas of objects, agility, design, innovation, and change.