the imperial clothing crisis
Marx and Engels [ Marx-1848 ] were ahead of the object crowd:
The history of all hitherto existing society is the history of class struggles.
A spectre is haunting software development - the spectre of design. There is a growing recognition that design is the essence of software development and, consequently, the missing link, or ingredient, in much software. Much of what has passed for design or design movement has just been theory without practice, rhetoric or fashion - design is the pictures that you draw, reuse is a principal foundation of object orientation, component-based development leads to flexible architectures. None of these views is particularly useful, and they can sometimes be considered harmful.
design considered useful
In software, design has often been seen as filler between a misunderstood concept of analysis and a view of implementation reminiscent of industrial manufacturing. Design has been seen as a phased activity divorced from code, associated with the power of pure thought and the production of never-to-be-quite-built blueprints. An air it retains today even in many iterative and incremental lifecycles - the old divisions are often still there, just more thinly sliced.
In the past, design has also suffered from an image problem: it has often been branded formal - sometimes, too formal - equating it to a set of rules, conventions and etiquette, and therefore, by association, stuffy, impenetrable and elitist. To counterbalance - and hopefully displace - these rigid views, design can be considered an endeavour that is continuous and embraces both the code and the conceptualisation of the system. We can use models - drawings or prototypes - to demonstrate things that cannot be shown directly or conveniently in code. But design is a federal concept, and such models may be a part but they are not - except in hegemonies like RUP (the Rational Unified Process) - the whole. It is pragmatism that has acknowledged a more inclusive definition of design, placing it at the centre, recognising that design is indeed formal, but in the same sense that code is formal and in the sense of the word that means "concerned with form".
What properties should we expect of a good design? The influential ten-volume de Architectura , by first century BC Roman architect Marcus Vitruvius Pollio, suggested that all construction should possess "strength, utility and beauty" (sometimes translated as "firmness, commodity and delight"). We can relate these directly to code: robustness is clearly something that we value; use is the measure of what is built; and, yes, aesthetics matter, although a full exploration of beauty is outside the scope of this article.
I don't want to go on record as saying that a two-thousand year old book on the built environment has the last word on what software design is all about, but you have to admit that it's not a bad start. Are there other things that we should be looking for that could not be re-shelved under one of those headings? Quite possibly, but it would be fair to say that the quest for goodness in design has attracted a number of camp followers - wannabes that aspire to inclusion on that A-list of desirable properties. Not all are bad, but some have made more progress than they should - cyberspace is right next door to hypespace - and it's time we cleaned up a little. In particular, I would like to think that the grim reaper is slowly stalking the mantras of reuse and flexibility .
Reuse has proven to be a false idol to worship. It is at best an illdefined term, and at worst an incorrect one. Moving to objects or components because of reuse is like buying a car because the advert says it's the best: for some people it is wishful thinking, for others it is grasping at straws, and for a few it is simply gullibility. What exactly is reuse ? If we define it with respect to our experience, it would seem that reuse is often a time waster, an obfuscator, and more of a problem than a solution. I made this claim recently on a course and got a round of applause. What was telling is that most of the applause came from the managers in the group; the precise target audience for much of the reuse hype that kicked off in the late 1980s and is still rolling today.
The word flexible is like reuse: it should alert you that something nebulous is probably up. Classes and functions are not designed to be flexible, they are designed for a purpose: flexibility is not a purpose, nor is it either a quality or a quantity; it is a bucket term, a catch all, snake oil.
All of this is not to say that we cannot reuse software or that software cannot be flexible. Far from it, it is just that in common parlance they are either vague hand-waving terms ("Our architecture is flexible" - what does that mean? Can it bend over backwards so that it touches its own heels?), or incorrect ("We reused the third-party library" - err, doesn't that just mean you used it... for the purpose for which it was intended?). Without qualification these words mean nothing - but have tremendous power to mislead - and with qualification they do not turn out to be the gleaming foundations of a discipline for software development. Like the emperor's new clothes, there is not much there. When it comes to answering the question "What is important in design?" we should perhaps avert our eyes and look elsewhere.
simplicity before generality
A point I am often at pains to make is that minimalism in software is not about throwing everything out leaving you with nothing. That is nihilism. The emperor needs some clothes, just not too many, and obviously not none. And, of course, it is best if the clothes fit - a few clothes that fit is better than an apparent wealth of clothing that does not.
However, my real motivation in bringing the emperor story into all of this is not specifically the emperor's attire - or lack thereof - as related to a pragmatic interpretation of minimalism, but to point the finger and declare "This has no clothes, there's nothing there!". To be precise - as you may have already established - two fingers. The two virtual properties of reusability and flexibility are twinned with generality , and from generality flows a river of good intentions so deep you could - and many do - drown in it. It might at first glance be assumed that reuse would form the cornerstone of a minimalist philosophy of software development, but the opposite transpires.
I confess that the promise of reuse was never one that attracted me, and was a topic that I never felt entirely at home with, at least not in the sense that it was most talked about. My traditional stance was that reuse was a social issue not a technological one, a matter of culture rather than of mechanism - mechanism could assist but it could never cause. This view was still reachable from the published party line, although in truth most others in the party did not follow the line either.
In recent years I have called into question many of the buzzwords that pass for communication (but pass all understanding). The realisation has crept up on me that even the mainstream nonmainstream view, so to speak, is inaccurate and insufficient. These days, I adopt a more republican stance: when it comes to clothing, so to speak, we should not even be talking about the emperor. It doesn't help. Reuse is not the main challenge facing software engineering; typing is not the main bottleneck in software development (which means that most third-party code generation tools are actively solving the wrong problem); bureaucracy is not the missing link in the development process. Sometimes the shine is taken off our ability, as a profession, to solve problems by a frequent and uncanny knack of identifying the wrong problem to solve. Let's try to simplify before we generalise.
The following is from an email I sent Bruce Eckel following a request on his list for design principles to include in his book, Thinking in Patterns [ Eckel ]:
Simplicity before generality: A common problem we find in frameworks is that they are designed to be general purpose without reference to actual systems. This leads to a dizzying array of options that are often unused, misused or just not useful. However, most developers work on specific systems, and the quest for generality does not always serve them well. The best route to generality is through understanding well-defined specific examples. So, this principle acts as the tiebreaker between otherwise equally viable design alternatives. Of course, it is entirely possible that the simpler solution is the more general one.
The slightly flippant tone of the last sentence may hide my degree of conviction: it is not just that it is "entirely possible", it is actually "quite likely".
Many things that are designed to be general purpose often end up satisfying no purpose. Software components should, first and foremost, be designed for use, and to fulfil that use well. Designing for all seasons is both difficult and not always desirable, a realisation that helps explain the small markets for thermal bikinis and Ford Edsels, as well as the challenge of designing general-purpose software components.
Reflecting both on my work in library and framework development and on my role as a user of such commodities, I have seen the strong temptation and wasteful consequences of general featurism. I have also seen a more restrained approach bear fruit. It may be a cliché, but less really can be more.
Generality is not, of itself, necessarily bad, but we can often identify the odour of speculative generality [ Fowler1999 ]:
Brian Foote suggested this name for a smell to which we are very sensitive. You get it when people say, "Oh, I think we need the ability to this kind of thing someday" and thus want all sorts of hooks and special cases to handle things that aren't required. The result is often harder to understand and maintain. If all this machinery were being used, it would be worth it. But if it isn't, it isn't. The machinery just gets in the way, so get rid of it.
Generality should equate to simplicity and simplification. Generalisation can be used as a cognitive tool, allowing us to reduce a problem to something more essential, a clearer abstraction that offers greater compression [ Henney2001 ]. However, too often generalisation becomes a work item in itself, and pulls in the opposite direction, adding to the complexity rather than reducing it. The initial sweetness of a general solution can become overwhelming as it grows, to the point that we feel like we are drowning in syrup.
Design is compromise, and all flexibility is a double-edged sword. Many people mistakenly see design decisions made in the name of flexibility as win-only situations, a narrowness that belies reality. If we equate flexibility with degrees of freedom, then the degrees of freedom in a design should be reasonable - which I mean that in the deepest sense of the word: based on reason. In pursuit of arbitrary flexibility you can often lose valuable properties, accidental or intended, of alternative designs [ Petroski1999 ]:
In the mid-twentieth century it became the fashion in library architecture to design buildings as open-floored structures in which furniture, including bookcases, could be moved at will. The Green/Snead Library of Congress bookstack that six decades earlier had been declared "perfect" was now viewed as disadvantageously locking a stack arrangement into the configuration of its construction. In the new approach, reinforced concrete floors carry the loads of bookshelves, so that they can be arranged without regard for window placements. This apparently has the appeal of flexibility in the light of indecision, for planners need not look at the functional and aesthetic requirements of their space and its fittings with any degree of finality; they can always change the use of the space as whim and fashion and consultants dictate. It is unfortunate that such has become the case, for it reflects not only a lack of sensitivity to the historical roots of libraries and their use but also rejects the eminently sensible approach to using natural light as a means of energy conservation if nothing else. There is little more pleasing experience in a library than to stand before a bookshelf illuminated not by fluorescent lights but by the diffused light of the sun.
And speaking of libraries, it is worth noting that one the strange things about a so-called reuse library is that it's one of the few libraries people only seem to deposit things in but never take things out. A more honest term is reuse repository .
A more pragmatic and minimal design style does not mean writing code that hugs its assumptions so closely that only major surgery will separate the two in the event of change. It is not about hardcoding everything: it is about both sufficiency and finding the right amount of space between the elements of your solution, offering the right amount of slippage or wriggle room. The challenge of design is in seeking and maintaining local minima of sufficient simplicity with sufficient generality that the integrity of the design is not easily disturbed, and the energy required to adapt to change is proportionate to the degree of change.
Generic programming often provides good examples of sufficient generality. Note that generic is not the same as reusable : something may be generic to express the simplest and most stable model. But at the same time, genericity can be a hard tap to turn off, whether expressed through template parameters, function arguments or an interpreted interface. It is tempting to create some kind of final solution - a killer class template that is parameterisable beyond belief, or indeed comprehension. In practice such parameterisation severely reduces the utility of code. One size does not fit all: I don't shop at a single place to buy all of my goods - food, cars, etc - and although this is possible, one is left with a sense of diminished quality through lack of appropriate specialisation. I get better fruit from the local grocer. Likewise, restaurants: if they try to cater for all types of food, they do so blandly and uniformly, squeezing out the variety and standardising the experience.
ex libris
The benefits of libraries are different to those of reuse repositories. The idea that an arbitrary piece of code is a candidate for reuse differs from the idea of taking a piece of code and generalising it - promoting it through practice and empiricism- into a library, or conceiving of a code library that scratches a particular design itch, expresses a particular design idea, refactors a common code chunk.
You cannot reasonably refer to reuse in the context of a library - "We are reusing the AWT." "Oh, and what did you do with it the first time around?" - because there are only three things you can do with a library: use it; misuse it; not use it. Some might attempt to leverage a library, but it transpires that this is just a neologistic circumlocution for use. (Aside: In addition to a popular suggestion that its use as a verb should be banned, there could be an additional fine on native English speakers pronouncing it "l evv erage" if their common cultural pronunciation is "l eev erage". You could probably raise a lot of good money for charity with an office swear box filled on such jargon.)
Libraries offer value based on use, not reuse. And library architecture, whether in a loose confederation of parts or the more tightly knit community of a framework, is based ultimately on modular concepts. But what kind or scale of module forms the basis of design? Often use at the level of the small is dismissed as insignificant. And yet this is the level at which most libraries and infrastructure projects have been most successful. The view that we are striving ever upwards, always building neatly on the layer below, moving towards a greater object society or component order seems to have ensnared a mindshare. It has created a mantra all of its own: once I worried about structured control flow; then I worried about my classes; and then I worried about components; but now life is easy, all I need to worry about is how to interface and tune these off-the-shelf distributed systems - I no longer need to worry about any details. It's not true in software and it's not true elsewhere [ Salingaros-2001 ]:
A free design process that allows for numerous subdivisions permits mathematical substructure on many different scales. By abandoning an empty modularity, one has access to solutions based on a far richer approach to design that creates visually successful buildings. Art Nouveau architects like Antoni Gaudí used small modular elements (such as standard bricks) to create curved large-scale structures. This freedom of form contrasts with those instances where a building reproduces the shape of an empty rectangular module. The small scale can link to the large scale mathematically, because scaling similarity in design is a connective mechanism of our perception. Therefore, the materials can and do influence the conception of the large-scale form, and the larger a module, the stronger the influence. When one chooses to use large, empty rectangular panels, these will necessarily influence the overall building; often implying a monotonous, empty rectangular façade.
Uncannily, this sounds like many contemporary large-scale component-based architectures. It is not modules but inappropriate modularisation that causes problems: it can be as bad as the absence of modularity. On the one hand you have a monolithic slab of code and on the other lots of prefabricated slabs that somehow just don't seem to quite fit or make sense together [ Salingaros-2001 ]:
There are arguments to be made in favor of modularity, but not for the way it is used in many buildings. If we have a large quantity of structural information, then modular design can organize this information to prevent randomness and sensory overload. In that case, the module is not an empty module, but a rich, complex module containing a considerable amount of substructure. Such a module organizes its internal information; it does not eliminate it. Empty modules, on the other hand, eliminate internal information, and their repetition eliminates information from the entire region that they cover. Modularity works in a positive sense only when there is substructure to organize.
Undifferentiated modularity seems to be a genuine problem. We should work with more than one unit of modularity, and many programmers do so successfully: method, class, package, component, etc. Software design, and therefore architecture, is recursive. The reason many people move to object and component technologies is precisely because of the many and varied levels of granularity on offer - a better fit to their grasp of the problem and expression of a solution, a finer level of control over the detail and its relevance, better choice of abstraction, better resulting compression. Rather than the brusque and FORTRAN-esque levelling of program then function, we have a view that can zoom out or in to the system to the level that we find appropriate to understand a particular behaviour or solve a particular problem.
And, to be effective, a good grasp of modular diversity must be coupled with an appropriate sensibility, an understanding of its intent and reach [ Gabriel-2000 ]:
The real problem with modular parts is that we took a good idea - modularity - and mixed it up with reuse. Modularity is about separation: When we worry about a small set of related things, we locate them in the same place. This is how thousands of programmers can work on the same source code and make progress. We get in trouble when we try to use that small set of related things in lots of places without preparing or repairing them.
The module, at whatever scale, is one of our best tools. But identifying good modules is hard; software development is a matter of design. It may seem almost counterintuitive, but a design born of a minimal and pragmatic approach will often have more modular parts than one that does not. However, the parts will not all serve the same purpose - and nor will they serve any arbitrary purpose - and they will not all be the same size.
commodity
So if it is not reuse and flexibility, what role then do libraries and related infrastructure play? Commodity - which can be found, by happy coincidence, in the alternative translation of Vitruvius' three essential architectural properties. A commodity is quite a different thing to something that is reusable. A commodity is something of value and of - and for - use. A commodity defines a stable platform, something that can be assumed or taken for granted. A commodity is a product that has been built (or mined and refined) intentionally, rather than an accidental but serendipitous artefact. In real world terms we would never confuse the quite different ideas of commodity and reuse. Whilst there is certainly overlap, they are not even mistakably synonymous.
We can see that developing a commodity is quite a different undertaking to developing reusable code. A lot of per-project code that is designed to be reusable simply isn't, and is often borderline useable. The open pursuit of reuse is unfocused and adds an inappropriate overhead to some projects, often to the point of compromising quality or schedule. The return on investment - time, effort, complexity, money - is often not recouped. Far better to create code that is fit for purpose and fits with its purpose. If you follow some of the common practices for decoupling - easier testing, less impact of change - it is more likely that the code will see more general use, perhaps even as a commodity. Some projects recognise that it will cost them extra to get some kind of reuse, but if they replaced the word reusable with commodity they might reconsider how much extra was really needed to be effective.
In a single project you don't have reuse if a piece of code is used in more than one place, that's just what should be going on. This is just good local design: usage with minimum duplication. A definition of reuse based on the simplistic and unqualified definition of "use more than once" would be trite and quite useless - what value would be gained by replacing "function A calls function B " with "function A reuses function B "?
If you view a system in terms of layering there is an interesting relationship between the use of refactoring and the balance of logic in the system [ Collins-Cope-2000 ]:
Refactoring visibly lowers the centre of gravity of the application by finding the commonality and factoring out the difference.
This is like annealing. Refactoring provides enough energy to a system for it to relax into a new and more comfortable state, a new local minimum. The effect of refactoring commonality is to tame the complexity of your system. Repeated tempering, with a conscious effort to reshape, is more likely to move code towards commodity than a vague 'plan' or 'strategy' based on reuse.
And, as already noted, if you use a library or framework then this again is not reuse, this is the idea of commodity and platform. Features migrate into platforms over the years, e.g. threading, networking, GUIs, etc. Using an application server is not reuse, it is use of a commodity as was intended by its design.
Most concepts of reuse are invalid or unachievable in practice. Therefore, by definition, most reuse strategies are destined to fail. Consider the inherent contradictions or muddled goals that sometimes pop up on company technology adoption plans: "We will start with a pilot project to demonstrate code reuse on a threeprogrammer four-month development". Often what started out as the path of least resistance can become the path of least convenience.
accidentally on repurpose
It appears, in part, that the problem is one of vocabulary: we are misusing words. For some reason, reuse is seen as a more glamorous and elevated term than use . We can reduce most uses of reuse to something more precise: use of a commodity, such as a library, or the result of refactoring to avoid duplication of logic within a system. We're in programming not marketing: precision matters. At least refactoring now has a certain cachet. (Perhaps there's something going on with the prefix re -? Rework ? Recode ? Retest ? Hmm, then again, perhaps not.)
So how did we end up with the reuse groupthink? And why has it so often been allied with inheritance in object-oriented approaches? Both hindsight and foresight suggest that it is a damaging mindset that upsets both schedules and design. We know that hype merchants are responsible in part, but the cause can also be traced to the more honest confidence and ebullience of expert individuals [ Meyer1997 ]:
Here are the most important external quality factors, whose pursuit is the central task of object-oriented software construction....
Reusability
Reusability is the ability of software elements to serve for the construction of many different applications.
A few choice words come to mind when I read this, even without the exploration I have just undertaken of what reuse is not. However, a quick count to ten and I can respond with something a little more moderate: there is no such thing as reusable software, only software that has been reused. This response is adapted from the porting maxim that "there is no such thing as portable code, only code that has been ported". However, unlike reusability, portability is actually a quality that has some meaningful quantification, both empirically and theoretically, and can be reasoned about without resort to theological debate.
Whilst the definition given of reusability is not necessarily a bad one, the problem is that it is listed as being a core goal and activity of object development. The problem is then compounded by sewing up this tidy view of the world with the following mythtake [ Meyer1997 ]:
Progress in either reusability or extendibility demands that we take advantage of the strong conceptual relations that hold between classes: a class may be an extension, specialization or combination of others. We need support from the method and the language to record and use these relations. Inheritance provides this support.
The perspective given, which some might consider as charmingly naïve and others would view as downright misleading, is countered by a wealth of theory and practice that suggests that such an approach to development, and especially such a view of inheritance, is unsound. Commentary reflects this [ Murray1993 ]:
A myth in the object-oriented design community goes something like this:
If you use object-oriented technology, you can take any class someone else wrote, and, by using it as a base class, refine it to do a similar task.
Even in the design of commodity items such as class libraries - commonly and mistakenly equated with reuse - inheritance is not necessarily viewed in glowing terms, as witnessed by Martin Carroll and John Isner, designers of USL C++ Standard Components [ Gabriel1996 ]:
We take the minimalist approach to inheritance. We use it only when it makes our components more efficient, or when it solves certain problems with the type system.
We do not intend for our components to serve as a collection of base classes that users extend via derivation. It is exceedingly difficult to make a class extensible in abstracto (it is tenfold harder when one is trying to provide classes that are as efficient as possible). Contrary to a common misconception, it is rarely possible for a programmer to take an arbitrary class and derive a new, useful, and correct subtype from it, unless that subtype is of a very specific kind anticipated by the designer of the base class. Classes can only be made extensible in certain directions, where each of these directions is consciously chosen (and programmed in) by the designer of the class. Class libraries that claim to be "fully extensible" are making an extravagant claim which frequently does not hold up in practice.... There is absolutely no reason to sacrifice efficiency for an elusive kind of "extensibility."
Yes, reuse does happen, but it is not in the tidy centralised or library-governed vision that you may have been sold, framed originally by many High-Modernist Object Gurus. It is normally a more ad hoc, grassroots, opportunistic and - yes - cut-and-paste affair [ Raymond2000 ]. It has a haphazardness to it that belies the feed-forward, deterministic nature of many software development lifecycles that claim to address reuse. It is not sufficiently deterministic or controllable to form the basis for project planning or an economic model of software development. Reuse strategy is practically an oxymoron. How can you have a reasonable strategy based on good luck? We normally insure against accident, not for it.
Reuse in the outside world is also a less organised affair [ Brand1994 ], sometimes built almost purely on compromise. It is normally concerned with recycling and repositioning; not necessarily the glamorous vision of rapid development, but a matter of development by necessity and disparate parts, subdivision and differentiation. The charm of something that is reused often arises from its accidental properties rather than from a reduced Cartesian order.
But I do not wish to misrepresent the object community. For most of us the reason to adopt an OO or related style has been about the qualities that it gives development, not the quantities. You can say what may actually amount to the same thing in two radically different ways: "I use X because it is more expressive, allowing me to articulate a more eloquent design" versus "I use X because it makes me more productive". The former is about the individual, the human, and the latter is about an automaton; more cog than cogitation. I tend to find surveys and articles that talk about productivity do so from a pseudoscientific point of view. Such dehumanisation is also found in the rebranding of programmers as plug-and-play resources . Being a resource is not the same as being resourceful: oil, coal and tin are natural resources; a third-party code library is a software resource; even our skills and experience can be considered knowledge resources. However, we, as individuals, are not resources: it is our use of resources that makes us resourceful. The productivity of a resource does not sit on one side of an equation with reuse on the other. It would be a tidy but deceptive fiction.
But I digress. Where any form of reuse can be of assistance is in the reduction of a problem to something similar, something familiar. Where reuse is most prevalent is in our knowledge, the use of experience to resolve similar and out-of-context problems. In each case, the reuse is a matter of adaptation, repurposing, learning. It is distinctly unmodular, and rarely preplannable at a detailed level. But the absence of a fixed and fine plan does not denote chaos, any more than the omission of a detailed leaf plan filed in your local town hall suggests that trees do not represent some form of order.
So remember, design is central to software development and most of the worthwhile reuse and flexibility in software development comes from your side of the keyboard.
references
[Brand1994] Stewart Brand, How Buildings Learn , Phoenix, 1994.
[Collins-Cope-2000] Mark Collins-Cope and Hubert Matthews, "Let's Get Layered: A Proposed Reference Architecture for Refactoring in XP", XP 2000, http://www.oxyware.com/Publications.html .
[Eckel] Bruce Eckel, Thinking in Patterns , work in progress that can be downloaded from http://www.mindview.net/Books/TIPatterns/ .
[Fowler1999] Martin Fowler, Refactoring: Improving the Design of Existing Code , Addison-Wesley, 1999.
[Gabriel1996] Richard P Gabriel, "Patterns of Software: Tales from the Software Community", Oxford, 1996.
[Gabriel-2000] Richard P Gabriel and Ron Goldman, "Mob Software: The Erotic Life of Code", delivered by Richard Gabriel as a keynote at OOPSLA 2000, http://oopsla.acm.org/oopsla2k/postconf/Gabriel.pdf .
[Henney2001] Kevlin Henney, "Minimalism: Omit Needless Code", Overload 45 ,
[Marx-1848] Karl Marx and Friedrich Engels, The Communist Manifesto , Phoenix, originally published 1848.
[Murray1993] Robert B Murray, C++ Strategies and Tactics , Addison-Wesley, 1993.
[Petroski1999] Henry Petroski, The Book on the Bookshelf , Vintage, 1999.
[Raymond2000] Eric S Raymond, The Cathedral and the Bazaar , O'Reilly, 2000, http://www.tuxedo.org/~esr/writings/cathedralbazaar/ .
[Salingaros-2001] Nikos A Salingaros and Debora M Tejada, "Modularity and the Number of Design Choices", Nexus Network Journal 3(2), 2001, http://www.nexusjournal.com/Sali-Teja.html .
