Thinking about "reuse"

Call me a heretic if you like, but it seems to me that "reuse" - that Holy Grail of software coders - seems to be a false idol. I'm not alone in this, Kent Beck and the Extreme Programming crowd have been denying reuse for several years in their efforts to "do the simplest thing that works" and Kevlin Henney [ Henney2002 ] has questioned some assumptions too.

What is happening here? Well for me the serious doubts set in last year when reviewing Generative Programming by Czarnecki and Eisenecker [ Czarnecki-2000 ]. For me, their cure was worse than the disease!

Further, times are changing and so are the economics of software. We are no longer confined to traditional models of reuse. The development of web services means there are new models appearing. Got a good library of chemical simulators? Why not SOAP enable it and charge per use from your web server? And why not experiment with new ways of funding development? Chris Rasch has suggested using the bond market to fund software development [ Rasch ].

How do we get here?

The appeal of reuse is obvious - especially if you have to pick up the bill for software development. If we could just reuse what we have already done…

This was one of the early promises of object-orientation. The idea was that objects represented some kind of entity, which would be applicable in more than one program. As Czarnecki and Eisenecker point out, the idea that reusable objects would simply drop out of a system development proved false.

It transpires that producing an object usable in more than one system is very difficult, in some cases more than double the effort. Czarnecki and Eisenecker introduce the idea of "designing for reuse" and "designing with reuse". (I agree with them on everything up to this point. Beyond this, my problem with Generative Programming is that I don't see how their solutions can be used in a real world environment. Much of the code they produce comes dangerously close to being unmaintainable.)

This division seems to recall ideas of the "problem domain" and "solution domain": in analysing the problem domain we identify and create objects which could be reused, when working in the solution domain we take these prefabricated modules to create our solution.

It appears that finding the right granularity for reuse is somewhat difficult: function level reuse gave way to object reuse and object reuse has given way to component reuse. (The word module is so malleable that I'm deliberately avoiding it here.)

All the time reuse seems almost within our grasp. The truth is, as an industry we reuse more code than ever before. We make extensive use of code libraries, code-generating wizards (which are basically a form of meta programming), cut-and-pasted code, and packaged applications with macro languages or automation interfaces. So maybe we are not giving ourselves credit where credit is due.

So, why do I say reuse is a false idol?

There are two things wrong with reuse as we generally talk about it. First, I think we need to look at the costs of reuse. Second, reuse is a vague term; not only does it depend on who is using it, but it covers a multitude of ideas.

The cost of reuse

Sometimes costs are obvious: Joe will be tasked with creating a reusable library of widgets, so the cost is Joe's salary for the six months of development. More often the cost of reuse is hidden. As a developer you will decide to make your object slightly more generalised than it needs to be, or you will allow for some obvious enhancement which is not yet required. In effect you are doing more work than you need to, so the cost is perhaps 20% of your salary.

In either case you should have some idea of when the payback will come - a few weeks? months? Are you sure the payback will come at all? Even if it does come, who will see the benefit?

In addition to the financial cost there is the cost of work foregone. If a less generalised object had been produced in half the time you would be free to move onto the next work item. Economists call this "opportunity cost", it is the cost of not doing something else.

Building more generalised code, frequently (but not always) leads to more complex code. By its nature this is more prone to errors and more difficult to maintain.

Further, can we be sure that the additional as yet unused functionality is tested as well as the functionality that is to be used? If I create a Widget class with three methods, and then I decide to add an extra three methods (which I'm sure we are going to need in future) do I test the second three as completely as the first three?

There are other issues with this scenario:

• If we change the class before we get to use the three extra methods do we need to change them?

• If Fred takes over the class from me, won't he be wondering what is going on?

• If we need to port the class, or rewrite it, or refactor it, we have twice as many methods to work with than we actually need.

If these methods aren't used we have wasted our time writing them. Even if they are used, between creating them and using them we have to support them, they represent the "cost of carry" - to borrow another term from economists.

Finally, consider those three methods again. When they are finally used they aren't even being reused - they are being used for the first time! Sure, the Widget object is being reused but are we putting the cart before the horse here?

What does reuse actually mean?

The next problem with reuse is the very word itself. In part this varies according to your perspective. If I create my Widget class for program Foo, and I use it again in program Bar, then I have reused it because I have used it before. But if you were to write program Bar are you reusing Widget or just using it?

Reuse in the car industry

The car industry has many differences to the software industry. Most obviously in production costs; once your magnum opus software is complete production costs are minimal, just the cost of CD duplication or server bandwidth. In the motor industry completing the design is but the first stage: production costs dominate.

In recent years just in time assembly lines and lean practices have revolutionised car production, while increased competition between manufacturers has led to increased product diversification and a reduction in the lifetime of models. Consequently, competitive advantage rests with those manufacturers that can produce a diverse range of frequently updated models. This places the onus back on design.

In an effort to reduce design costs manufacturers have turned to reuse: within seven years Volkswagen moved from making 30 different models on 16 floorpans to 54 models on 4 floorpans [ The Economist , 27th April 2000], look how the Passat is an Audi A4, is a Skoda Octavia.

With this background Cusumano and Nobeoka [ Thinking Beyond Lean , Freepress, 1998] have looked at the design process, and in particular how design reuse is enabling car companies to meet this challenge. Although Thinking Beyond Lean was published in 1998 its lessons are not dated and provide valuable insights into reusing design and transferring technology between projects. Almost as vindication, since 1998 we have seen platform sharing as a major factor behind the Renault-Nissan, Mercedes-Chrysler-Mitsubishi and Ford-Mazda-Volvo link ups.

Cusumano and Nobeoka identify four development models, three of which involve transferring technology:

• New design : "projects that primarily develop platforms from scratch"

• Concurrent technology transfer: "a new project begins to borrow a platform from a base preceding project before the base project has completed the design work. Generally, this transfer occurs within two years"

• Sequential technology transfer : "a project inherits a platform from a base project that has finished the design work"

• Design modification : "a project that replaces an existing product without creating a new platform or borrowing a platform from another production line"

I suggest that these models hold true for software development. A car platform differs from a software framework in that a platform is built for a specific product but may be reused with modification; while a software framework is designed from the start as a generic environment. Platforms provide greater focus on the initial project.

Cusumano and Nobeoka concentrate on Toyota, and their use of development centres and concurrent technology transfer of platforms between Toyota and Lexus lines. Honda uses more traditional matrix management practices, but has great success sharing common platforms between more diverse products - witness the Civic platform, which is also used for the CRX sports car and CRV SUV while internationally the same platform was used by Rover for several years. Elsewhere Chrysler's continued use of sequential transfer and design modification in the 1990s reduced short-run costs, but led to an outdated product line that had problems competing in the market.

From analysis to coding, software development is an exercise in design, so comparisons with car design are highly relevant. Perhaps the biggest lesson is: reuse does not just happen, you have to manage it, you have to put processes in place to deliver reuse. (Which brings us back to Conway's law - "align process and design.")

My experience is that most code-level reuse is bottom-up, it comes from engineers designing a part to be reusable. Cusumano and Nobeoka show how management can be aligned to this process and how they can encourage it. If you want reusable software you must align your process, engineers must know how to reuse, management must be aligned.

Listing the lessons of Cusumano and Nobeoka would take an article in their own right, better you go and read the book. However, here are a few ideas:

• Look to design a product with a platform; plan for the platform to be re-used, follow-on projects should overlap with the original project

• Organise around product lines not functional abilities

• Use strong, autonomous, project managers to drive projects forward

• Physically co-locate project teams together

• Integrate R&D with product development

• Keep communication complexity to a manageable level

• The "company memory" approach does not work well, leading to reuse of old components and infrequent replacement cycles

• Informal "tech clubs" can be used to promote reuse without formal multi-project management.

• Written documentation is good for transferring component knowledge but poor at transferring integrative knowledge; perhaps worryingly, evidence shows that placing a lot of emphasis on retaining prior knowledge can hamper innovation and use of new technologies

Finally, sometimes we need to forget about reuse, sometimes reinventing the wheel brings benefits. Mazda's Miata (MX-5) was developed by a "guerrilla team", physically separated from the main development groups, freed from reuse restraints, with a mandate to produce a car to enhance Mazda's image. Reuse is a valuable tool, but it is not an end in its own right.

Few, if any, software projects are on the scale of designing a car, still, there are useful lessons and ideas here. The software industry is not an island, we must look to established business, management and engineering models to understand and improve ourselves.

Adding "re" to the front of the word "use" doesn't actually buy us very much. When you got on the number 65 bus this morning, were you using the bus to get to work? Or were you reusing the bus to get to work?

OK, maybe I'm playing semantic games here but when we talk about code reuse we cover a lot of ground:

• using a third party library, maybe even the STL

• using code salvaged from the system we are replacing

• using code from a different project within the company

• developing two programs in parallel and sharing some code

This list could go on, but I hope you get my point. What is use ? And what is reuse ? The term reuse is not just overloaded but is actually pretty devoid of any real meaning!

So where does this leave me?

So far I've pointed out that reuse is a good thing. I've also claimed that much of what we do in the name of reuse is wasteful. And finally, I've attacked the word itself. So, where am I going with this?

Reuse has become too generalised, too much of a good thing. I think we need some new words, some new terms, new ideas to describe what we are actually talking about here.

OK, so we will end up sounding like management consultants, but I don't think one word can really cover all the things we use it for. (And maybe, just maybe, we will see our salaries get a bit closer to those of management consultants!)

What else can I say?

Now what I'm about to suggest may come across as jargon-making, for which I apologise. However, jargon can serve a useful purpose: to specifically identify an idea or concept. So here goes, here are some terms and ideas.

Commodity

This is a word I like. I was already playing around with it in this context before I saw Kevlin Henney's use of the word in Overload 47. If we are developing an object, a module, a component, or whatever, which we are intending to use in several places, commodity fits: we are developing a piece of software which can be used in various ways.

Commodity also has a nice overtone of trading; we buy and sell commodities. This highlights the commercial nature of most software development.

Another benefit of commodity is that it implies certain properties, interfaces and standards - some commonality. A television is a commodity, we can be sure that in the USA it will use one standard, and in England another - inconvenient yes, but a known issue. We can develop DVD players to work with this interface for the TV, we don't need specialised sets.

Product has similar commercial overtones to commodity but must be rejected for our purposes. The word carries too much other baggage. It implies a complete, or completed, item. Commodity is much more granular.

Technology Transfer

Although this term smells a lot like management-consultant-speak, I think it is much more accurate than reuse. It clearly states what we are talking about and is free of the baggage that reuse carries.

As the sidebar outlines, the term is already used elsewhere and it can be expanded to describe the temporal characteristics of the transfer.

There are two distinct forms of software transfer that can be identified:

• Horizontal transfer: this is the sharing of common commodities between diverse projects, e.g. when we use a strings library or a threading library we are transferring technology horizontally. The projects may have little business commonality but they have a great deal of raw technology in common.

• Vertical transfer: this is the transfer of technology between similar products, typically within a family of software. For example, a bank that uses the same evaluation model for equities options, foreign exchange trading and risk management is transferring technology vertically.

Vertical transfer usually relates to the application domain so it is more common within organisations, while horizontal transfer, which frequently relates to the system domain, is more common between organisations. Program families, as discussed by Parnas [ Parnas1976 ], are akin to vertical technology transfer.

Most of the well-known libraries available to developers (e.g. Boost, Rogue Wave Source Pro, Microsoft MFC, etc.) are horizontal libraries. They deal well-known themes (threads, database connectivity, GUIs, etc.) which developers are actually rather good at solving because we deal with the same problems all the time.

A vertical library inhabits a niche that is less well known. For example NAG's libraries are less well known than Rogue Wave because fewer people need advanced mathematical algorithms. Even so, more developers know about NAG's libraries than know about, say, libraries for computing fuel consumption in cars, because many developers have a mathematical background.

Integration

Integration is the process of embedding the transferred technology into another development. We may want to distinguish between a donor project (where the technology comes from) and a recipient project (where the technology is used.)

Intellectual Property - IP

IP is the stuff that technology businesses are based on. Increasingly, this is becoming a commodity as patent databases are set up and companies are encouraged to place their property on line for others to buy. In the hardware world this is established practice - think of the way ARM has built an entire business on the licensing of its CPU technology.

Software can travel the same road. Our intellectual property is expressed in terms of source code, UML diagrams and documentation - although a significant part is still locked up inside developers' heads.

When we use a third party library we are not just avoiding the need to write some code ourselves - we are actively seeking to use others' work. For example, rather than try and understand the mathematics behind optimisation theory I just go and buy the work NAG have done; I use their intellectual property to save myself time in development, testing and verification.

A word of warning though: some people have taken a dislike to the idea of IP expressed in software. The problem is not with the idea of IP itself, but with some of the patents granted, and enforced, which are giving it a bad name.

Publishing

Once we start to think of source code as a commodity with the ideas and work contained therein as intellectual property, it becomes natural to think of publishing the work. Once our IP is expressed in a commodity, which may be bought and sold, we need to publish it.

Publishing may be external, we may place an advert in Dr Dobbs and request payments, or it may be internal, we may announce to our company that we now have a library that performs some business function, or may be just publishing at the level of our development group.

Commissioning

To continue the publishing metaphor we may like to think of the initial request as a commission. One could imagine a large company, say a bank, identifying vertical application families with common features, e.g. credit appraisal, and commissioning one development group to develop a commodity solution which other groups could also use.

Packaging

Once we have our intellectual property, we must ensure it is packaged for delivery as a commodity for publication. We may choose to package it as source code, a shared library, or a service accessible over the internet. How we package it will in part be determined by our revenue model, and by how we wish to divide the market.

For example, look at the way Cola-Cola subdivides its market. Although it is nominally selling the same product (sugared water with flavouring) the pricing depends on the packaging. A 330ml can (sold to people on the move) costs more litre-for-litre than the 2 litre bottle (sold in the supermarket to families at home) which in turn costs much more, litre-for-litre, than the cola concentrate sold to restaurants for sale with food - and where it is mixed with water and marked up.

We may make our IP available as a SOAP enabled web service and charge per use. We may repackage this as a DLL and sell it for a fixed price to a large corporation. Finally, we may enter into revenue sharing deals with other producers who compile our commodity into their own.

Conclusion

That several people in the software industry are questioning "The Holy Grail of Reuse" is probably a sign of a maturing industry. Reuse is a worthy goal, and has its uses, but it should not be all consuming. We must understand its costs and implications.

Reuse will probably remain a general all embracing term, it is too entrenched to go away, but we may be better served by coming up with new labels for some of the ideas it covers. I'm not saying my labels are the right labels or that they will stick but I'm starting the debate.

Nor am I claiming that we can simply replace the word reuse. We may try to do a mental "global replace 'reuse' with 'technology transfer'" but this wouldn't change the real situation. We need to change the way we think about our code and the way we use it.

Even without our own debate on what constitutes software reuse the economics of software are changing. For many applications we can now assume an Internet link which enables us to think about pay-as-you-go and leasing models.

References

[Czarnecki-2000] Czarnecki & Eisenecker, Generative Programming , Addison-Wesley, 2000

[Parnas1976] David Parnas, "On the development and design of Program Families", 1976, reprinted in Software Fundamentals: collected papers by David L. Parnas , edited by Hoffman & Weiss, Addison-Wesley, 2001

[Henney2002] Kevlin Henney, "minimalism : the imperial clothing crisis", Overload 47 , February 2002

[Rasch] Chris Rasch, "The Wall Street Performer Protocol : using software completion bonds to fund open source development" http://www.firstmonday.dk/issues/issue6_6/rasch/index.html