An Industry that Refuses to Learn
I reckon it's now getting on for two and a half years since I sat in that meeting with the rest of the development group. I'd made the mistake of giving up my independence and joining a company owned by someone else - a small and apparently promising company about to get bigger, and with a bunch of development staff that were well above the industry norm in terms of their enthusiasm for their jobs. Then one day and out of the blue, management hired a technical architect, and soon after his arrival, a development group meeting was called to brief the group on his vision for future technical direction. Components featured highly on his agenda because, he explained, if we developed components, we would get the benefit of reusability.
Now, most of the development group members had been the industry for two to three years, except one who had been in the industry for about a decade and a half - he and I spontaneously started to reminisce about how this was exactly the rhetoric surrounding objects a decade earlier! We pointed out that this promised reusability hadn't been delivered then, and we were not confident it would be this time around. We also pointed out some other common sense things, such as you need to get past the margin of usability first because all too often software is difficult enough just to use, and if it's difficult to use then forget about reusing it. However we couldn't get the architect to understand any of this, and he went on to lead the company in reaping the cost of his dubious wisdom.
So, why am I telling you this story? Well, the above tale is an illustration of the search for the silver bullet - that one thing that is the solution to all problems. It seems like a poignant way of starting off this editorial, in which I want to talk about what I see as the greatest troubles of our industry. These are the troubles that are uncomfortable - even painful - to talk about. These are the troubles that I encounter far too often, that condemn our industry as nothing short of dysfunctional, and that every time I encounter them make me wish I had a different career (ok, the grass is always greener on the other side, I know). What I have to say may come across as something of a rant, and if it does, so be it!
At the time of writing this I'm in my seventeenth year in the industry, and I've seen my share of failed projects. In fact, I've seen more fail than succeed. I say this with some confidence that readers who have a comparable number of years in the industry will, with some sadness, share the sentiment (recently a friend told me of a former colleague who after fifteen years in the industry has yet to work on a successful project). We work in an industry that worships technology. We've all been privy to debates about the merits of one programming language versus another and one technology versus another, and why one is better for some projects and not for others, but let me say this: I have never seen a project fail as a direct result of a particular technology! In fact, I think it's fair to say that most of the projects I've seen fail were doomed to fail, before any technical work was done or before any technology had been put in place. I've just mentioned the term silver bullet, a term coined by Frederick Brooks in his "No Silver Bullet" essays, which can be found included as chapters in the second edition of his book "The Mythical Man-Month". There's much of relevance to discuss in these essays, but I'm going to let that pass because I want to turn my attention to another chapter of the same book: the one entitled "Why Did the Tower of Babel Fail?" In this chapter the author argues compellingly that those building the tower had a clear purpose (even if naïve), and they had adequate technology, plenty of time and all other resources. Yet the project failed. Again Brooks' arguments are compelling: it was lack of communication (and consequently a lack of organisation) that was to blame. It has been my experience too, that much of the time projects fail because of lack of communication. I say much of the time, because some of the time there are other reasons too, but I don't want to get into detail because it'll distract from the point I want to make.
Frederick Brooks was project manager for IBM's OS/360, and in "The Mythical Man-Month" he documents the lessons he learned on this and other projects. Now be afraid, be very afraid, because the first edition was first published back in 1975, which means anyone in the industry under twenty-eight years of age (and that's a significant proportion) wasn't even born when the first edition of "The Mythical Man-Month" was first published - and we're still getting it wrong today !
Just writing about this makes me shudder, but let's press on anyway, I have another story to tell...
A few weeks ago, I was having a conversation with a web programmer. Now web programming is something I still haven't done, but there was a possibility of me getting into it and helping out on a project with this programmer's company, so he was giving me some background on what's involved before I started pitching into the detail. I'm précising the conversation rather a lot here, but basically, the programmer explained that the most significant issues are to do with the transactional nature of programming for the web - the browser collects information, sends it to the web server, and when the web server replies there is no knowledge in the browser of what went before. As he was telling me this I began to recall a couple of projects I worked on in the early 1990s that involved writing Windows front-ends for mainframe systems. This involved tapping into the 3270 terminal protocol commonly in mainframe systems, where the terminal is not interactive with the mainframe - rather it collects input from a form presented on the screen, sends the whole lot back to the mainframe in a buffer, and springs back into action when the mainframe sends a new buffer full of information back to be presented. I think I started to detect a pattern emerging.
As I mulled over the above conversation I began to think about the two friends I made at the time of my first job as a contractor in 1997, working in a small software house based in Buckinghamshire - a company with a history of supplying mainframe systems. While I was there I became friends with two of the permanent staff who I still regularly visit. Both have been in the industry for nearly two decades longer than I have. Now the scary bit: these two old timers are part of a generation of programmers who solved the problems associated with the transactional programming model over a quarter of a century ago! They solved them when programming mainframes in assembler, the new generation of web programmer solves them using a variety of modern technologies, but the problems and their solutions haven't changed much in a quarter of a century .
Let's face it, how could these two old timers pass their knowledge on to the modern generation of programmers anyway? What mechanisms are available for facilitating this kind of communication? Well, in the early 1990s hope appeared in the form of the patterns movement, which offered a framework for capturing problems, their solutions, together with all relevant information such as the tradeoffs involved. Around that time the "Design Patterns" book (by Gamma et al) appeared, and made Patterns visible to the development community at large. However, Patterns were hijacked and became another bandwagon, another buzzword! This is the most damning indictment against the industry: Patterns were hijacked not because of any fault of Erich Gamma or any of his co-authors, but because it is in the nature of an industry that is more impressed by fads than by progress.
I fear that my two Bucks based friends are now part of a whole generation of programmer that the technology and silver bullet crazed industry has seen fit to forget about. In recent times the Agile Development movement, and the Extreme Programming approach in particular, has brought practices such as unit testing - practices that my two friends regarded, in their youth, as a way of life - to the attention of the modern programmer. I fear that such practices will now join the armoury of buzzwords that must be present on a CV to make sure it is found by agency database searches. I fear the same will happen to these practices as happened with Patterns: the industry will pay lip service with enthusiasm, but because of its deep rooted dysfunction it will lose out to the desire to adopt - or rather the desire to be seen to adopt - the latest fad.
The software development industry has not made any significant progress towards improving its productivity over the last quarter of a century. It continues to ignore communication and instead looks to fads - in the form of fashionable practices or technologies - to provide a silver bullet that will produce huge leaps in productivity. I'm not saying practices and technologies are not of value, on the contrary their continued development is an essential part of progress, assuming they are used for their merit and not just for their novelty value. Unfortunately, developing something that is an essential part of progress is not enough to make progress happen, and real progress - the improvement of productivity - is not happening. Problems are encountered in the course of any project, and it is the resource - mainly in the form of people's time - expended in solving problems that significantly inflates project costs. This cost can be cut dramatically if known solutions can be applied at the strategic point in the project. This is where the price of looking for the silver bullet really takes its toll - it distracts from the process of reusing and cultivating existing knowledge because its promise seduces the industry into concentrating on the search for what can't be found.
Making progress will be a struggle (if it is even possible) because there are too few people who understand that there is a problem. Further, of those who understand that there is a problem, few have any idea how big the problem really is. I believe it is unlikely that things will improve over the next decade, and possibly longer. The software development industry is an industry that refuses to learn.
