Software Development in the 21st Century

Software development is an observational science rather than an experimental one: that is, it is difficult if not impossible to try varying the conditions that affect the outcome one at a time to disentangle their effects. Observing projects can be difficult though: some organisations are hostile to their intimate practices being exposed and observers filter their observations through their individual perceptions of what is important.

The current authors are as prone as others to subjective reporting, but hope that the following experience report describing a project lifting itself from primordial chaos to 21st century Agility will provide useful examples to others working in this field.

In the beginning

Despite all the progress made in the latter half of the 20th century a lot of software development is still a disorganised mess which delivers results only through heroic developers sacrificing their lives (or at least their home lives) for the good of the project. Such are the beginnings of our tale.

When one of the authors was first asked to consider working on the project it was known by management to be in difficulties - the project manager (who was based on the far side of the Atlantic to the rest of the team) had, despite frequent visits and long phone calls, difficulty understanding what work was being undertaken and why. As is often the case, the lead developer came under criticism for not communicating effectively, the situation worsened when frustration forced a senior developer on the team to tender his resignation. The brief was simple: get things back on track!

The software involved was a server-side system supporting numerous end-user applications around the globe all of whom were competing to get their favoured features implemented. Luckily most of the developers of these applications were co-located in London with the developers on the project. This meant that user requirements capture and eliciting feedback were never problematic during our time with the project.

We join the team

Timing was good - one of us had a week's handover from the departing developer which afforded a sure footing and a rapid start. The reality on the ground bore some resemblance the situation described at interview - although it has to be said that the lead developer was willing to explain what he was doing and why, the difficulties arose as he was interacting directly with the clients in the same office and, understandably, failing to report every conversation about issues, features and changing priorities across the Atlantic.

The first step was to consolidate the current issues, feature requests and state of work in a form that was accessible to everyone involved. As there was an existing project Wiki this was used for the purpose. This got a lot of positive feedback from both the project manager and from the client teams all of whom could see what was going on for the first time. It was during this exercise that the second author joined the team.

However, despite the initial enthusiasm for the Wiki based task display it slowly became obvious that these web pages only make progress visible if people visit those pages. And so, as the novelty wore off these slowly fell into neglect. Even regular planning 'meetings' (conference phone calls to review the Wiki) failed to keep the information flowing. This became dramatically apparent when emails began to circulate with the project manager demanding to know why a feature wasn't being worked on only days after he'd ran a conference call with the users that had agreed on the work for the iteration.

Organisational changes happen

While we were pondering how to address this problem some changes happened that presented an opportunity. In this case there were some changes at board level which resulted in the appointment of a new project manager located in the same office as the project team!

This made a major difference, as he could interact directly with user groups and quickly discovered that the work being done was not giving the best return. In particular, the team was working on a fourth generation implementation of the system whilst the production environment was a patchwork of the second and third generations. He decided to stop work on the fourth generation and focus effort on completing the roll-out of the third generation. The lead developer quit in disgust.

Shortly after this we moved offices and the opportunity arose to acquire a whiteboard: there were several allocated to the office, but as they had not been assembled we found them languishing in the corner of a side office. By assembling one behind our desks we established possession and moved our planning onto this display.

We divided the whiteboard horizontally into three columns - for the release being rolled out, for the release being developed and for future activities. A post-it was used to denote each task and moved from the bottom (not started) to the middle (started) to the top (completed). Planning meetings became brief and increasingly informal and centred around the whiteboard. (Figure 1)

Figure 1

While this was happening we also managed to get continuous integration running. Part of the legacy of the first developer who quit was a set of functional tests - after a bit of work on these they turned into a test suite that gave us (and the other users of the UAT environment) the ability to automate the promotion of each 'successful' build into the UAT environment. This got us early feedback from other teams about new features - one of whom had an independent 'acceptance test suite' that they ran regularly (and that caught our errors a couple of times before we got our own tests right - our system had features we didn't know about).

Time passed, the team grew and we successfully introduced a 'test first' culture. The legacy 'second generation' system was withdrawn from service and a raft of important new features made it into production. We had a steady 'heartbeat' of releases improving the system and good feedback from users.

Outsourcing

The next hurdle came when the manager decided that there were problems with the whiteboard planning system. These were varied: the unavailability of the whiteboard for other teams to use (particularly other ones he managed); the 'danger' of losing information if post-its fell off or were moved accidentally; and, the lack of visibly outside the office. It was probably the latter that was decisive (as his desk in a side office didn't have line of sight, there were client teams in another building, and he was considering outsourcing some of the development work to Yorkshire).

In any event, the corporate standard for project planning was a web based system: Jira. This avoided the principle problem encountered with the earlier Wiki based system (that people didn't see progress if they didn't visit the Wiki). With Jira stakeholders could subscribe to tasks that interest them and receive email notifications when they were updated.

With the arrival of the inevitable mandate to migrate the project planning system to Jira, the development team downed IDEs for web browsers for an afternoon. Converting each and every post-it to a Jira ticket was soon accomplished and the (now blank) whiteboard became (for many weeks) a monument to the past. Developers from the outsourcing company were each brought in for an introductory period to learn the system and meet the rest of the team. It seemed like no time before the goal of a geographically diverse team was accomplished.

The new approach propelled a new wave of productivity as new functionality passed through UAT into production, Jira streamlined the development team's efforts and, thanks to integrated report generation, promoted a higher degree of transparency with management and end users.

Unfortunately, as all too often is the case, the new approach fell foul of its own success. With increased productivity, release confidence and management support all on the increase - the publicity not only spurred on more interest in the project (affording a steady stream of functionality requests) but also promoted the project as a campaign poster child for Jira. (And outsourcing to Yorkshire!)

Quite frankly the support teams had understandably not requisitioned sufficient hardware resources to sustain the mushroom cloud that was to follow the eruption of Jira adopters. The knock on effect of this was that the beloved tool which had served the team so well now became a bottleneck. But success breeds success and the project budgets would now support a few upgrades - not only to servers but to desktops.

While multiple 24-inch monitors were a delight to the developers the project manager was especially pleased with getting a massive screen installed in a nearby conference room. (We didn't pay for this directly - but as a star outsourcing department we needed to demonstrate meetings with our offsite team members - the group 'outsourcing project' funded this on the basis that we were willing to share.)

April showers

Although our project continued it's meteoric progress during the recent recession this was not true of other parts of the organisation and it soon became apparent that our private conference room was going to have squatters (moved from another building that was no longer needed). So, one quiet evening we swapped the still blank whiteboard for the sexy display screen from the conference room.

We even 'found' (ok, one of us was sad enough to write it) a new screensaver for it that would show the project's Jira homepage - regaining the visibility of project progress that we'd missed since the whiteboard went blank. We could still conference with our rural colleagues - although, as a result of 'downsizing', there did seem to be fewer and fewer of them as time progressed until there were finally none left.

In addition to 'our' conference room these organisational changes also lost us the office formerly occupied by the manager. He was given, and took, the 'opportunity' to improve the in-team communications by joining the rest of us in the main office.

Jira gets an upgrade!

We were still rolling out enhancements at a steady rate when the project manager pointed out a cool new plugin for Jira: GreenHopper. 'What's GreenHopper?' we asked. Anyway, it turned out that this new plugin, amongst other things, allowed Jira to displayed tasks as 'post-its' and one could move those by 'click and drag'. Before very long our status screen was demonstrating the advantages of a proper use of technology over the whiteboard based 'prototype' we'd used earlier in the project. Not only did we have a high-visibility display of the state of the project, it could be viewed over the internet and could generate the all-important status emails automatically. (Figure 2)

Figure 2

The one thing it didn't do 'out of the box' was to allow us to walk up to the screen and move the post-its by hand. This, naturally, was too much of a temptation - very soon some 'gesture sensitivity' was added to the screensaver. It is so cool waving a task note from 'in progress' to 'completed' - Tom Cruise eat your heart out!

Naturally this was so good that we had to make it better - we only needed to get the integration between Subversions and Jira working properly and attach webcams to all the workstations and we could automate closing tickets by smiling as we checked in the code. By pulling up the Jira page and glancing in the right direction we could allocate the next piece of work. We had avoided a lot of unnecessary keystrokes and reached a pinnacle of automation.

Not long before the end

Unfortunately and perhaps quite predictably, the state of software development nirvana was about to undergo some radical change. Whilst the project's deliveries continued to roll out to production at a steady, maintainable pace; the powers that be thought they knew better. It turns out, quite ironically, Jira was making us look bad. The magic that went on behind the scenes to ensure the team's efforts were focused on functional requirements had become an intricate part of the day's activities. The resulting stream of Jira update e-mails soon overwhelmed our stakeholders' inboxes. Very soon rules were set up to 'mark as read' and 'delete' the daily 'spam' generated by the team.

Management finally realised that something special was happening - not only were features being delivered on (or before) schedule, but despite the loss of our offsite developers the pace of deliveries was accelerating. Clearly, they thought, the only thing holding progress back was the developers. No sooner was this thought formulated than it led to action - the authors wished the project success, invited the manager to a leaving do and found alternative, less modern, employment (where their talents were still needed). We trust that the management insight was valid and the project continues to progress with even greater efficiency - we will report further if we hear details.