Afterwood

One of the first meetups I attended – when we could finally come out of hiding once the pandemic was waning – was DDD London. After the talk on Executable Specifications had finished, we retired to the pub across the road and started chewing the fat. I’d had a slightly frustrating time at work that day, so opened the conversation with a minor rant about a colleague who’d raised a pull request where the code didn’t even parse, let alone fail some tests. This wasn’t the first time, either; we’d chatted before when I’d noticed them generating pull requests with lengthy commit histories due to typos and silly bugs. This tale generated some sympathetic sighs from my audience, along with some testing related chatter that eventually led to someone posing the question: “When do we become test-obsessed?”

If there’s something I learnt from Seb Rose’s extended lightning talk back at the ACCU 2013 conference, it’s that overly dramatic words have a tendency to come with unfortunate traits. His talk was titled Are you Passionate? and it pointed out that job adverts for ‘passionate programmers’ are effectively asking for people who are highly emotional and potentially reckless! He posited that we should really be asking for the opposite, i.e. dispassionate programmers, people who are methodical and considerate. (The ‘move fast and break things’ crowd would likely disagree, but most of us work for companies who actually have customers and a reputation worth protecting.)

Likewise, if you look up the term ‘obsessed’, it implies an unhealthy fascination, i.e. taking an extreme interest in a subject. I find great benefit in the practice of testing, but I don’t like the thought of being considered obsessed by their utility!

Of course, this question wasn’t intended literally, it was just a figure of speech, but it did get me thinking about my attitude towards testing in case it might be verging too far towards the wrong end of the scale. I seem to spend more time these days thinking about the idea of ‘working software as the primary measure of progress’, and how that is realized throughout the delivery pipeline to avoid wasting time building the wrong thing, or reworking badly implemented features. Adopting a Kaizen approach (continuous improvement) means every little mistake can be an opportunity to re-evaluate the process and work out how to eliminate that source of waste going forward. There are always trade-offs to consider, but in general software development is about the long game, and that means considering productivity as a function of the entire system’s lifetime.

Winding back to what originally triggered this obsession with being potentially obsessed, I don’t think anyone would consider it extreme to double-check locally that your code compiles, or in the case of a dynamically typed language, parses. Furthermore, running any automated tests should also be a no-brainer. The build server is great at watching my back, but doing all this locally first allows me to get the fastest feedback possible. That stops me context switching and ensures I concentrate on finishing what I’ve started. I can then get the change into production quicker, and consequently start to get a return on that investment.

Where, I suspect, the waters start to get murky is when considering the effort required to create a suite of fast running, reliable, automated acceptance tests. Once you step out of the world of pure functions and start factoring in side-effects and external services, writing those kind of tests starts to get much harder. The ‘GOOS’ book (Growing Object-Oriented Software, Guided by Tests) is over 15 years old now and introduced the ‘outside-in’ style of development, which depends on being able to write automated end-to-end tests (customer oriented), as well as the more traditional unit tests (programmer oriented). While in-memory mocks seem to have caught on, no doubt because they’re easy to write, the real gem – automated acceptance tests – typically remain far more elusive; this is despite the leaps in technology which make testing with external dependencies so much easier now.

Of course, not every change is a simple functional change that can be easily verified with a selection of automated tests, most notably deployment and infrastructure related changes, or where real world data is just too messy to distil into a comprehensive test set. Anywhere it’s not obvious from the committed tests how the testing was done I like to see some kind of a test plan in the commit message, PR description, or work item ticket. One of the problems with the term ‘code review’ is that it completely ignores any review of the test approach, which can easily escalate into ‘isn’t that QA’s problem?’ I get it… testing is way less fun than writing code, even test code, but fire-fighting means nobody gets to have any fun.

Once a feature is deployed to production, we can move on the next big thing, right? Not so fast! The term ‘testing in production’ makes some people nervous, so I prefer ‘validate in production’, i.e. is it actually doing what we expected, where it truly matters? There is a technical angle here – did we deploy/apply it correctly – because test environments are always a crude approximation of the real world, but there is also a business one. Observability tries to close the loop between the features we add (or remove) and the business outcome we obtain.

But what if we didn’t make it possible to observe the effects of our feature in a useful way? The way to ensure that happens comes right back at the beginning during the refinement process, and, for the delivery team it’s likely the first of many times when they get to ask the probing question, “But how do we test this?” (Only after “Should we even build this?”, of course.)

Hence, those of us who might be regarded as test obsessed are probably just the ones who feel the need to keep asking that awkward question because we’ve seen too many assumptions make an ass out of us and umption (ask Samual L. Jackson). You can dress it up as checking, validating, or evaluating, or talk about acceptance criteria and conditions of satisfaction, or customer demos, but it’s still all about the various feedback loops, and finding ways to keep them short.

Ultimately the term ‘working software’ boils down to just one test, the acid test: does it solve the customer’s problem? Unfortunately, getting to that point can sometimes involve a test of one’s patience.

Chris Oldwood is a freelance programmer who started out as a bedroom coder in the 80s writing assembler on 8-bit micros. These days it’s enterprise grade technology from plush corporate offices the comfort of his breakfast bar. He has resumed commentating on the Godmanchester duck race but continues to be easily distracted by emails and DMs.