Everyone’s talking about testing!

I can’t remember a time in my life when “testing” has been such a hot topic of coverage in the media. It feels like every news item leads with some mention of the number of tests conducted to detect the COVID-19 coronavirus, whether it be it locally or further afield. This level of coverage of the topic of testing even exceeds that during Y2K (according to my memory at least), albeit in a very different context.

It was interesting to see the reaction when the President of the United States said that the US case numbers are high because so many tests have been conducted – and that a reduction in testing might be in order. This led Ben Simo to tweet on this idea in the context of software testing:

Stop testing your software! Bugs are the result of testing. Bugs that don’t kill users dead instantly aren’t really bugs. No testing is the key to zero defect software! If you don’t see it, it doesn’t exist. If you don’t count it, it doesn’t matter. No testing!

I felt similarly when I read some of the coverage of the worldwide testing efforts during the pandemic. “Testing” for COVID-19 is revealing valuable information and informing public health responses to the differing situations in which we find ourselves in different parts of the world right now. (In this context, “testing” is really “checking” as it results in an algorithmically-determinable “pass” or “fail” result.)

When we test software, we reveal information about it and some of that information might not be to the liking of some of our stakeholders. A subset of that information will be made up of the bugs we find. In testing “more”, we will likely unearth more bugs as we explore for different kinds of problems, while “more” in the COVID-19 sense means performing the same test but on more people (or more frequently on the same people).

We should remain mindful of when we’ve done “enough” testing. If we are genuinely not discovering any new valuable information, then we might decide to stop testing and move onto something else. Our findings from any test, though, represent our experience only at a point in time – a code change tomorrow could cause problems we didn’t see in our testing today and an unlucky person could acquire COVID-19 the day after a test giving them the all clear.

There is a balance to be struck in terms of what constitutes “enough” testing, be that in the context of COVID-19 or software. There comes a point where the cost of discovering new information from testing outweighs the value of that information. We could choose not to test at all, but this is risky as we then have no information to help us understand changing risks. We could try to test everyone every day for COVID-19, but this would be hugely expensive and completely overwhelm our testing capacity – and would be overkill given what we already understand about its risks.

Many of us are testing products in which bugs are potentially irritating for our users, but not life and death issues if they go undetected before release. The context is clearly very different in the case of detecting those infected by COVID-19.

As levels of COVID-19 testing coverage have increased, the risk of acquiring the virus has become better understood. By understanding the risks, different mitigation strategies have been employed such as (so-called) social distancing, progressively more stringent “lockdowns”, and mandatory mask wearing. These strategies are influenced by risk analysis derived from the results of the testing effort. This is exactly what we do in software testing too, testing provides us with information about risks and threats to value.

It’s also interesting to observe how decisions are being made by bearing in mind a broader context, not just taking into account the testing results in a particular area or country. Data from all across the world is being collated and research studies are being published & referenced to build up the bigger picture. Even anecdotes are proving to be useful inputs. This is the situation we find ourselves in as software testers too, the software in front of us is a small part of the picture and it’s one of the key tenets of context-driven testing that we are deliberate in our efforts to explore the context and not just look at the software in isolation. In this sense, anecdotes and stories – as perhaps less formal sources of information – are incredibly valuable in helping to more fully understand the context.

Test reporting continues to be a topic of great debate in our industry, with some preferring lightweight visual styles of report and others producing lengthy and wordy documents. The reporting of COVID-19 case numbers continues to be frequent and newsworthy, as people look to to form a picture of the situation in their locale. Some of this media reporting is very lightweight in the form of just new daily case and fatality numbers, while some is much deeper and allows the consumer to slice and dice worldwide data. Charts seem to be the reporting style of choice, sometimes with misleading axes that either exaggerate or down play the extent of the problem depending on the slant of the publisher. Different people react to the same virus infection reports in quite different ways, based on their own judgement, biases and influences. We see the same issue with software test reporting, especially when such reporting is purely based around quantitative measures (such as test case counts, pass/fail ratios, etc.) The use of storytelling as a means of reporting is nothing new in the media and I’d argue we would be well served in software testing to tell a story about our testing when we’re asked to report on what we did (see Michael Bolton’s blog for an example of how to tell a three-part testing story – a story about the product and its status, a story about how the testing was done, and a story about the quality of the testing work).

While I don’t normally focus on counting tests and their results, I’ll be happy to see more COVID-19 tests taking place and fewer new daily positive results in both my area of Australia and the world more generally. Stay safe.

(Many thanks to Paul Seaman for his review of this post, his sage feedback has made for a much better post than it otherwise would have been.)