Common search engine questions about testing #2: How does software testing impact software quality?

This is the second of a ten-part blog series in which I will answer some of the most common questions asked about software testing, according to search engine autocomplete results (thanks to Answer The Public).

In this post, I address the question “How does software testing impact software quality?” (and the related question, “How is software testing related to software quality?”).

It’s worth taking a moment to clarify what I mean by “quality”, via this definition from Jerry Weinberg and Cem Kaner:

Quality is value to some person (that matters)

I like this definition because it puts a person at the centre of the concept and acknowledges the subjectivity of quality. What is considered to be a bug by one customer may well be viewed as a feature by another! This inherent subjectivity means that quality is more amenable to assessment than measurement, as has been well discussed in a blog post from James Bach, Assess Quality, Don’t Measure It.

So, what then of the relationship between testing and quality?

If we think of testing as an information service provider, then the impact of testing on the quality of the end product is heavily dependent on both the quality of that information and also on the actions & decisions taken on that information. If testing provides information that is difficult to interpret or fails to communicate in a way that is meaningful to its consumers, then it is less likely to be taken seriously and acted upon. If stakeholders choose to do nothing with the information arising from testing (even if it is in fact highly valuable), then that testing effort has no demonstrable impact on quality. Clearly then, the pervasive idea in our industry that testing improves quality isn’t necessarily true – but it’s certainly the case that good testing can have an influence on quality.

It may even be the case that performing more testing reduces the quality of your delivered software. If the focus of testing is on finding bugs – over identifying threats to the software’s value – then performing more testing will probably result in finding more bugs, but they might not represent the important problems in the product. The larger number of bugs found by testing then results in more change in the software and potentially increases risk, rather than reducing it (and the currently popular idea of “defect-free/zero defect” software seems to leave itself wide open to this counterintuitive problem).

Testers were once seen as gatekeepers of quality, but this notion thankfully seems to be almost resigned to the history books. Everyone on a development team has responsibility for quality in some way and testers should be well placed to help other people in the team to improve their own testing, skill up in risk analysis, etc. In this sense, we’re acting more as quality assistants and I note that some organisations explicitly have the role of “Quality Assistant” now (and it makes sense to say “I am a QA” in this sense whereas it never did when “QA” was synonymous with “Quality Assurance”).

I like this quote from James Bach in his blog post, Why I Am A Tester:

…my intent as a tester is not to improve quality. That’s a hopeful side effect of my process, but I call that a side effect because it is completely beyond our control. Testers do not create, assure, ensure, or insure quality. We do not in any deep sense prove that a product “works.” The direct intent of testing – what occupies our minds and lies at least somewhat within our power – is to discover the truth about the product. The best testers I know are in love with dispelling illusions for the benefit of our clients.

Testing is a way to identify threats to the value of the software for our customer – and, given our definition of quality, the relationship between testing and quality therefore seems very clear. The tricky part is how to perform testing in a way which keeps the value of the software for our customer at the forefront of our efforts while we look for these threats. We’ll look at this again in answering later questions in this blog series.

I highly recommend also reading Michael Bolton’s blog post, Testers: Get Out of the Quality Assurance Business, for its treatment of where testing – and testers – fit into building good quality software.

I’m providing the content in this blog series as part of the “not just for profit” approach of my consultancy business, Dr Lee Consulting. If the way I’m writing about testing resonates with you and you’re looking for help with the testing & quality practices in your organisation, please get in touch and we can discuss whether I’m the right fit for you.

The first part of this blog series answered the question, “Why is software testing important?“.

Thanks to my review team (Paul Seaman and Ky) for their helpful feedback on this post.

Advertisement