In response to Cigniti’s “The emerging trends in Software Testing and Quality Assurance” blog post

A LinkedIn post led me to a blog post from Cigniti, titled The emerging trends in Software Testing and Quality Assurance (published on 2nd November 2020).

These kinds of posts are particularly common in the later part of the year as predictions for the next big things for the year ahead start to surface. While I read such posts with no expectation of learning anything profound from them, I’m still generally disappointed by what they have to say. This latest effort from Cigniti continues that theme of disappointment.

Notionally providing only three “emerging” trends, the build up to their big reveal is worth critiquing, starting with their opening gambit:

Software testing and Quality Assurance practices are on the ride to continuous evolution, improvement, and inclusion. Rather than being a separate function, QA is all set to become implicit to the development itself. For many software development teams, the process of making QA a part of the software development has already begun. This transition in the perception regarding QA from being a choice to a necessity is one of the most significant milestones that the IT sector has achieved in the evolutionary process. 

Removing the filler words and fluff, the message seems to be that “QA” (by which I’m going to assume they mean “testing”) is more a part of the development process than a separate “function”. The idea that testing has been seen as a choice in the recent past but is now seen as a necessity doesn’t marry with my experience in the industry at all. With the move to more agile ways of working, most teams I’ve encountered in the last few years have made their testing efforts part of their development practice with embedded testers (or whatever label they have) rather than having separate testing teams (or the surely now deprecated idea of “Testing Centres of Excellence”).

Next up, the blog makes an appeal to authority – and authorities don’t come much more credible than Gartner (right?):

Gartner, in the ‘Top Strategic Technology Trends for 2021’ report, has emphasized the need for people centricity…

It’s worth making a mental note here about “people centricity” in readiness for the emerging trends that come later.

A number of big claims are made next:

Having a streamlined QA function eliminates the bottlenecks that may hinder timely code releases into production, resulting in a better satisfaction ratio among the intra-organizational teams. A high-quality code release translates into fewer defects into production and improved end user experience. An increased customer satisfaction level offers increased ROI to the organization.

There is no evidence provided in the post or in citations to support:

• “Having a streamlined QA function eliminates the bottlenecks…” – the implication here appears to be that it’s QA that hinders “timely” releases, whereas the reality is often quite different in my experience. There are many potential bottlenecks and obstacles to production releases given the uncertainty and changing environment of most software development projects.
• “… resulting in a better satisfaction ratio among the intra-organizational teams” – maybe I’ve missed the idea of “satisfaction ratio” coming into the industry, but I really don’t understand what they’re saying or how this claim can be backed up.
• “A high-quality code release translates into fewer defects into production and improved end user experience” – there is no indication here about what is meant by quality (nor an acknowledgement that “quality” can mean very different things to different people in different contexts in different organizations) but connecting “high-quality code” to “fewer defects in production and improved end user experience” seems to me to be highly questionable.
• “An increased customer satisfaction level offers increased ROI to the organization” – I cringe when I see mention of “ROI” around QA/testing and I cringed on cue here. The start of the very same blog post suggested that QA/testing is a necessity and not a choice, so why talk about its ROI? We don’t hear demands to increase the ROI of development or management or support, so why attempt to measure it for another necessary aspect of software development? (Note that Paul Seaman blogged on the use of terms from economics in testing recently, though not specifically on ROI.) And how exactly does “increased customer satisfaction” arise from a “streamlined QA function”? Customer satisfaction is again a many faceted thing, so suggesting that a change to the way testing is integrated into the development process necessarily increases customer satisfaction is oversimplifying what is a complex and deeply contextual rating.

We finally get to the three “emerging trends that will shape the [QA] function in the coming year and beyond”, starting with “Scriptless test automation”:

Test automation has been one of the top software testing trends for past few years. With a wider acceptance of test automation within the SDLC, there also comes the realization for constantly optimizing it for fulfilling the evolving requirements. One of the major challenges that organizations have been facing in adopting test automation has been the lack of skilled test automation resources for test script maintenance. Scriptless test automation is enabling organizations to overcome this challenge and still have efficient test cases for automating software testing.

Responding to this part alone could form its own post, but this is a perfect example of the kind of misguided thinking and advice that is so prevalent when it comes to the topic of automation. They are reinforcing the idea that automation is a trend but all that pesky code that’s required to tell the machines what to do is just too hard to maintain. It seems to be a hard message to get across, but creating automated checks is software development – and needs to be treated as such. The problem isn’t a “lack of skilled test automation resources for test script maintenance” as much as using people not skilled in programming to write the test code, in my opinion. While there is some value in scriptless automation technologies, I think their importance is generally overestimated at the same time as the effort involved in crafting genuinely maintainable and valuable automated check code is underestimated. (And, yes, I noted that they closed out this trend with the phrase “automating software testing”, as though that’s a thing.)

The next trend is “The marriage of AI, ML and QA”:

Introducing AI and ML into Quality Assurance process allows an organization to get out of the ‘Test Automation trap’, which can be explained as – “Test automation trap is when the test teams are not getting enough time to be able to do the failure triage from the previous test run before building the next test automation code.” 

It wouldn’t be a trend piece without mentioning “AI” and “ML”, of course. The fact that there’s three folks in the marriage should be a worrying sign in itself, but really this idea that AI and ML will significantly impact the way the majority of organizations perform testing anytime soon just doesn’t stack up. Looking at the common questions on LinkedIn, for example, most organizations are still struggling with such basic aspects of testing and quality management that introducing AI and ML would probably be more disruptive than helpful. I’ve not heard of this “Test Automation Trap” before either, which to me sounds like an issue of poorly written tests or an acceptance of many “test failures” as being the norm. Using AI/ML to tackle the issue of failure triage seems to be coming at this problem from the wrong end.

The final trend is “Continuous integration for continuous quality”:

With the help of DevOps, about 59% of organizations are now deploying multiple times a day, once a day, or once every few days. For these organizations, the code quality has been one of the biggest benefits of embracing DevOps.  

The CI/CD pipeline combined with test automation has done wonders for organization in terms of the quality of their releases. Not only has it positioned QA as an imperative instead of a bottleneck, it has resulted in higher returns as well. At present, only a handful of organizations have deployed bots to review their code, but this number is expected to go up as we move further into the future. 

Continuous integration is hardly a new thing and, in this “trend” towards “continuous quality”, there is conflation of the terms continuous integration, continuous delivery and DevOps. This is a common problem in the current discourse around these subjects unfortunately. Another common problem is the idea that more frequent deployments necessarily leads to improved quality. The use of CI/CD pipelines and more automation doesn’t in any way guarantee higher quality deliverables, it all too frequently seems to end up delivering poor quality more often, in fact. The post again uses this term “bottleneck” and implies that these more automated delivery mechanisms remove the bottleneck, but the human element of critically evaluating what’s been built before it’s delivered is implicitly being devalued here and is increasingly seen as an element to be minimized or removed. I’m not sure what the code review bots are about as referenced here, but it again stems from this notion that we’re better off having machines do this kind of critical analysis work rather than humans so as to remove “bottlenecks” to us deploying software (with little or no human critique along the way).

It seems to me that all of these trends are essentially centred on increasing our reliance on machines as part of our testing and quality practice, which seems to contradict the “people centricity” trend cited from Gartner at the top of the post. I can’t recall any recent trend identified in one of these reports that focuses on improving the testing skills of humans. If I was being cynical, I’d think that there’s more money to be made in identifying trends around which vendors can build tools for profit…

There was nothing of substance in Cigniti’s post that would prove useful to someone in testing with a genuine interest in learning where our craft is headed in the years ahead. I wonder if anyone is interested in a contrarian article about “trends”, something like “The folly of following trends” perhaps?

If the thoughts expressed in this post resonate with you, my new consultancy might be able to help you with improving your testing and quality practices – please see www.drleeconsulting.com.au for more details of my offering.