The tenth edition of the World Quality Report 2018/19 (WQR) was published recently. This 70-page opus was produced by Capgemini, Micro Focus and Sogeti (a division of Capgemini anyway) and is perhaps the largest survey of its kind.
After digesting this report, I feel it’s important to apply some critical thinking to both the data and the conclusions drawn from it in this report. This is a long blog post as there is a lot to unpack here.
The survey (pages 66-69)
I’m always interested in understanding where the data comes from when I see survey results like these and this information is openly provided in the report. Understanding the origin of the data is important context and I read here first (whereas the report itself presents it at the end).
The survey consisted of 1700 interviews. In terms of the organizations taking part, the survey was restricted to only organizations with more than 1000 employees (actually it was 40% from those with more than 10,000, 34% from those with 5000-10000 and 26% from those with 1000-5000) so the results are in fact heavily skewed towards the very largest corporations. The survey had a good spread of countries & regions as well as industry sectors (although the top three sectors accounted for almost half of the responses, viz. financial services, public sector/government, and telecommunications).
The types of people who provided survey responses is more interesting, though – in terms of job title breakdown, they were grouped as follows: CIO (27%), IT director (22%), QA/Testing Manager (20%), VP Applications (18%), CMO/CDO (7%) and CTO/Product Head (6%). With the (possible) exception of the QA/Testing Manager types, most of these people are likely a long way away from the actual day-to-day testing work happening in their organizations.
Let’s look at each section of the report now.
Introduction (pages 4-5)
In his introduction, Brad Little of Capgemini (page 4) says the WQR is “a comprehensive and balanced overview of the key trends shaping quality assurance (QA) and testing today”. In his introduction, Raffi Margaliot of Micro Focus (page 5) says “The results from this tenth edition of the WQR are conclusive: all but one percent of organizations are now using DevOps practices in their organization. Their focus is no longer on whether to move to DevOps; rather how to refine their DevOps approach and continuously improve.” Perhaps coincidentally, it’s worth noting that Micro Focus offers solutions in the DevOps space and, as you’ll see later in my post, the “conclusive” part of this statement is highly questionable.
Executive Summary (pages 6-9)
On the role of AI in QA and testing, “AI will enable enterprises to transform testing into a fully self-generating, self-running and self-adapting activity”, really? And why would that be seen as a good thing?
On agile: “Our survey also reveals that organizations are customizing Agile and combining it with waterfall to develop hybrid frameworks that are a best fit to their organizational, regulatory, cultural and business requirements” – so not thinking of a move towards agility as a mindset change, but rather a process/framework from which to cherrypick the bits that are easy and then calling themselves “agile”.
On automation: “The objective of automation has also changed as there is less focus on shortening of testing times and more on better coverage and effective use of test cases. This, again, is related to the dictum of “Quality at Speed”” – what about the almost complete uptake of DevOps and agile you just mentioned? Aren’t these dependent on fast feedback loops? I’d argue that the focus of automation has changed, but largely in support of CI/CD pipelines where fast feedback on recent changes before deployment is key. “Moving forward, organizations will need to move toward higher levels of end-to-end testing automation”, why?
The cost of testing is also covered in this executive summary, I’ll talk in detail about that later but for now enjoy the statistic that QA and testing account for 26% of the IT budget (the same number as in the previous year’s report).
WQR findings (pages 10-11)
The findings kick off with the following revelation: “Expecting QA and testing to directly contribute to “ensuring end-user satisfaction” is not an obvious or intuitive expectation. However, this year, it came out as the top objective of QA and testing strategy.” I’m not sure why this is so surprising to the authors of this report, but I’m intrigued by having this specifically as an objective of a test strategy.
AI is a big focus of this report and this big claim is made here, “The convergence of AI, ML, and analytics and their use in carrying out smarter automation will be the biggest disruptive force which will transform QA and testing over the next two to three years.” It will be interesting to see whether such significant disruption really does occur in such a short timeframe, I don’t see a great deal of evidence to support this claim from my part of the testing world but I acknowledge that others may have different views and their organizations might actually be more active in these areas than I’d expect: “57% of our respondents said they had AI projects for quality assurance, already in place or planned for the next 12 months.” (though the “or planned” part of that survey question leaves a lot of wriggle room).
On the topic of automation: “These challenges around automation, test data, and environments, create a situation where organizations are unable to keep pace with the volume and frequency of testing required. Essentially, they slow down testing, thus defeating one of the main objectives of adopting frameworks such as agile and DevOps. This also came through in our survey results, when 43% of respondents said that “too slow testing process” was a challenge when it came to developing applications today.” It’s interesting that agile and DevOps are referred to as “frameworks” and that this also seems to imply that faster testing is one of the main objectives of agile and DevOps.
Key recommendations (pages 12-13)
The authors make five key recommendations out of the mountain of survey data underlying this WQR, viz.
- Increase the level of basic and smart test automation but do so in a smart, phased manner
- Implement a non-siloed approach for test environment and data provisioning
- Build quality engineering skills beyond SDETs
- Improve tracking to optimize spends
- Develop a testing approach for AI solutions now
The first recommendation around automation is based on this conclusion: “We believe that automation is the biggest bottleneck holding back the evolution of QA and testing today. This is due, in part, to automation’s key role as an enabler of successful agile and DevOps transformation. With increasing agile and DevOps adoption (99% according to the 2018 survey), the importance of automation for delivering “Quality at Speed” has also risen.” While I agree that automation has an important role to play in our more modern development approaches, I’m not convinced that a lack of automation is holding back the “evolution of QA and testing” – I’d see the lack of focus on genuine human testing skills to be a much bigger issue in fact. The 99% DevOps adoption statistic is reeled out again in support of their conclusion, see the “Trends” section below for more on the dubious grounding of this number.
When it comes to test environments, the survey responses indicate that a lot of organizations have a lot of issues with generating and maintaining appropriate test data, something that’s been a common thread in the industry for years. The recommendation here is to centralize test data and environment provisioning and “move towards “smart” test data and test environment management, i.e. the creation of self-provisioning, self-monitoring and self-healing environments”.
In terms of skillsets for those in QA and testing, the “first priority is to attract/reskill towards agile test specialists who have functional automation skills and domain testing skills. We would recommend that automation be a must-have skill for everyone in the QA function today.” This topic is pretty hot in our industry right now and reports like this making recommendations like this mean there is even more weight behind the idea that every tester needs to be able to “do automation”. None of their recommendations in this area deal with upskilling humans in performing excellent testing and this is a huge gap in most organizations. Recruiting recently for an expert exploratory tester here in Melbourne showed just how few highly-skilled testers are around, while there is a vast supply of “traditional”/ISTQB style testers plying their wares in large organizations. I do think that testers need to understand where automation makes sense and where it doesn’t, and can greatly assist those writing automation code to focus on the right places – but that doesn’t mean every tester needs to write automation code in my opinion.
On the subject of tracking spend on testing, the report makes the fairly obvious observation that “The adoption of agile and DevOps in project teams has led to a situation where QA and testing activities are being done by many, including developers as well as specified testing professionals. This makes it tough to accurately track, understand or optimize QA and testing spends.” The recommendation is to “create a detailed and elaborate tracking mechanism” to work out who is spending what time on testing activities so it can be more accurately tracked. Given that the report itself claims that everyone is working in an agile fashion with whole team approach to quality, I’m not sure why anyone would want to try to track the spend in this way. Surely the spend of interest is the total spend on developing the product and trying to split out testing reinforces the old ways of thinking of divisions between development and testing. I’ll talk about cost more in the “Trends” section, but recommending this artificial and arbitrary boundary between testing and everything else required to build, test and produce the product is an anti-recommendation at best and dangerous at worst (in that the percentage spent on QA and testing is always an easy target for “optimization”, and we all know what that means!).
I acknowledge that testing AI solutions is a tricky problem so the recommendation that “every organization needs to start developing a QA and test strategy for their AI applications now” has some merit. I still think the idea that so many organizations are actively working on genuine AI solutions is questionable (and the data in this report is not conclusive, given the nature of the survey responses).
Current Trends in Quality Assurance and Testing (pages 14-43)
Almost half of the WQR is dedicated to discussing current trends in QA and testing and this was by far the most revealing content in the report for me. I’ll break down my analysis in the same way as the report.
Key trends in IT (pages 16-21)
This section discusses the results of survey questions around executive management objectives with QA and testing, before looking specifically into digital transformation & the API economy, internet of things, Cloud, cybersecurity, and blockchain. The conclusion in this section is bang on: “It’s also important to remember that the new IT model is not just about the latest technologies or improved processes. Above all, it is a change in culture, attitude, and mindset, which will also require a change in the traditional ways of working or delivering services. This means that people, processes, and technology will all have to go through a period of change and improvement before we can fully realize the benefits promised by the new technologies and frameworks.” Finally a mention of culture and mindset change – and the challenges that always brings.
Artificial intelligence (pages 22-25)
I just watched a EuroSTAR webinar from Jason Arbon (CEO of test.ai) titled AI Will Soon Test Everything (there’s a lot to unpack in that webinar too, especially around exploratory testing, but that’s a whole other blog post!) so was interested in the data in the AI section of this report, especially given the big prominence given to AI in the summary, recommendations and key findings sections before it.
A little over half of the respondents indicated that AI is “in place or planned on quality assurance” and the report comments that a “big challenge lies in identifying possible use cases for AI in testing” (with over half of the respondents citing “identifying where business might actually apply AI” as a challenge). Interesting, exactly half of the respondents indicated that they think there is no change required in skillset around “test strategy and test design skills” when including AI. I agree with the report here when it states “Clearly, there is enthusiasm for and excitement around AI technologies and solutions, but their actual application in testing is still emerging.”
Test automation (pages 26-30)
This trend is subtitled “The single-biggest enabler of maturity in QA and testing”, interesting! The authors nail their colours to the mast right from the introduction in this section, with “For QA, this means an increased focus on the concept of “Quality at Speed” and its associated promises of avoiding human intervention wherever possible, reducing costs, increasing quality, and achieving better time to market. And the way to achieve each of these goals? Automation.” I don’t think we should be viewing any one thing as offering solutions to so many different issues in more agile software delivery projects – and “avoiding human intervention wherever possible” is not a goal for me either.
The well-worn 99% statistic is presented yet again as a factor driving more automation: “the adoption of agile and DevOps, which seems to have reached a tipping point today, with 99% of our respondents saying they use DevOps for at least some part of their business”. The statistics around DevOps adoption (see next section) don’t suggest a tipping point anytime soon.
A whopping 61% of the respondents suggest application change as an obstacle to automation success: “When asked about the main challenges in achieving their desired level of automation, 61% of respondents said they had difficulties automating as their applications changed with every release. This could be a direct result of the flexibility provided by frameworks like agile and DevOps, which allow organizations to change their requirements or stories frequently. This often leads to too many changes with every release and puts additional pressure on testers as test cases generated earlier or previous automation work no longer remains relevant.” It amazes me that people think application changes in a release are some kind of surprise and even more so that all those pesky changes we make to help and please customers are thought of as “too many changes with every release”! For products under active development, we should expect plenty of change and adapt our practices – not just in relation to automation but throughout our development process – to match that reality.
In discussing the benefits of automation, 60+% of respondents indicated “better test coverage”, “better control and transparency of test activities”, “better reuse of test cases”, “reduction of test cycle time”, “better detection of defects” and “reduction of test costs”. It would be interesting to understand more about what these categories mean and how organizations are measuring such benefits. It would seem likely to me that different organizations would have quite different goals around their automation initiatives, so it’s unlikely those goals would lead to the same set of benefits in all cases.
In summary, the authors say “To be successful, organizations must understand that automation is not only about replacing manual testing and chasing some incremental cost savings. Instead, focus on delivery quality at speed and supporting frameworks such as agile and DevOps to deliver much greater results and take QA and testing to the next level.” I would argue that to be successful, organizations need to realize that automation cannot replace manual testing but can extend and supplement human testing when applied appropriately. Chasing cost savings is a fools errand in this case, automation really is just writing (and maintaining) more code, so why would that save money?
The quality assurance organization (pages 31-35)
This is the section of the report that details the claim that “according to this year’s survey, a full 99% of respondents said that they were using DevOps principles in their organization”. This sounds like a pretty impressive statistic, so impressive that it is littered throughout the rest of the WQR. Looking into the actual survey results, though, the story is not quite so impressive:
- 42% of respondents said “Fewer than 20% of our projects use DevOps principles”
- 30% of respondents said “20-50% of our projects use DevOps principles”
- 15% of respondents said “50-70% of our projects use DevOps principles”
- 9% of respondents said “70-90% of our projects use DevOps principles”
- Just 3% of respondents said “90-95% of our projects use DevOps principles”
In other words, almost three quarters of the respondents are using DevOps principles in the minority (i.e. less than 50%) of their projects. These underlying statistics are much more instructive than the “banner” 99% claim the authors choose to perpetuate in the WQR. What’s also interesting in these same statistics over time (2015-2018) – which the authors either didn’t spot or chose not to mention – is that they actually suggest a decrease in the use of DevOps! In 2015, for example, some 58% of respondents fell into the “using DevOps principles for 50% or more of their projects” categories.
My general impression here (and elsewhere in the report) is that DevOps and agile are not clearly understood as being different by the authors. This is reinforced by these comments: “As already stated, only one percent of respondents indicated that they were not experimenting with or applying DevOps in any form. According to the survey, the top DevOps processes being followed were “breaking down large efforts into smaller batches of work” (44% of respondents currently using and 38% planning to use), “cloud-based development and test environments” (43% using and 40% planning to use), and the “continuous monitoring of apps in production” (41% using and 40% planning to use).” Isn’t the idea of breaking down work into smaller pieces very much an agile mindset? What does it really have to do with DevOps?
Looking at the conclusions in the section on challenges in applying testing to agile developments, one of the conclusions is “As the skillset is moving from functional to SDET (Software Development Engineer in Test), organizations are faced with challenges of reskilling the existing testing teams and attracting the right testing talent to build future-ready testing teams.” The data shows that 41% of respondents cite “Lack of a good testing approach that fits with the agile development method” as a challenge. This challenge is not solved by making everyone an SDET, in fact probably quite the opposite. The role of expert human testing is again not discussed at all here when the data clearly supports a view that skilled human testers are critical in solving many of the challenges seen when testing in agile environments.
Test data and environments management (pages 36-39)
I didn’t find anything particularly surprising or controversial in this part of the report. The authors identified increased containerized, virtualized and Cloud test environments and also noted the impact that standards & regulations such as GDPR and IFRS9 are having on test data management.
[Note that there is some errant data from a different section of the report mistakenly placed in this part of the report.]
Efficiency and cost containment in quality assurance (pages 40-43)
It’s a case of leaving the best (or is that worst?) to last in terms of my objections to the findings in this report, viz. when talking about “efficiency and cost containment”. The authors’ opening gambit relates to the proportion of IT budget spent on testing: “According to this year’s survey, the proportion of the IT budget spent on QA and testing is pegged at 26%. This is the same as last year, though considerably below the highs of 31% in 2016 and 35% in 2015. Before that, QA and testing budgets accounted for 26% in 2014 and 23% in 2013.” I’ll leave it you as the reader to ponder why these statistics are the way they are and how you would even measure this percentage in your own organization. The authors surmise that “as organizations have gained experience and maturity in handling these new frameworks and technologies [e.g. agile, DevOps, cloud test environments, etc.], they have started reaping the benefits of these changes. A number of testing activities have gained efficiency and this has driven down costs. This is reflected in the fall in the proportion of IT budgets devoted to testing that we have seen over the last two years.”
Interestingly, the authors note that “According to our 2018 survey, when respondents were asked whether they had seen an increase in the proportional effort and cost spending on QA and testing over the last four to five years, a whopping 72% said “yes”. This directly contradicts the overall budgetary trends.” The authors then dig “deep” into the reasons for this confusing (or contradictory) data.
Firstly, they make the reasonable argument that IT budgets have generally been increasing due to the take up of new technologies, digitalization and so on, so the absolute test effort and budget has increased but stayed relatively the same against those increased overall budgets. The second argument is around divvying up test effort and spend across back-end legacy systems compared to front-end systems, with the back-end benefiting greatly in terms of cost from increased automation while the drive for speed at the front-end means more spend there to keep up with changing business requirements.
These first two arguments are somewhat reasonable. It’s the third observation, however, that to me makes a mockery of the whole idea of measuring the cost of testing and QA as a percentage of total IT budget: “The third and final factor is probably the biggest of them all. This is the difficulty in accurately capturing testing spends due to the coming of age of agile and DevOps. Before agile and DevOps, testing often operated as a separate profit or cost center, with operations centralized in a Test Center of Excellence (TCoE). This made it easier to measure spends and track how much was being spent on what. However, agile and DevOps have made this kind of tracking difficult since testing is now integrated into the project or the Scrum teams. This makes it extremely difficult to track exactly how much time is spent on testing activities, especially when the now-prevalent Software Development Engineer in Test (SDET) profile (who engages in development, analysis, and testing activities). It is entirely possible, for instance, that the efforts of these SDETs, or of entire Scrum teams, is being tagged to the development or the testing budget or allocated on the basis of a thumb-rule percentage between these two budgets.” At least the authors have acknowledged here that trying to measure this is a largely pointless exercise in agile teams and, since they keep claiming that almost everyone is doing agile, why even persist in trying to measure this? They essentially say here that it’s almost impossible to measure or, when asked to do so, people just make it up (“a thumb-rule percentage”).
Another interesting bunch of statistics comes next, in the breakdown of QA and testing spend between hardware/infrastructure, tools (licenses), and human resources with these coming in at 44%, 31% and 26% respectively this year. I’m simply amazed that the human element in the total cost is the lowest proportion and this indicates to me a lot of misplaced redirection of effort away from human interactions with product and towards automation (much of which is probably of questionable value).
The claim that “expert opinion holds that the increased number of test cycles brought about by the shift to agile and DevOps is perhaps one of the biggest reasons for a rise in testing effort and expenditures” is not supported by reference to who these experts are (and the survey responses do not support this conclusion directly in my opinion).
The summary for this section makes some recommendations and this is the most depressing part of all: “To gain the maximum benefit from their QA and testing spends, we would recommend that organizations focus on three key areas over the next couple of years. First, work on creating successful use cases (in testing) for new technologies such as AI, machine learning (ML), or robotics process automation. Second, create detailed and elaborate tracking mechanisms to understand exactly how much cost and effort is going into testing in Agile or DevOps teams. It would be impossible to reduce costs without understanding clearly how much is being spent and where. Finally, there is one step that organizations can immediately take to improve testing efficiencies, that is the use of end-to-end automation in testing. While investments are being made, they are nowhere near the optimal levels. All three of these steps will go a long way to improving testing efficiency and the quality of their IT systems in the long term.”
The last thing a truly agile team should be doing is layering on pointless bureaucracy such as “detailed and elaborate tracking mechanisms” to track testing time and effort. At least the authors make it clear in their recommendations that the point here is to “reduce costs” but this is probably in opposition to the main business driver for QA and testing which was “contribute to end-user satisfaction”. Not every organization will see the need to cut costs in QA and testing if the end products are attractive to customers and making good revenues. Also, suggesting that everyone should add more end-to-end test automation goes against almost every recent published practitioner article on this topic in my part of the testing community.
Sector Analysis (pages 44-65)
I didn’t find this section of the report as interesting as the trends section. The authors identify eight sectors and discuss particular trends and challenges within each. The sectors are:
- Consumer products, retail and distribution
- Energy and utilities
- Financial services
- Healthcare and life sciences
- Government and public sector
- Telecom, media and entertainment
A huge amount of data collection obviously goes into producing a report of this nature and the high profile publishers will no doubt mean its recommendations get plenty of publicity. As I’ve tried to detail in the above, some of the conclusions drawn from the data don’t make sense to me and the skewed nature of the sample (opinions from CIO/CTO types in the very largest corporations) means most of the recommendations don’t resonate with the testing industry as I’m familiar with it.
A few other points I’d like to draw attention to:
- The report always says “QA and testing” together, with neither term being defined anywhere so it’s not clear what they’re talking about and whether they correctly view them as separate concepts or not. I wonder whether the interview questions were also couched in this language and, if so, how might that have affected the answers?
- Similarly, the report usually says “agile and DevOps” together, as though they also necessarily go together. For me, they’re only somewhat related and I know of plenty of organizations practising agile while not taking on board DevOps yet. It is also worrying that both agile and DevOps in this report are most often referred to as “frameworks”, rather than focusing on them more as mindsets.
- There is almost no talk of “testing” (as I understand it) in the report, while there is a heavy focus on agile, DevOps, automation, AI, ML, etc. I would have liked to see some deep questions around testing practises to learn more about what’s going on in terms of human testing in these large organizations.
The report claims to be “a comprehensive and balanced overview of the key trends shaping quality assurance (QA) and testing today”, but I don’t see the balance I’d like and the lack of questioning around actual testing practices means the report is not comprehensive for me either. The target audience for these corporate type reports will probably take on board some of the recommendations from this high-profile report and I imagine in some cases this will simply perpetuate some poor decision-making in and around testing in large organizations.