It’s not hard to imagine that the demand for software testing is set to continue to grow, as software pervades more and more aspects of our lives (with the Internet of Things adding its own challenges and opportunities).
How this growing demand will be met is uncertain, but the current options include in-house, outsourced, offshored, crowdsourced, and onshored. Each of these options comes with its own history and set of perceived advantages and disadvantages (and prejudices).
In-house
Developing and testing our software in the same office (or virtual team) using our own company’s employees is where it all began – and remains – for many companies. The idea of separate (independent) testing teams in such organizations has been challenged by agile adoption, but it is still common for everyone in the software development organization to be a company employee.
Advantages of this model include having control over staffing and keeping all IP within the company. Disadvantages include the high cost of recruitment and training and a relatively inelastic response to changes in demand on your teams. (For a fairly biased judgement on the pros and cons, see articles from outsourcing companies such as this: http://www.bughuntress.com/ext_vs_int).
What impetus is there for change in these organizations? Cost is oft cited, with alternatives such as those discussed below being seen by bean counters as offering lower cost to get the “same work” done, especially in terms of testing. Outside of cost, the idea of using a crowd to broaden or diversify coverage by real users is also a common motivation.
Outsourced
According to wikipedia:
“Software Testing Outsourcing is software testing carried out by an independent company or a group of people not directly involved in the process of software development. Software testing is an essential phase of software development, however it is often viewed as a non-core activity for most organisations. Outsourcing enables an organisation to concentrate on its core development activities while external software testing experts handle the independent validation work. This offers many business benefits which include independent assessment leading to enhanced delivery confidence, reduced time to market, lower infrastructure investment, predictable software quality, de-risking of deadlines and increased time to focus on development.”
The most common outsourcing models seem to be either fully outsourcing the entire testing functions to another company (the Managed Testing Service or Centre of Excellence model) or provisioning additional testing resources into a team from an external company (sometimes referred to as the “body shop” model).
With the widespread adoption of agile approaches, the idea of having “testing” being performed in a different company (and often a different country) than where the software is developed immediately presents challenges – the benefits of co-location are lost and issues around time zones, language barriers and cultural differences also come into play. Some of these challenges are mitigated to varying degrees by the idea of “nearshore outsourcing”.
A significant issue in my (limited) experience is the difference in goals between the company employees and the outsourced resources. While you probably have ways to motivate and encourage retention in your own company, the outsourced resources are unlikely to have the same motivation to please you and to genuinely care about their work (as they will be swapped in and out of projects to meet demand, this also creates a problem with building up knowledge and skills).
There is no doubt that outsourcing has been a very popular model over the last decade when it comes to software testing, but I have rarely read of improvements in the way the testing work is done as being a primary motivator for moving to an outsourced model. Cost is a big driver and it is unfortunate that – generally speaking – the software testing industry has done such a poor job of clearly communicating the value that great testing can add to teams. When testing is seen as a job anyone can do (if it’s seen as being scriptable, repeatable, low value work) then of course it’s a race to the bottom in terms of cost and the perceived advantages of outsourcing this easy work to lower cost locations are appealing.
While India remains the epicentre of testing outsourcing (with the big vendors talking of test resources in the order of tens of thousands), I was surprised during a recent visit to Vietnam to discover that it is also becoming a popular choice (for some reasons, presented by a Vietnamese outsourcer, see http://www.tmasolutions.com/whyvietnam). I presented at the Ho Chi Minh City Software Testing Club conference in January and almost everyone I met there was working for a company providing outsourced testing services, most commonly to US customers.
Offshored
I have a few years experience (in my work at Dell Software) of operating an offshored model – in our sense, this means testing being performed by Dell employees in a different location than the development team, most often in lower-cost locations such as China and the Czech Republic.
This model presents some of the same challenges as (farshored) outsourcing, but with the benefit that the entire team is made up of Dell employees motivated to the same goals. My particular experience of working closely to build context-driven testing capabilities in China has resulted in successful outcomes and I have spoken about these experiences at international testing conferences (such as Let’s Test) in the hope that sharing these positive experiences will help others achieve good outcomes when using similar models. The three major challenges I have encountered in this situation are cultural differences, language barriers and traditional testing mentality. These challenges are listed in order of difficulty, so I believe working on the cultural aspects of the relationship with offshored teams is the best place to focus your efforts so that trust relationships can form to make life easier on both sides. Getting good testing from the offshored teams in this case has been the relatively easy part, after working hard to establish good personal relationships between the local and remote teams.
(Colin Cherry wrote an interesting blog piece on the onshore vs offshore testing models, at http://itesting.com.au/2012/10/14/onshore-vs-offshore-testing-a-short-debate.)
Crowdsourced
The idea of crowdsourcing has been around in the software testing arena for around ten years now, with uTest (founded in 2007) being the most immediately recognizable name in this area. According to wikipedia:
“Crowdsourced testing is an emerging trend in software testing which exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform. It differs from traditional testing methods in that the testing is carried out by a number of different testers from different places, and not by hired consultants and professionals. The software is put to test under diverse realistic platforms which makes it more reliable, cost-effective, fast, and bug-free. In addition, crowdsource testing allows for remote usability testing because specific target groups can be recruited through the crowd.
This method of testing is considered when the software is more user-centric: i.e., software whose success is determined by its user feedback and which has a diverse user space. It is frequently implemented with gaming, mobile applications, when experts who may be difficult to find in one place are required for specific testing, or when the company lacks the resources or time to carry out the testing internally.”
This model has gained favour in more recent times especially in the mobile space, where device proliferation is a real issue for in-house testing teams trying to build up a representative test environment. This model is also now widely used for beta testing purposes.
A significant disadvantage to this approach in my opinion is the focus on defect finding (since most providers of such crowdsourcing platforms pay the testers on a “per bug” basis), rather than leveraging testing skills earlier in development cycles to prevent defects from ever being introduced. But this model certainly has its place and uTest has grown rapidly, no doubt inspiring a number of other players who have more recently entered this market such as Passbrains and Pay4Bugs.
(I came across a somewhat bizarre offering while researching for this blog post, a guy who will test your website for usability while he is drunk! He can be found at http://theuserisdrunk.com and records his sessions – and publishes them for all to see using https://www.screenmailer.com)
Onshored
I’ve been familiar with the idea of “onshoring” for a while, mainly due to the work of Keith Klain at Doran Jones in New York, aiming to bring testing jobs currently outsourced to overseas markets back to the US. It was this latest article in Wired magazine on the onshoring initiatives currently starting up in the Bronx that was the inspiration for this blog post in fact:
http://www.wired.com/2014/11/urban-onshoring/
Keith Klain’s reputation has led to Doran Jones attracting some good testing talent from the context-driven testing community already and their work with Per Scholas is truly inspiring. Not only is this programme training a bunch of great testers (via the likes of Paul Holland), it is also giving those with poor prospects a genuine career opportunity, as well as adding to the US – rather than some overseas – economy.
The Doran Jones “urban development centre” idea is just starting off and has lofty ambitions (a target of 450 testers in their Bronx facility), I will be following their onshoring efforts with keen interest.
As the way we build and deploy software changes, so does our approach to testing – what will our testing “mix” look like in ten years from now? As always in the IT business, we live in interesting times!
Rockin' and Testing All Over The World 