ER: Attending the Cambridge Exploratory Workshop on Testing (CEWT)

One of the great things about being out of Australia for a while is the ability to experience testing community events in other parts of the world.

I recently attended a Belfast Testers meetup and, shortly afterwards, received an invitation from James Thomas to take part in the third Cambridge Exploratory Workshop on Testing – an invitation I readily accepted!

This peer workshop took place on Sunday 6th November and was held in the offices of games developer Jagex on the (enormous) Cambridge Science Park, with a total of 12 participants (the perfect size for such an event), as follows:

Michael Ambrose (Jagex)

James Thomas, Karo Stoltzenburg, Sneha Bhat, Aleksandar Simic (all from Linguamatics)

Alan Wallace (GMSL)

James Coombes (Nokia)

Neil Younger (DisplayLink)

Chris Kelly (Redgate)

Iuliana Silvasan

Chris George (Cambridge Consultants)

Lee Hawkins (Quest)

The workshop theme was “Why do we Test, and What is Testing (Anyway)?” and, after some introductions and housekeeping about how the workshop would be run, it was time for the first ten-minute talk, from Michael Ambrose with “Teach Them to Fish”. He talked about teaching developers to test at Jagex, as well as upskilling testers to be pseudo-developers. He said there was a technical need to cover more and more as well as a desire to get testers learning more (as a different approach to, say, pushing developers to do automation). Michael noted that there were a number of implications of these changes, including the perception of testers, working out what’s unique about what testers do, and knowing how far to go (getting testers to the level of junior developers might be enough). This was an interesting take on the current “testers need to be more technical” commentary in the industry and the twenty-minute discussion period was easily filled up.

Next up was James Coombes with “Who should do testing and what can they test?” He talked about the “I own quality” culture within Nokia and how he sees different roles being responsible for different aspects of quality. James suggested that developers should find most of the bugs and fix them, while QA then find the next highest number of bugs. Security testers act as specialists with generally few (but important) bugs being found. Documenters/trainers are well placed to find usability bugs, while customer support staff have good knowledge of how customers actually use their products and so can provide good testing tours. Alpha test engineers are responsible for integration/end-to-end testing and catch the low frequency bugs. Finally, customers are hopefully finding the very low frequency bugs. This was an interesting talk about getting everyone involved in the testing activity (and highlighted the “testing is an activity, not a role” idea). I particularly liked what James said about unit testing – “if someone changes your code and they don’t know they’ve broken it, it’s not their problem, it’s yours for not writing a good enough unit test”.

After a short break, I was up next with my talk “What is Testing? It depends …” I decided to tackle the latter half of the theme (i.e. the “what” rather than the “why”) and my idea was to discuss what testing means depending on the perspective of the stakeholder. We focus a lot of time and effort in the community on refining a definition of testing (and I favour the James Bach & Michael Bolton definition given towards the end of the Exploratory Testing 3.0 blog post) but this (or any other) definition is probably not very helpful to some stakeholders. I covered anumber of perspectives such as “Testing is a way to make money” (if you’re a testing tools vendor or a testing outsourcing services provider), “Testing is a cost centre” (if you’re a CFO) and “Testing is dead” (if you’re a CxO type reading some of the headline IT magazines & websites). There was a good discussion after my talk, mainly focused on the cost centre perspective and how this has impacted people in their day-to-day work. I was pleased with how my talk went (especially given the short time I had to prepare) and received some good feedback, particularly on the concise nature of the slides and the confidence with which it was presented. My slide deck can be seen at What is Testing? It depends…

The last session before lunch saw Aleksandar Simic with “A Two-day Testing Story”. He did a fine job of breaking down a two-day period in his work into various different activities, some testing-related (e.g. pairing on test design) and some not (e.g. working with IT support on a networking issue). Aleksandar’s level of self-inspection was impressive, as was his naming of the various activities, learning opportunities and challenges along the way. His “testing diary” seems to be working well for him in identifying and naming his testing activities and this would make an interesting conference talk with some further development.

Lunch provided a good chance for us all to chat and unwind a little after the intensive morning spent talking testing.

First up after the lunch break was Karo Stoltzenburg with “I test, therefore I am”. She had adopted the idea of substitution in preparing her talk so looked to answer the question “Why do I test?” and see where that took her. Karo’s answer was “Because I like it” and then she explored why she liked it, identifying investigation, learning, exploring, use of the scientific method, collaborating, thinking in different contexts and diversity as aspects of testing that appealed to her. I liked Karo’s closing remarks in which she said “I test because it makes me happy, because it’s interesting, challenging and varied work”. We really need more positive messages like Karo’s being expressed in the testing community (and wider still), so I’d love to see this become a full conference talk one day. She did a good job of communicating her passion for testing and there were some interesting discussions in the group following her talk, with a degree of agreement about why testing is so engaging for some of us.

The sixth and final talk of the day came from James Thomas with “Testing All the Way Down, and Other Directions” He walked through an in-depth analysis of Elisabeth Hendrickson’s “Tested = Checked + Explored” from her book, Explore It! James decided to explore this definition of testing using techniques from that definition which wouldn’t classify his actions as testing. He described how he’d interacted with Elisabeth on some of his questions after exploring the idea in this way and finally presented his proposed alternative definition of testing as “the pursuit of actual or potential incongruity” (Note that James more fully describes this talk in his blog post, Testing All the Way Down, and Other Directions) The main focus of discussion after James’s talk was around his proposed definition of testing and I’ll be following the broader community’s response to his proposal with interest.

A few discussion points arose during the day for which we didn’t have time to go deep between talks, so we dedicated ten minutes to each of the following topics to close out the workshop content:

• Quality – what does it mean? (Weinberg definition, but are others more helpful?)
• Domain knowledge (can bias you, can empathy with the end user be a disadvantage? How do we best adjust strategy to mitigate for any lack of domain knowledge?)
• Evaluating success (how do we measure the success of spreading testing into development and other disciplines?)
• Is testing just “the stuff that testers do”? (probably not!)
• How do we make a difference? (blogging, workshops in our own workplaces, brown bag sessions, broader invitation list to peer conferences)

To wrap up, a short retrospective was held where we were all encouraged to note good things to continue, anything we’d like to stop doing, and suggestions for what we should start to do. There were some good ideas, briefly discussed by the group and I’d expect to see some of these ideas taken up by the CEWT organizers as part of their fourth incarnation.

The CEWT group standing outside number 10 Downing Street (or inside the Jagex office, maybe):

This was a really good day of deep-diving with a passionate group of testers, exactly what a peer conference should be all about. Thanks again to James for the invitation and thanks to all the participants for making me so welcome.

For reflections on the event from others, keep an eye on the CEWT blog at

http://cewtblog.blogspot.co.uk/2016/11/cewt-3-reflections.html