2020 in review

It’s time to wrap up my blogging for the year again, after a quite remarkable 2020!

I published 22 blog posts during the year, a significant increase in output compared to the last few years (largely enabled by the change in my employment situation, but more on that later). My blog attracted about 50% more views than in 2019 and I’m very grateful for the amplification of my blog posts via their regular inclusion in lists such as 5Blogs, Testing Curator’s Testing Bits and Software Testing Weekly. November 2020 saw my blog receiving twice as many views as any other month since I started blogging back in 2014, mainly due to the popularity of my critique of two industry reports during that month.

I closed out the year with about 1,100 followers on Twitter, up around 10% over the year – this surprises me given the larger number of tweets around veganism I’ve posted during the year, often a cause of unfollowing!

COVID-19

It wouldn’t be a 2020 review blog without some mention of COVID-19, but I’m not going to dwell too much on it here. I count myself lucky in so many ways to have escaped significant impact from the pandemic. Living in regional Australia meant restrictions were never really too onerous (at least compared to metropolitan Melbourne), while I could continue working from home (until my COVID-unrelated retrenchment).

The only major inconvenience caused by the pandemic was somewhat self-inflicted when we made the unwise decision to travel to the UK in mid-March, arriving there just as restrictions kicked in. It was a stressful and expensive time finding a way back to Australia, but I’m very glad we escaped when we did to ride out the pandemic for the rest of the year at home in Australia. (I blogged about these interesting international travels here and here.)

The end of an era

My 21-year stint at Quest Software came to an end in August. It was an amazing journey with the company, the only job I’ve had since moving to Australia back in 1999! I consider myself lucky to have had such a great environment in which to learn and develop my passion for testing. Of course, the closing out of this chapter of my professional life took a while to adjust to but I’ve spent the time since then focusing on decompressing, helping ex-colleagues in their search for new opportunities, looking to new ventures (see below) and staying connected with the testing community – while also enjoying the freedoms that come with not working full-time in a high pressure corporate role.

Conferences and meetups

I started the year with plans to only attend one conference – in the shape of CAST in Austin – but 2020 had other ideas of course! While in-person conferences and meetups all disappeared from our radars, it was great to see the innovation and creativity that flowed from adversity – with existing conferences finding ways to provide virtual offerings, meetups going online and new conferences springing up to make the most of the benefits of virtual events.

Virtual events have certainly opened up opportunities for attendance and presenting to new people in our community. With virtual conferences generally being very affordable compared to in-person events (with lower registration costs and no travel & accommodation expenses), it’s been good to see different names on attendee lists and seeing the excitement and passion expressed by first-time conference attendees after these events. Similarly, there have been a lot of new faces on conference programmes with the opportunity to present now being open to many more people, due to the removal of barriers such as travelling and in-person public speaking. It feels like this new model has increased diversity in both attendees and presenters, so this is at least one positive out of the pandemic. I wonder what the conference landscape will look like in the future as a result of what organisers have learned during 2020. While there’s no doubt in my mind that we lose a lot of the benefits of a conference by not being physically present in the same place, there are also clear benefits and I can imagine a hybrid conference world emerging – I’m excited to see what develops in this area.

I only attended one meetup during the year, the DDD Melbourne By Night event in September during which I also presented a short talk, Testing Is Not Dead, to a largely developer audience. It was fun to present to a non-testing audience and my talk seemed to go down well. (I’m always open to sharing my thoughts around testing at meetups, so please let me know if you’re looking for a talk for your meetup.)

In terms of conferences, I participated in three events during the year. First up, I attended the new Tribal Qonf organised by The Test Tribe and this was my first experience of attending a virtual conference. The registration was ridiculously cheap for the great range of quality presenters on offer over the two-day conference and I enjoyed catching up on the talks via recordings (since the “live” timing didn’t really work for Australia).

In November, I presented a two-minute talk for the “Community Strikes The Soapbox” part of EuroSTAR 2020 Online. I was in my element talking about “Challenging The Status Quo” and you can see my presentation here.

Later in November, I was one of the speakers invited to participate in the inaugural TestFlix conference, again organised by The Test Tribe. This was a big event with over one hundred speakers, all giving talks of around eight minutes in length, with free registration. My talk was Testing Is (Still) Not Dead and I also watched a large number of the other presentations thanks to recordings posted after the live “binge” event.

The start of a new era

Starting a testing consultancy business

Following my unexpected departure from Quest, I decided that twenty five years of full-time corporate employment was enough for me and so, on 21st October, I launched my testing consultancy business, Dr Lee Consulting. I’m looking forward to helping different organisations to improve their testing and quality practices, with a solid foundation of context-driven testing principles. While paid engagements are proving elusive so far, I’m confident that my approach, skills and experience will find a home with the right organisations in the months and years ahead.

Publishing a testing book

As I hinted in my 2019 review post at this time last year, a project I’ve been working on for a while, both in terms of concept and content, finally came to fruition in 2020. I published my first testing book, An Exploration of Testers, on 7th October. The book contains contributions from different testers and a second edition is in the works as more contributions come in. All proceeds from sales of the book will go back into the testing community and I plan to announce how the first tranche of proceeds will be used early in 2021.

Volunteering for the UK Vegan Society

When I saw a call for new volunteers to help out the UK’s Vegan Society, I took the opportunity to offer some of my time and, despite the obvious timezone challenges, I’m now assisting the organisation (as one of their first overseas volunteers) with proofreading of internal and external communications. This is a different role in a different environment and I’m really enjoying working with them as a way to be more active in the vegan community.

Thanks to my readers here and also followers on other platforms, I wish you all a Happy New Year and I hope you enjoy my posts to come through 2021.

I’ll be continuing my ten-part blog series answering common questions around software testing (the first four parts of which are already live) but, please remember, I’m more than happy to take content suggestions so let me know if there are any topics you particularly want me to express opinions on.

Common search engine questions about testing #4: “How is software testing done?”

This is the fourth of a ten-part blog series in which I will answer some of the most common questions asked about software testing, according to search engine autocomplete results (thanks to Answer The Public).

In this post, I ponder the question “How is software testing done?” (and the related questions, “What are software testing methodologies?”, “What is the software testing life cycle?” and “What is the software testing process?”).

There are many different ways in which software testing is performed, by different people in different organizations with different ideas about what constitutes “good testing”. Don’t be fooled into believing there is “one way” to do testing! There is certainly no single, approved, credible and official way to perform testing – and this is actually a good thing, in my opinion.

So, the question should perhaps be “How might software testing be done?” and, in answering this question, the idea of context is paramount. James Bach defines “context” (in Context-Driven Methodology) as follows:

When I say “context” I mean the totality of a situation that influences the success or failure of an enterprise.

(and Dictionary.com similarly offers “the set of circumstances or facts that surround a particular event, situation, etc.”) The first principle of Context-Driven Testing says “The value of any practice depends on its context.” The way you would approach the testing of a medical device (where a defect could result in loss of life) is likely quite different to how you would test a website for a local business, for example. The context is different – and the differences are important.

While there may be books or certifications that propose a “testing process” or methodology, you should consider the context of your particular situation to assess whether any of these processes or methodologies have valuable elements to leverage. Remember that testing requires a broad variety of different skills and activities: working with other people, formulating hypotheses, creating & changing strategies, critical thinking & evaluation, finding the right people when you need help, assessing what all this might mean for risk and then finding ways to relate this information in compelling and credible ways. What we need is a way of thinking about testing that is flexible enough to cover such a range of skills and activities across many different contexts.

The following from context-driven-testing.com puts it well, I think:

Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing. The essence of context-driven testing is project-appropriate application of skill and judgment. The Context-Driven School of testing places this approach to testing within a humanistic social and ethical framework.

Ultimately, context-driven testing is about doing the best we can with what we get. Rather than trying to apply “best practices,” we accept that very different practices (even different definitions of common testing terms) will work best under different circumstances.

Bearing the above in mind, the only software testing methodology that I feel comfortable to recommend is Rapid Software Testing (RST) developed by James Bach and Michael Bolton. RST isn’t a prescriptive process but rather a way to understand testing with a focus on context and people:

[RST] is a responsible approach to software testing, centered around people who do testing and people who need it done. It is a methodology (in the sense of “a system of methods”) that embraces tools (aka “automation”) but emphasizes the role of skilled technical personnel who guide and drive the process.

Rather than being a set of templates and rules, RST is a mindset and a skill set. It is a way to understand testing; it is a set of things a tester knows how to do; and it includes approaches to effective leadership in testing.

https://rapid-software-testing.com/about-rapid-software-testing/

RST is therefore quite different from some of the prevalent processes/methodologies that you might come across in searching for resources to answer the question of how testing is done, such as ISTQB and TMap. These systems are often referred to as “factory-style testing” and an excellent summary of how RST differs from these can be found at https://www.satisfice.com/download/how-rst-is-different-from-factory-style-testing

Given how different your context and testing mission is likely to be on different projects in different organizations at different times for different customers, the way “testing is done” necessarily needs to be flexible and adaptable enough to respect these very different situations. Any formal process or methodology that seeks to prescribe how to test is likely to be sub-optimal in your particular context, so I suggest adopting something like the mindset proposed by RST and adapting your approach to testing to suit your context.

You can find the first three parts of this blog series at:

• Why is software testing important?,
• How does software testing impact software quality? and
• When should software testing activities start?

I’m providing the content in this blog series as part of the “not just for profit” approach of my consultancy business, Dr Lee Consulting. If the way I’m writing about testing resonates with you and you’re looking for help with the testing & quality practices in your organisation, please get in touch and we can discuss whether I’m the right fit for you.

Thanks again to my patient and dependable review team (Paul Seaman and Ky) for their helpful feedback on this post.

“Calling Bullsh*t” (Carl T. Bergstrom and Jevin D. West)

It was thanks to a recommendation from Michael Bolton that I came across the book Calling Bullsh*t by Carl T. Bergstrom and Jevin D. West. While it’s not a book specifically about software testing, there are some excellent takeaways for testers as I’ll point to in the following review of the book. This book is a must read for software testers in my opinion.

The authors’ definition of bullshit (BS) is important to note before digging into the content (appearing on page 40):

Bullshit involves language, statistical figures, data graphics, and other forms of presentation intended to persuade or impress an audience by distracting, overwhelming, or intimidating them with a blatant disregard for truth, logical coherence, or what information is actually being conveyed.

I was amazed to read that the authors already run a course at a US university on the same topic as this book:

We have devoted our careers to teaching students how to think logically and quantitatively about data. This book emerged from a course we teach at the University of Washington, also titled “Calling Bullshit”. We hope it will show you that you do not need to be a professional statistician or econometrician or data scientist to think critically about quantitative arguments, nor do you need extensive data sets and weeks of effort to see through bullshit. It is often sufficient to apply basic logical reasoning to a problem and, where needed, augment that with information readily discovered via search engine.

The rise of the internet and particularly social media are noted as ways that BS has proliferated in more recent times, spreading both misinformation (claims that are false but not deliberately designed to deceive) and disinformation (deliberate falsehoods).

…the algorithms driving social media content are bullshitters. They don’t care about the messages they carry. They just want our attention and will tell us whatever works to capture it.

Bullshit spreads more easily in a massively networked, click-driven social media world than in any previous social environment. We have to be alert for bullshit in everything we read.

As testers, we tend to have a critical thinking mindset and are hopefully alert to stuff that just doesn’t seem right, whether that’s the way a feature works in a product or a claim made about some software. It seems to me that testers should naturally be good spotters of BS more generally and this book provides a lot of great tips both for spotting BS and learning how to credibly refute it.

Looking at black boxes (e.g. statistical procedures or data science algorithms), the authors make the crucial point that understanding the inner workings of the black box is not required in order to spot problems:

The central theme of this book is that you usually don’t have to open the analytic black box in order to call bullshit on the claims that come out of it. Any black box used to generate bullshit has to take in data and spit results out.

Most often, bullshit arises either because there are biases in the data that get fed into the black box, or because there are obvious problems with the results that come out. Occasionally the technical details of the black box matter, but in our experience such cases are uncommon. This is fortunate, because you don’t need a lot of technical expertise to spot problems with the data or results. You just need to think clearly and practice spotting the sort of thing that can go wrong.

The first big topic of consideration looks at associations, correlations and causes and spotting claims that confuse one for the other. The authors provide excellent examples in this chapter of the book and a common instance of this confusion in the testing arena is covered by Theresa Neate‘s blog post, Testing and Quality: Correlation does not equal Causation. (I’ve also noted the confusion between correlation and causality very frequently when looking at big ag-funded “studies” used as ammunition against veganism.)

The chapter titled “Numbers and Nonsense” covers the various ways in which numbers are used in misleading and confusing ways. The authors make the valid point that:

…although numbers may seem to be pure facts that exist independently from any human judgment, they are heavily laden with context and shaped by decisions – from how they are calculated to the units in which they are expressed.

It is all too common in the testing industry for people to hang numbers on things that make little or no sense to look at quantitatively, counting “test cases” comes to mind. The book covers various ways in which numbers turn into nonsense, including summary statistics, percentages and percentage points. Goodhart’s Law is mentioned (in its rephrased form by Marilyn Strathern):

When a measure becomes a target, it ceases to be a good measure

I’m sure many of us are familiar with this law in action when we’re forced into “metrics programmes” around testing for which gaming becomes the focus rather than the improvement our organizations were looking for. The authors introduce the idea of mathiness here: “mathiness refers to formulas and expressions that may look and feel like math – even as they disregard the logical coherence and formal rigour of actual mathematics” and testing is not immune from mathiness either, e.g. “Tested = Checked + Explored” is commonly quoted from Elisabeth Hendrickson‘s (excellent) Explore It! book. Another concept that will be very familiar to testers (and others in the IT industry) is zombie statistics, viz.

…numbers that are cited badly out of context, are sorely outdated, or were entirely made up in the first place – but they are quoted so often that they simply won’t die.

There are many examples of such zombie statistics in our industry, Boehm’s so-called cost of change curve being a prime example (claiming that the cost of changes later in the development cycle is orders of magnitude higher than earlier in the cycle) and is of one the examples covered beautifully in Laurent Bossavit’s excellent book, The Leprechauns of Software Engineering.

The next statistical concept introduced in the book is selection bias and I was less familiar with this concept (at least under this name):

Selection bias arises when the individuals that you sample for your study differ systematically from the population of individuals eligible for your study.

This sort of non-random sampling leads to statistical analyses failing or becoming misleading and there are again some well-considered examples to explain and illustrate this bias. Reading this chapter brought to mind my recent critique of the Capgemini World Quality Report in which I noted that both the size of organizations and roles of participants in the survey was problematic. (I again note that from my vegan research that many big ag-funded studies suffer from this bias too.)

A hefty chapter is devoted to data visualization, with the authors noting the relatively recent proliferation of charts and data graphics in the media due to the technology becoming available to more easily produce them. The treatment of the various ways that charts can be misleading is again excellent with sound examples (including axis scaling, axis starting values, and the “binning” of axis values). I loved the idea of glass slippers here, viz.

Glass slippers take one type of data and shoehorn it into a visual form designed to display another. In doing so, they trade on the authority of good visualizations to appear authoritative themselves. They are to data visualizations what mathiness is to mathematical equations.

The misuse of the periodic table visualization is cited as an example and, of course, the testing industry has its own glass slippers in this area, for example Santhosh Tuppad’s Heuristic Table of Testing! This chapter also discusses visualizations that look like Venn diagrams but aren’t, and highlights the dangers of 3-D bar graphs, line graphs and pie charts. A new concept for me in this chapter was the principle of proportional ink:

Edward Tufte…in his classic book The Visual Display of Quantitative Information…states that “the representation of numbers, as physically measured on the surface of the graphic itself, should be directly proportional to the numerical quantities represented.” The principle of proportional ink applies this rule to how shading is used on graphs.

The illustration of this principle by well-chosen examples is again very effective here.

It’s great to see some sensible commentary on the subject of big data in the next chapter. The authors say “We want to provide an antidote to [the] hype” and they certainly achieve this aim. They discuss AI & ML and the critical topic of how training data influences outcomes. They also note how machine learning algorithms perpetuate human biases.

The problem is the hype, the notion that something magical will emerge if only we can accumulate data on a large enough scale. We just need to be reminded: Big data is not better; it’s just bigger. And it certainly doesn’t speak for itself.

The topics of Big Data, AI and ML are certainly hot in the testing industry at the moment, with tool vendors and big consultancies all extoling the virtues of these technologies to change the world of testing. These claims have been made for quite some time now and, as I noted in my critique of the Capgemini World Quality Report recently, the reality has yet to catch up with the hype. I commend the authors here for their reality check in this over-hyped area.

In the chapter titled “The Susceptibility of Science”, the authors discuss the scientific method and how statistical significance (p-values) is often manipulated to aid with getting research papers published in journals. Their explanation of the base rate fallacy is excellent and a worthy inclusion, as it is such a common mistake. While the publication of dodgy papers and misleading statistics are acknowledged, the authors’ belief is that “science just plain works” – and I agree with them. (From my experience in vegan research, I’ve read so many dubious studies funded by big ag but these don’t undermine my faith in science, rather my faith in human nature sometimes!) In closing:

Empirically, science is successful. Individual papers may be wrong and individual studies misreported in the popular press, but the institution as a whole is strong. We should keep this in perspective when we compare science to much of the other human knowledge – and human bullshit – that is out there.

In the penultimate chapter, “Spotting Bullshit”, the discussion of the various means by which BS arises (covered throughout the book) is split out into six ways of spotting it, viz.

• Question the source of information
• Beware of unfair comparisons
• If it seems too good or bad to be true…
• Think in orders of magnitude
• Avoid confirmation bias
• Consider multiple hypotheses

These ways of spotting BS act as a handy checklist I think and will certainly be helpful to me in refining my skills in this area. While I was still reading this book, I listened to a testing panel session online and one of the panelists was from testing tool vendor, Applitools. He briefly mentioned some claims about their visual AI-powered test automation tool. These claims piqued my interest and I managed to find the same statistics on their website:

I’ll leave it as an exercise for the reader to decide if any of the above falls under the various ways BS manifests itself according to this book!

The final chapter, “Refuting Bullshit”, is really a call to action:

…a solution to the ongoing bullshit epidemic is going to require more than just an ability to see it for what it is. We need to shine a light on bullshit where it occurs, and demand better from those who promulgate it.

The authors provide some methods to refute BS, as they themselves use throughout the book in the many well-chosen examples used to illustrate their points:

• Use reductio ad absurdum
• Be memorable
• Find counterexamples
• Provide analogies
• Redraw figures
• Deploy a null model

They also “conclude with a few thoughts about how to [call BS] in an ethical and constructive manner”, viz.

• Be correct
• Be charitable
• Admit fault
• Be clear
• Be pertinent

In summary, this book is highly recommended reading for all testers to help them become more skilled spotters of BS; be that from vendors, testing consultants or others presenting information about testing. This skill will also come in very handy in spotting BS in claims made about the products you work on in your own organization!

The amount of energy needed to refute bullshit is an order of magnitude bigger than [that needed] to produce it.

Alberto Brandolini (Italian software engineer, 2014)

After reading this book, you should have the skills to spot BS and I actively encourage you to then find inventive ways to refute it publicly so that others might not get fooled by the same BS.

Our industry needs those of us who genuinely care about testing to call out BS when we see it, I’m hoping to see more of this in our community! (My critique of the Capgemini World Quality Report and review of a blog post by Cigniti are examples of my own work in this area as I learn and refine these skills.)

Common search engine questions about testing #3: “When should software testing activities start?”

This is the third of a ten-part blog series in which I will answer some of the most common questions asked about software testing, according to search engine autocomplete results (thanks to Answer The Public).

In this post, I answer the question “When should software testing activities start?” (and the related question, “When to do software testing?”).

It feels very timely to be answering this question as there is so much noise in the industry at the moment around not only “shifting” testing to the left but also to the right. “Shifting left” is the idea that testing activities should be moved more towards the start of (and then throughout) the development cycle, while shifting right is more about testing “in production” (i.e. testing the software after it’s been deployed and is in use by customers). It seems to me that there is a gap now forming in the middle where much of our testing used to be performed (and, actually, probably still is), viz. testing of a built system by humans before it is deployed.

Let’s start by looking at what we mean by “testing activities” and who might perform these activities.

For teams with dedicated testers, the testers can participate in design meetings and ask questions about how customers really work. They can also review user stories (and other claims of how the software is intended to work) to look for inconsistencies and other issues. Testers might also work with developers to help them generate test ideas for unit and API tests. Testers with coding skills might work with API developers to write stubs or API tests during development. Testers might pair with developers to test some new functionality locally before it even makes it into a more formal build. For teams without dedicated testers, the developers will be covering these – and other – testing activities themselves, perhaps with assistance from a roaming testing/quality coach if the organization is following that kind of model. All of the above activities are performed before a built system is ready for testing in its entirety, so are probably what many would now refer to as “shift left” testing in practice.

The shifting left of testing activities seems to have been heavily influenced by the agile movement. Practitioners such as Janet Gregory and Lisa Crispin have written books on “Agile Testing” which cover many of these same themes, without referring to them as “shift left”. The idea that the critical thinking skills of testers can be leveraged from the earliest stages of developing a piece of software seems sound enough to me. The term “agile tester”, though, seems odd – I prefer to think of testing as testing, with “agile” being part of the context here (and this context enables some of these shift-left activities to occur whereas a different development approach might make these activities difficult or impossible).

In more “traditional” approaches to software development (and also in dysfunctional agile teams), testing activities tend to be pushed towards the end of the cycle (or sprint/iteration) when there is a built “test ready” version of the software available for testing. Testing at this point is highly valuable in my opinion and is still required even if all of the “shift left” testing activities are being performed. If testing activities only start at this late stage, though, there is a lot of opportunity for problems to accumulate that could have been detected earlier and resolving these issues so late in the cycle may be much more difficult (e.g. significant architectural changes may not be feasible). To help mitigate risk and learn by evaluating the developing product, testers should look for ways to test incremental integration even in such environments.

The notion that “testing in production” is an acceptable – and potentially useful – thing is really quite new in our industry. Suggesting that we tested in production when I first started in the testing industry was akin to a bad joke, Microsoft’s release of Windows Vista again comes to mind. Of course, a lot has changed since then in terms of the technologies we use and the deployment methods available to us so we shouldn’t be surprised that testing of the deployed software is now a more reasonable thing to do. We can learn a lot from genuine production use that we could never hope to simulate in pre-production environments and automated monitoring and rollback systems present us with scope to “un-deploy” a bad version much more easily than recalling millions of 3.5-inch floppies! This “shift right” approach can add valuable additional testing information but, again, this is in itself not a replacement for other testing we might perform at other times during the development cycle.

In considering when testing activities should start then, it’s useful to broaden your thinking about what a “testing activity” is away from just system testing of the entire solution and also to be clear about your testing mission. Testing activities should start as early as makes sense in your context (e.g. you’ll probably start testing at different times in an agile team than when working on a waterfall project). Different types of testing activities can occur at different times and remember that critical thinking applied to user stories, designs, etc. is all testing. Use information from production deployments to learn about real customer usage and feed this information back into your ongoing testing activities pre-deployment.

And, by way of final word, I encourage you to advocate for opportunities to test your software before deployment using humans (i.e. not just relying on a set of “green” results from your automated checks), whether your team is shifting left, shifting right or not dancing at all.

You can find the first two parts of this blog series at:

• Why is software testing important? and
• How does software testing impact software quality?

I’m providing the content in this blog series as part of the “not just for profit” approach of my consultancy business, Dr Lee Consulting. If the way I’m writing about testing resonates with you and you’re looking for help with the testing & quality practices in your organisation, please get in touch and we can discuss whether I’m the right fit for you.

Thanks again to my review team (Paul Seaman and Ky) for their helpful feedback on this post.