Reading matter & Simple Rules

I see two very common patterns among testers when it comes to their reading habits – most have never read a book dedicated to software testing and even fewer have thought to read books in other fields (such as psychology and social sciences) that would help them understand the problems of performing great testing. This is an amazing state of affairs for “professionals”, especially when there is so much excellent material just waiting to be read, absorbed and applied from good practitioners in both our field and related ones.

So, if you haven’t read any software testing books, I suggest you do so – something like Lessons Learned in Software Testing: A Context-Driven Approach (Cem Kaner, James Bach, Brett Pettichord) or Perfect Software: And Other Illusions about Testing (Jerry Weinberg) would be pretty good places to start. But also think about some broader reading, with Thinking Fast and Slow (Daniel Kahneman) and Tacit and Explicit Knowledge (Harry Collins) being regularly cited as helpful texts for testers to broaden their horizons.

As a case in point, I recently stumbled across the book Simple Rules: How to Thrive in a Complex World (by Donald Sull and Kathleen M. Eisenhardt) as it came on my wife’s radar in the trading world and she thought it might be interesting in relation to software testing too.

 

The book recognizes the complexity of modern life and how simple rules can help us to cut through some of that complexity, rather than coming up with increasingly complicated solutions.

For most of us, complexity is a problem we struggle to manage in our own lives every day… By limiting the number of guidelines, simple rules help maintain a strict focus on what matters most while remaining easy to remember and use. In a wide range of decisions, simple rules can guide choice while leaving ample room to exercise judgement and creativity.

I liked the idea that simplification leaves room for judgement and creativity, this resonating closely with the benefits I see in using exploratory testing (over detailed scripted tests).

Fighting complexity is an ongoing battle that can wear us down. Disheartened, people tolerate complicated solutions that don’t work, or cling to overly simplistic narratives… that deny the interdependencies characterizing modern life.

Again, I immediately thought of how the world of context-driven testing acknowledges this complexity and interdependency, but doesn’t try to come up with one size fits all hefty processes or standards to address these realities.

The authors talk about four key aspects of simple rules:

First off, simple rules consist of a handful of guidelines applied to a specific activity or decision… They’re intended to offer a limited amount of guidance, so there’s no need for a lot of them. Keeping the number of rules to a handful forces you to focus on what matters most. You might think that capping the number of rules would result in guidelines that are too simplistic to solve complex problems. Not so. In many situations, a handful of factors matter a great deal, while a long tail of peripheral variables can be safely ignored.

Second, simple rules are tailored to the situations of the particular people who will use them, versus one-size-fits-all rules that apply to everyone.

Third, simple rules are applied to a single well-defined activity or decision… Simple rules are most effective when they apply to critical activities or decisions that represent bottlenecks to accomplishing an important goal.

Finally, simple rules give concrete guidance without being overly prescriptive… Simple rules leave room to exercise creativity and pursue unanticipated opportunities.

They include some interesting examples of where such simple rules have been successful in battling complexity and resulted in good decision making, without being so overly prescriptive as to be useless or ignored.

Simple rules work best when flexibility matters more than consistency… Both flexibility and consistency have their advantages, but increasing one reduces the other. Then a large number of highly directive rules… are the best tools to use. Detailed rules are particularly useful for avoiding catastrophic errors, such as plane crashes, mishaps in nuclear power plants, and surgical deaths that result from known causes. Pilots are fond of saying that “checklists are written in blood”, a reference to how these lists are developed in the first place.

Checklists are a valuable tool in testing, just as they are for doctors and pilots. But flexibility is also very important in testing, acknowledging that there is much we don’t know about a system (despite whatever level of specification you might have) until we start to explore its behaviour.

Simple rules impose a threshold level of structure while avoiding the rigidity that results from too many restrictions. The resulting flexibility makes it easier to adapt to changing circumstances and seize fleeting opportunities. Simple rules can also produce better decisions than more complicated models can, particularly when time and information are limited.

Have you ever been in a situation during software development (including testing) where time and information were limited?! I like the idea of seizing “fleeting opportunities” as might be revealed during exploratory testing.

In the context-driven testing world, you will hear the term heuristics a lot and, for those of us who are practitioners, using heuristics turns out to be very powerful in our testing. The authors of this book also recognize the benefits of “rules of thumb”:

Simple rules enable people to make quick, reasonably accurate decisions that require less effort than more complicated approaches. When there is not much time or when information is at a minimum, these rules of thumb can save the day. Simple rules work because they focus on key aspects of a decision while ignoring peripheral considerations. By using simple rules, people can function without constantly stopping to rethink every aspect of a decision every time they make it.

Rules of thumb are often viewed a second-rate measures for use when people lack the time or information to come to a more considered judgement. Indeed, the term rule of thumb refers to a rough and practical approach that is not particularly accurate or reliable in every situation…

Counterintuitive as it may sound, simple rules can outperform more analytically complicated and information-intensive approaches even when there is ample time and information to make a decision. This is especially true in situations where links between cause and effect are poorly understood, when important variables are highly correlated, when a few factors matter most, and when a gap exists between knowing what to do and actually doing it. Simple rules do not trump complicated models every time, but they do so more often than you might think.

Heuristics are a powerful decision-making tool, often matching or even outperforming more sophisticated approaches. They are easy to remember and use, attributes that increase the odds that people will not only make the right choice, but translate their decision into action and stick with it over time.

The case for using simple rules seems pretty strong and I’ve certainly found heuristics to be a powerful means of generating test ideas during exploratory testing.

The authors ask the obvious question:

 

 

These rules also raise an important question: if simple rules are so effective in so many situations, why do complex solutions remain so prevalent? Regulators churn out ever more detailed rules, personnel departments promulgate thick policy manuals, and self-styled experts promote ever more arcane diet and exercise regimes. People crave simplicity and… simple rules often outperform more complicated approaches. Why aren’t simple rules even more common? What are the obstacles to simplifying our lives, corporations and societies? And, more importantly, how can we overcome these obstacles to achieve simplicity?

While most testers I’ve met are keen to simplify the way they work, they are often caught up in highly bureaucratic procedures and heavyweight documentation that result in less time spent actually testing than they would prefer.

The first obstacle is the effort required to develop simple rules. Like most worthwhile endeavours, it takes time and energy to get them right. The process of developing simple rules requires ruthless prioritization – honing in on the essential and decluttering the peripheral… The payoffs of simplification often dwarf the costs of getting there.

The people who benefit from complexity pose a second obstacle to simplicity. The costs of complex solutions are distributed across many people while the benefits of complexity tend to be concentrated in the hands of a few. These beneficiaries have, as a consequence, strong incentives to resist simplification. Much of the complexity of the US tax code, for example, exists because special interest groups secure tax breaks… that benefit a small number of individuals. These special interest groups obviously benefit from complexity, but so do the lobbyists who make their case to legislators, as well as the lawmakers themselves. After creating a labyrinth of rules, regulators and politicians often walk through the revolving door to join the companies they formally supervised.

As in many other industries, the software testing industry has seen the rise of certifications and standards that appear to resist simplicity and generate lots of work for the bodies overseeing them.

The third obstacle to simplicity is what we call the “myth of requisite complexity,” the mistaken belief that complex problems demand complicated solutions. There are, naturally, situations when complicated solutions are appropriate (e.g. detailed checklists used by pilots and surgeons). But detailed rules and regulations aren’t the only possible way to deal with complexity… Complicated solutions should be a considered choice, not the result of regulatory autopilot.

Often, complex rules and regulations arise out of a distrust of human nature. If people cannot be trusted to do the right thing, detailed regulations are necessary to prevent malfeasance. Many corporations, for example, rely on thick policy manuals to control people who might abuse their discretion. But these bad apples represent a tiny fraction of all employees.” Talking about Netflix as an example, “the company’s policy for expenses, travel, gifts, and conducting personal business at work… was reduced to four rules: (1) expense what you would not otherwise spend, (2) travel as if it were your own money, (3) disclose nontrivial gifts from vendors, and (4) do personal stuff at work when it is inefficient not to.

There are other good examples in the book of where simple solutions actually work, where complex problems don’t necessary demand complicated solutions. Good software testing is a complex problem (as we all know!) but we can tackle this problem in simple ways that add genuine value, rather than always resorting to complicated approaches, processes or standards of how to go about our work.

As they come to close out their book, the authors say:

..Simple rules work because they provide a threshold level of structure while leaving ample scope to exercise discretion. Complex rules, in contrast, attempt to anticipate every contingency and dictate what to do in each scenario, thereby reducing people to automatons who do what they are told. But human discretion is not a defect to be eliminated, it is our greatest hope in the battle against complexity. Close to the facts on the ground, individuals can draw on their judgement and creativity to manage risks and seize unexpected opportunities. The latitude to exercise discretion not only makes simple rules effective, it makes them attractive. People thrive given the opportunity to apply their judgement and creativity to the situations they face day to day. And if they benefit from simple rules, they are more likely to use them and use them well.

These are all great arguments for the use of more exploratory approaches to testing, where the tester is in control of their own destiny, “human discretion is not a defect to be eliminated”!

I really enjoyed this book and (as I’ve summarized above) saw many parallels with the world of software testing. It’s great to see a book that isn’t anything to do with software testing providing such insight and applicable ideas. Reading this also gave me hope that simplicity can be seen as a positive thing and that pragmatic approaches to software testing (for which I am a strong advocate) have the ability to add huge value to software development projects.