Testing & “Exponential Organizations” (Salim Ismail, Michael S. Malone & Yuri Van Geest)

I’m not sure how I came across the book Exponential Organizations (by Salim Ismail, Michael S. Malone & Yuri Van Geest) but it ended up on my library reservation list and was a fairly quick read. The book’s central theme is that new technologies allow for a new type of organization – the “ExO” (Exponential Organization) – that can out-achieve more traditional styles of company. The authors claim that:

An ExO can eliminate the incremental, linear way traditional companies get bigger, leveraging assets like community, big data, algorithms, and new technology into achieving performance benchmarks ten times better than its peers.

This blog post isn’t intended to be an in-depth review of the book which, although I found interesting, was far too laden with buzzwords to make it an enjoyable (or even credible) read at times. The content hasn’t aged well, as you might expect when it contains case studies of these hyper-growth companies – many of which went on to implode. A new study of ExO’s from 2021 will form the second edition of the book coming later in 2022, though.

The motivation for this blog post arose from the following quote (which appears on page 140 of the 2014 paperback edition):

One of the reasons Facebook has been so successful is the inherent trust that the company has placed in its people. At most software companies (and certainly the larger ones), a new software release goes through layers upon layers of unit testing, system testing and integration testing, usually administered by separate quality assurance departments. At Facebook, however, development teams enjoy the full trust of management. Any team can release new code onto the live site without oversight. As a management style, it seems counterintuitive, but with individual reputations at stake – and no-one else to catch shoddy coding – Facebook teams end up working that much harder to ensure there are no errors. The result is that Facebook has been able to release code of unimaginable complexity faster than any other company in Silicon Valley history. In the process, it has seriously raised the bar.

I acknowledge that the authors of this book are not well versed in software testing and the focus of their book is not software development. Having said that, writing about testing as they’ve done here is potentially damaging in the broader context of those tangential to software development who might be misled by such claims about testing. Let’s unpack this a little more.

The idea that “separate quality assurance departments” were still the norm when this book was written (2014) doesn’t feel quite right to me. The agile train was already rolling along by then and the move to having testers embedded within development teams was well underway. What they’re describing at Facebook sounds more in line with what Microsoft, as an example, were doing around this time with their move to SDETs (Software Development Engineers in Test) as a different model to having embedded testers focused on the more human aspects of testing.

The idea that “development teams enjoy the full trust of management” and “Any team can release new code onto the live site without oversight” is interesting with the benefit of hindsight, given the many public issues Facebook has had around some of the features and capabilities included within its platform. There have been many questions raised around the ethics of Facebook’s algorithms and data management (e.g. the Cambridge Analytica scandal), perhaps unintended consequences of the free rein that has resulted from this level of trust in the developers.

It’s a surprisingly common claim that developers will do a better job of their own testing when there is no obvious safety net being provided by dedicated testers. I’ve not seen evidence to support this but acknowledge that there might be some truth to the claim for some developers. As a general argument, though, it doesn’t feel as strong to me as arguing that people specializing in testing can both help developers to improve their own testing game while also adding their expertise in human testing at a higher level. And, of course, it’s nonsense to suggest that any amount of hard work – by developers, testers or anybody else – can “ensure there are no errors”.

While I can’t comment on the validity of the claim that Facebook has released complex software “faster than any other company in Silicon Valley history”, it doesn’t seem to me like a claim that has much relevance even if it’s true. The claim of “unimaginable complexity”, though, is much more believable; given the benefit of hindsight and the evidence that suggests they probably don’t fully understand what they’ve built either (and we know that there are emergent behaviours inherent in complex software products, as covered expertly by James Christie in his many blog posts on this topic).

The closing sentence claiming that Facebook has “seriously raised the bar” doesn’t provide any context, so what might the authors be referring to here? Has Facebook raised the bar in testing practice? Or in frequently releasing well-considered, ethically-responsible features to its users? Or in some other way? I don’t consider Facebook to be high quality software or a case study of what great testing looks like, but maybe the authors had a different bar in mind that has been raised by Facebook in the area of software development/delivery/testing.

In wrapping up this short post, it was timely that Michael Bolton posted on LinkedIn about the subject matter that is so often lacking in any discussion or writing around software testing today – and his observations cover this paragraph on testing at Facebook perfectly. I encourage you to read his LinkedIn post.

“The Great Post Office Scandal” (Nick Wallis)

I’ve been following the story of the UK Post Office and its dubious prosecutions of sub-postmasters based on “evidence” of their wrongdoings from its IT system, Horizon, for some years.

My mother worked in the Post Office all of her working life and I also used to work there part-time during school and university holidays. There were no computer terminals on the counters back then; it was all very much paper trail accounting and I remember working on the big ledger when it came to balancing the weekly account every Wednesday afternoon (a process that often continued well into the evening).

Nick Wallis’s book covers the story in incredible detail, describing how the Post Office’s Horizon system (built by Fujitsu under an outsourcing arrangement) was badly managed by both the Post Office and Fujitsu (along with poor Government oversight) and resulted in thousands of innocent people having their lives turned upside down. It is both a moving account of the personal costs shouldered by so many individuals as well as being a reference piece for all of us in IT when it comes to governance, the importance of taking bugs seriously, and having the courage to speak up even if the implications of doing so might be personally difficult.

It’s amazing to think this story might never have been told – and justice never been served – were it not for a few heroes who stepped up, made their voices heard and fought to have the truth exposed. The author’s dedication to telling this story is commendable and he’s done an incredible job of documenting the many travesties that comprise the full awfulness of this sorry tale. This case is yet another example of the truth of Margaret Read’s quote:

Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it’s the only thing that ever has.

One of the more surprising aspects of the story for me was the fact that very complex IT systems like Horizon have been considered in UK law (since 1990) to be “mechanical instruments” and they’re assumed to be working correctly unless shown otherwise. This was a key factor in the data shown by Horizon being trusted over the word of sub-postmasters (many of whom had been in the loyal service of the Post Office in small communities for decades).

Jones wanted the Law Commission’s legal presumption (that ‘in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time’ [from 1990]) modified to reflect reality. He told the minister, ‘If people found it difficult to prove a computer was operating reliably in the early 1990s, we can only imagine how difficult it might be to do that today, with the likes of machine-learning algorithms coming to conclusions for reasons even the computer programmer doesn’t understand.’

Darren Jones, chair of the BEIS Select Committee, p. 456 of “The Great Post Office Scandal”

It’s now clear that the complex systems we all build and engage with today (and even back when Horizon was first rolled out) have emergent behaviours that we can’t be predicted. The Post Office’s continued denial that there were any bugs in Horizon (and Fujitsu’s lack of co-operation in providing the evidence to the contrary) seems utterly ridiculous – and it was this denial that allowed so many miscarriages of justice in prosecuting people based on the claimed infallibility of Horizon.

Program testing can be used to show the presence of bugs, but never to show their absence!

Edsger W. Dijkstra

Reading this story really made me think about what the onus on testers is in terms of revealing important problems and advocating for them to be addressed. The tragic cases described in the book illustrate how important it is for testing to be focused on finding important problems in the software under test, not just proving that it passes some big suite of algorithmic checks. Fujitsu, under duress, eventually had to disclose sets of bug reports from the Horizon system and acknowledged that there were known bugs that could have resulted in the balance discrepancies that resulted in so many prosecutions for theft. There are of course much bigger questions to be answered as to why these bugs didn’t get fixed. As a tester raising an issue, there’s only so far you can go in advocating for that issue to be addressed and your ability to do that is highly context-dependent. In this case, even if the testers were doing a great job of finding and raising important problems and advocating for them to be fixed, the toxic swill of Fujitsu, Post Office and government in which everyone was swimming obviously made it very difficult for those problems to get the attention they deserved.

Coming back to my anchors that are the principles of context-driven testing, these seem particularly relevant:

• People, working together, are the most important part of any project’s context.
• Projects unfold over time in ways that are often not predictable.
• Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

I think part of our job as testers is not only to test the software, but also to test the project and the processes that form the context around our development of the software. Pointing out problems in the project is no easy task, especially in some contexts. But, by bearing in mind cases like the Post Office scandal, maybe we can all find more courage to speak up and share our concerns – doing so could quite literally be the difference between life and death for someone negatively impacted by the system we’re working on.

It would be remiss of me not to mention the amazing work of James Christie in discussing many aspects of the Post Office scandal, bringing his unique experience in both auditing and software testing to dig deep into the issues at hand. I strongly encourage you to read his many blog posts on this story (noting that he has also written an excellent review of the book).

“The Great Post Office Scandal” is available direct from the publisher and the author maintains the Post Office Scandal website to share all the latest news of what is, incredibly, still an ongoing story.