The annual conference of the Association for Software Testing (AST) took its first step outside of North America in 2017, with the CASTx conference in Sydney on February 20 & 21. Since I align myself with the context-driven principles advocated by the AST, I decided to attend the event’s conference day on the 21st (disclaimer: I submitted a track session proposal for this conference but it was not accepted.)
The conference was held in the stunning Art Deco surrounds of the Grace Hotel in the Sydney CBD and drew a crowd of about 90, mainly from Australia and New Zealand but also with a decent international contingent (including representatives of the AST). The Twitter hashtag for the event was #castx17 and this was fairly active across the conference and in the days since.
The full event consisted of a first day of tutorials (a choice of three, by Michael Bolton, Goranka Bjedov and Abigail Bangser & Mark Winteringham) followed by a single conference day formed of book-ending keynotes sandwiching one-hour track sessions. The track sessions were in typical peer conference style, with forty minutes for the presentation followed by twenty minutes of “open season” (facilitated question and answer time, following the K-cards approach).
My conference day turned out to include:
- Conference opening by Ilari Henrik Aegerter (board member of the AST), Anne-Marie Charrett (conference program chair) and Eric Proegler (board member and treasurer of the AST).
- Opening keynote came from Goranka Bjedov (of Facebook), with “Managing Capacity and Performance in a Large Scale Production Environment”.
- Track session “Rise of the Machine (Learning)” from Stephanie Wilson (of Xero)
- Track session “Testing with Humans: How Atlassian Validates Its Products With Customers” by Georgie Bottomley (of Atlassian)
- Track session “To Boldly Go: Taking the Enterprise to SBTM” by Aaron Hodder (of Assurity Consulting NZ)
- Track session “Auditing Agile Projects” by Michelle Moffat (of Tyro Payments)
- Closing keynote by Michael Bolton (of Developsense) with “The Secret Life of Automation”
The opening keynote was fantastic. I last heard Goranka speak when she keynoted the STANZ conference here in 2011. She started off by saying how well Facebook had prepared for the US elections in terms of handling load (and the coincidental additional load arising from India’s ban on large currency notes), but then told the story of how around half of all Facebook users had been declared dead just a few days after the election (an unfortunate by-product of releasing their new “memorial” feature that didn’t actually bother to check that the member was dead before showing the memorial!). This was an example of her theme that Facebook doesn’t care about quality and such changes can be made by developers without being discovered, but their resolution times are fast when such problems immediately start being reported by their users. The stats she provided about Facebook load were incredible – 1.7 billion monthly active users for the main site, around 1 billion for each of WhatsApp and Messenger, plus around 0.5 billion for Instagram. Facebook now has the largest photo storage in the world and already holds more video content than YouTube. Her 2013 stats showed, per 30 minutes, their infrastructure handled 108 billion MySQL queries, the upload of 10 million photos and scanned 105TB with Hive! This load is handled by Facebook’s private cloud built in ten locations across the US and Europe. Servers are all Linux and all data centres are powered using green power (and it was interesting to note that they rely on evaporative cooling to keep power usage down). The reasons for a lack of an Australian data centre became obvious when Goranka talked about the big long-term power contracts they require and also “world class internet” (at which point the room burst into laughter). Details of all the server specifications can be found at OpenCompute.org Her objectives in managing capacity and performance are: low latency for users, the ability to launch things quickly (succeed or fail quickly, don’t worry about efficiency, don’t care about quality) and conservation (in terms of power, money, computers, network and developer time). Her goals are: right things running on the right gear, running efficiently, knowing if something is broken or about to break, and knowing why something is growing. She also talked through their load testing approach – which runs every second of every day – and their testing around shutting down an entire region to be ready for disasters. Although this wasn’t really a pure testing talk, it was fascinating to learn more about the Facebook infrastructure and how it is managed and evolving. It was made all the more interesting by Goranka’s irreverent style – she openly admitted to not being a Facebook user and cannot understand why people want to post photos of cats and their lunches on the internet!
From the tracks, it was interesting to hear about Xero’s QA mission statement, viz. “Influence Xero culture to be more quality oriented and transform software from “good” to “wow”” (Stephanie Wilson’s talk) and it was surprising to me to learn that Atlassian was not doing any decent sort of UX research until so recently (from Georgie Bottomley’s talk), but maybe that explains some of the quirky interactions we’ve all come to known and love in JIRA!
I’ve seen Aaron Hodder present a few times before and he always delivers real experiences with a unique insight – and this session was no exception. His talk was a fascinating insight into dysfunctional client/vendor contract-heavy enterprise IT environments. The novel approach he came up with at Assurity was session-based test management in a light disguise in order to make it palatable in its terminology and reporting, but it was very cleverly done and the project sounds like it’s in much better shape than it was as a result. A really good talk with handy takeaways, and not just for a tester finding themselves in the unfortunate position of being in a project like the one Aaron experienced.
Michelle Moffat presented the idea that the agile practices are, in audit terms, controls and it is the way evidence is gathered in this environment that is so different – she uses photos, videos, attends meetings and automated controls (for example, from the build system) rather than relying on the creation of documents. This was a really interesting talk and it was great to see someone from well outside of our sphere taking on the ideas of agile and finding ways to meet her auditing responsibility without imposing any additional work on the teams doing the development and testing.
Michael Bolton’s closing keynote was a highlight of my day and he used his time well, offering us his usual thought-provoking content delivered with theatre. Michael’s first “secret” was that a test cannot be automated and automated testing does not exist. He made the excellent point that if we keep talking about automated testing, then people will continue to believe that it does exist. He has also observed that people focus on the How and What of automated button-pushing, but rarely the Why. He identified some common automation (anti)patterns and noted that “tools are helping us to do more lousy, shallow testing faster and worse than ever before”! He revealed a few more secrets along the way (such as there being no such thing as “flaky” checks) before his time ran out all too soon.
There were a few takeaways for me from this conference:
- There is a shift in focus for testing as SaaS and continuous delivery shifts the ability to respond to problems in production much more quickly and easily than ever before.
- The “open season” discussion time after each presentation was, as usual, a great success and is a really good way of getting some deeper Q&A going than in more traditionally-run conferences.
- It’s great to have a context-driven testing conference on Australian soil and the AST are to be commended for taking the chance on running such an event (that said, the awareness of what context-driven testing means in practice seemed surprisingly low in the audience).
- The AST still seems to struggle with meeting its mission (viz. “to advance the understanding of the science and practice of software testing according to context-driven principles”) and I personally didn’t see how some of the track sessions on offer in this conference (interesting though they were) worked towards achieving that mission.
In summary, I’m glad I attended CASTx and it was good to see the level of support for AST’s first international conference event, hopefully their first of many to help broaden the appeal and reach of the AST’s effort in advocating for context-driven testing.
An excellent set of summary photos has been put together from Twitter, at https://twitter.com/i/moments/833978066607050752
A worthwhile 40-minute roundtable discussion with five CASTx speakers/organizers (viz. Abigail Bangser, Mark Winteringham, Aaron Hodder, Anne-Marie Charrett and Ilari Henrik Aegerter) can also be heard at https://t.co/A0CuXGAdd7