Review: “How Google Tests Software” by James Whittaker, Jason Arbon and Jeff Carollo

(Goodreads link here)

I really wanted to like this book. I did. Unfortunately, the reality is a little bit more complex than that. On the one hand, there are lots of really cool, useful ideas in here. On the other hand, I am not really enamoured with the style. It’s self-congratulatory and at the same time weirdly self-conscious. Finally, I think it tries to do too much in one book.

Let me tackle these points in reverse order (that way we end up with a nice set of useful ideas I picked up right at the end of the post!).

Too much for one book to shoulder

This book tries to do an awful lot. It tries to describe the various roles at Google that testers have. These are the main chapter in the book. Around this structure, it tries to fit in lots of other things: introduce people to the culture of testing at Google, describe software testing techniques (e.g. ACC, test-tours), interview questions and advice, some important projects (e.g. test certification), some tools used during the testing (e.g. Selenium Web-Driver, record-playback tools, ) and a view on the future of testing at Google. Whoa – that is a lot of stuff! And it shows. For me, it really strained the original structure in which everything is meant to fit in. Maybe this is due to me reading the book on the Kindle, but I constantly found myself surprised by the next bit of content, unsure of how things were fitting together. Really good, well thought-through parts were followed by “rather meh” parts.

Altogether, this book is a little bit like an intro course to Google testing, where the instructors try to cram all of the important bits in, and then a couple of important bits more, and then some…

Some notes on style and stuff

Normally, I don’t mind too much about style, but here there are some bits worth pointing out about this book, which I found rather odd: I have not really seen an interview with the author and co-authors itself in a book. It also talks a lot about how Google is leading the way in software testing. In a book about Google’s software testing by Google employees, I think that is a little weird.

Anyway, here are my two cents on Google’s test practices: I think it is worth appreciating that they are leading the way on some levels (e.g. a lot of their tools and some processes/practise look pretty awesome), but on other levels I am not too sure: For example, the division of roles between SWE, SET and TE to me felt artificial (even after a long time in the book is spent justifying it).

Essentially, the process they have gone through is…

  • From a large company where there used to be little enthusiasm for automated testing / unit testing and a traditional division between testers and developers

and transformed this towards a more Agile focussed practice where

  • the roles in the team are much more fluid (i.e. developers take responsibilities for testing) and automated testing is held in high regard


  • throwing a large amount of money at the system and using a specialist SET role as a crutch to show “normal developers” the importance of caring about quality

The “future of testing” as they describe it in the last chapter, then, is a future where they can start to throw away the crutch and get rid of the SET role.

The good bits

Enough with the book bashing. I did learn a lot from it after all. Here are the good bits.

  • General spirit

    While I don’t like the tone of the book, I do really like the spirit of the testing culture at Google: Make quality a feature of the main system and make everyone (especially devs) responsible for it. Automate as much of the testing as you possibly can because it scales best.

  • Test certifying a project

    I also like the process of test certification that the book describes (details on this can also be found here): Effectively, it is a way of giving projects within Google a measure of progress how they are doing with their testing. It also gives them a set of steps of what they can tackle next on their way to testing mastery. Finally, it drives visibility of effort across the organisation.

  • Test planning – the Attribute, Component and Capability (ACC) approach

    It’s been a little while since I wrote my last test plan (start of 2008 maybe?) and I know test case design processes have changed quite a bit (with the adoption of Agile). This seems an interesting way of minimising test design. What I particularly like is the way the it gives you a type of heat map where you may want to focus your testing efforts. Here is a short introduction to the topic.

  • This is a tool that I have been meaning to check out. I heard about it before for writing tests for the web. Now that I do a little more with javascript and stuff in my spare time, maybe it is time to have a closer look again and work some web TDD magic…

  • Layers of automation – small, medium and large (unit, integration and system tests)

    Off the back of this, I also finally understood the importance of different layers of automation. To be fair, this is not really the benefit of this book, but rather of a book I’ve been reading at the same time. Having said that, the layering into small, medium and large and their guideline ratios of 70-20-10 now make a whole lot sense!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s