12 KiB
There is a common hyperbolic stereotype of scientists that paints them as obdurately reductionist killjoys who seek only the schadenfreude found in aggressively shattering the metaphysical wonderment one experiences from the delicate grandure found in a work of art like the universe.
I find this stereotype fascinating, because it describes a very similar phenomenon experienced by many testers. This can be seen in the phrase "negative thinker", often used to characterize the difference between a programmer and a tester. As Bach, Et. al.1 describe in the book, some charge that "testers complain... [and] take a special thrill in delivering bad news". I've even seen a few testers take this identity to heart, revelling in "making developers cry".
But, as Bach, Et. al. correctly point out:
good
In other words, it is a serious mistake to assume that because programmers think in terms of positive goals, and testers think differently, then it must mean that testers oppose those goals. Nothing could be further from the truth. In addition to being just plain wrong, it seems to me this sort of adversarial characterization is also part of the reason why the discipline of testing is in so much trouble.
A better way to understand the mindset of the tester, particularly in relation to the developer, is to understand that they both have the same goal, but they get there from different directions. In effect, the developer is making an argument, or a claim about reality, and the tester wants to help him by seeing if the claim holds up to scrutiny. To put it in simpler terms: If programming is the art of software development, then testing is its science.
And, like any good scientist, at the core of a good tester's character, is his reliance upon the methods of science, his commitment to critical thinking in conjunction with it, and his passion for (and proper understanding of the value of) sharing the discoveries he makes, with his teammates.
The Toolbox Of A Thinker
A central tenet of this mindset, according to Bach, et. al., is an epistemological commitment to a variety of empiricism peculiar to Karl Popper (and, indeed, all of modern science). The authors stress the importance of an awareness of philosophical concepts like Justified True Belief, logical tools like abductive inference, and methods such as Karl Popper's conjecture and refutation (more colloquially understood as "falsificationism").
But they don't stop there. In addition to critical reasoning and empiricism, The book outlines three other forms of thinking essential to the thoughtful tester: Technical, Creative, and Practical. Each of these forms of thinking is punctuated by lessons particular to it. The "technical" lessons identify the need for analytical skills, and the ability to model systems accurately for succesful testing. The "creative" lessons identify the need for lateral thinking, and its role in determining the boundaries of your testing efforts. Lastly, the "practical" lessons identify the need for technical competency, and implicitly warn testers that a limit here will necessarily limit what is possible for them, regardless of how creative they might be. A discussion of these forms of thinking could fill a whole post each on their own (and I've decided, as you'll see later, that I'm going to do just that).
Looking Inward
Before I get into the practical application of this toolbox, I want to highlight one final piece of mental equipment mentioned in this chapter. And, while I hesitate to elevate cognitive psychology in importance above all the other forms of thought, it is very difficult for me to imagine the other forms being very effective without this. Bach, et. al. Acknowledge this importance by giving cognitive psychology its own stand-alone lesson (lesson 18), but they seem only dimly aware of the breadth and depth of the implications of applying this tool, in one's individual life. They state:
Lots of people who never studied these subjects have done good testing. If you want to be better than good, studying cognitive psychology will help you understand the factors that affect your performance as a tester, as well as the factors that affect how people interpret your work.
Help you understand, indeed. I don't think this point could be any more understated. Bach, et. al. tease us by highlighting this as an independent lesson, but leave us with little more than a tepid suggestion.
This lesson, in my view, is about much more than simply obtaining facts about the reliability of senses and memory, or insights into the neurology of cognitive awareness and thoughts, or even individual awareness of cognitive influencers like stress, or fear, or pattern recognition. Though such things are very important to understand, they are secondary to the development self-awareness and empathy in oneself, and cognitive psychology is perhaps the most powerful approach to this need.
Self-awareness and empathy are the third leg of a three-legged stool that includes Popper-style empiricism, and critical reasoning, altogether making up a stable platform from which to ground all relations in life, not just your testing relations. Without self-awarness and empathy, you cannot understand yourself, much less imagine the motivations and experiences of your users, or the developers you work with. If it were up to me, I'd put everyone in tech through at least one course in cognitive psychology, and couple it with "lab work" in a therapist's office.
Bach, et. al., offer the uninspiring goal of getting "better than good", for the effort of understanding cognitive psychology. But, I would argue that if we all took it seriously, humanity as a whole would be much better than good. We'd be fantastic.
Doing Science
With that said, let's explore a little more of what Bach, et. al., have to say about all of this thinking, as it is applied. For starters, they assert in lesson 24, that "all tests are experiments performed to answer some question about the relationship between what the product is, and what it should be." A better way to state this might have been to say, "...between what the product is, and what the your team team asserts that it is".
But your goal as a tester, is not to get to the answer that your team (most probably wants) wants. Your goal, as an empiricist, is to answer the question as precisely and confidently as you can. And, if Popper's view is to be accepted, this is accomplished by executing an experiment that applies a technique of falsification.
This lesson is important, because it sets the stage for an important insight into good testing, and its similarity to good science. The tester is not "validating" a product. He is not setting out to prove the hypothesis by experiment. His job is to falsify specific assertions. Here's a very simple example:
The product team asserts: "Given such-and-such conditions, the login screen includes a functioning link to a password recovery form".
The tester asks himself: not "Is this true?", but rather, "What if this isn't true? How could I discover that it wasn't true? How could I be confident of such a discovery?", and then designs a test -- an experiment -- to answer those questions.
In short, the team offers the conjecture, the tester offers the refutation. In this way (and at the risk of elevating the tester to even more grandiose heights in this review), the tester is quite literally, a scientist. He knows that it is impossible to prove the hypothesis "true" under every circumstance, and that positive approaches to the hypothesis leave him especially vulnerable to cognitive errors like Confirmation Bias (which will be discussed at length later).
One might also suppose that this is another reason why testers are seen as "negative".
The Story Of Testing
Ok, I'm already well over budget on the length of this post. So, I've decided to separate the review of this chapter into multiple posts, which I'll describe more in a moment.
But, to end this post, I'd like spend some time on one seemingly throw-away phrase in lesson 19. A note that deserves a great deal more attention than was given by the authors of this chapter. Bach, et. al. tell us, in lesson 19:
The difference between excellent testing and mediocre testing is how you think: your test design choices, your ability to interpret what you observe, and your ability to tell a compelling story about it. [emphasis added]
So much of this book is absolutely littered with random nuggets of essential wisdom. Little mentions, almost asides, that any casual reader would breeze past, but that make all the difference. The bit that I have highlighted above, is just such a nugget.
What does it mean to "tell a compelling story" about our testing, our test design choices, and our ability to interpret what we observe? Well, let's look at what we know. The world is chock-a-block with good analysts, technicians, mechanics, engineers, researchers, and scientists. But among these multitudes, who are the ones that stand out? The ones we admire? The ones we envy? Who are the ones who inspire us to their view of life, and the world? It is the great storytellers. And, as it happens (and contrary to the popular stereotype), science and technology are positively teaming with examples. Just off the top of my head, I can think of three:
- Carl Sagan, especially his Cosmos TV Series
- Richard Dawkins, especially his book Unweaving The Rainbow
- James Bach, especially his book Secrets of a Buckaneer Scholar
And there is one more example I want to highlight, specifically because they are perhaps the most accomplished testers, particularly as storytellers, in the western world. The show is called Mythbusters. Adam, Jamie, and the other hosts, have been applying the scientific method to empirical questions for 14 years, and the show is still going strong. They ask many of the same kinds of questions that testers do, they must design good tests to answer those questions, and most especially they must tell a compelling story about their observations. You really can't get any more accomplished than that.
So, what do all these folks have in common? What is it that makes their stories compelling, as opposed to simply inspirational or convincing? I have come up with a short list of traits that I think together comprise the necessary ingredients for a compelling story, and a compelling storyteller:
- A commitment to the principles we've talked about in this post: Critical reasoning, scientific empiricism, and self-knowledge.
- A willingness to empathize with your audience, and to regard them as equally curious and intelligent as yourself.
- A strong sense of humility in the light of #1 and #2. This is described in the book, in lesson #40: "...the person easiest to fool is the one who is absolutely convinced he cannot be fooled..."
- A passion for sharing your findings, even in the face of strong opinions or fearful opposition. Mr. Bach is certainly no stranger to this.