work-blog/articles/published/five-essential-lessons.md

Relative to software programming, design, marketing and management, there are comparatively far fewer good books written on the topic of testing and test engineering. In most instances, it is treated as a hygiene component of software development in a few chapters on test construction, or it is described as part of a process management philosophy like Domain Driven Design, or Test Driven Development. However, in spite of what these books would suggest, software testing is in fact, a technical discipline of its own, distinct from all the other roles mentioned here, including software development, and there have been a few well-written books on the topic.

One of the best books an aspiring software tester can read to improve his practice, is the Kaner, Bach, and Pettichord book “Lessons Learned In Software Testing”. This book is a comprehensive bible of 293 battle-tested insights into software testing, accumulated over decades, and drawn from direct experiences as well as academic study. It does much more than just train you to pass a certification test. It is a practical guide to testing as a distinct practice in a way that no certification process could ever hope to be.

There’s no room in a blog post to cover all 293 lessons. However, what I will do here is borrow a technique from Mortimer Adler, and try to distill the book down to the most important lessons that I think every tester really needs to know, if he wants to be a good tester. These “lessons” aren’t one-for-one duplications of lessons from the book, but rather distillations of several, or augmentations of them, to include details I think are missing or obscure. They’re also not the only important lessons. Mostly, they’re just meant to inspire you to go read the original book.

Lesson 1: Testing And “Quality Assurance” Are Different Things

There is a long standing practice in software development to call the tester the “QA”, but “assuring quality” is not what a tester does. His testing may contribute to the quality of the product, but the tester is first and foremost a fact-finder, not an auditor or a gatekeeper (this point will become more apparent, and more important as we proceed).

As noted in lesson 11 in the book: “…test results and bug reports provide information that facilitates the assurance of quality on the project, but that assurance results from the effort of the entire team…”. In other words, product design, product development, and product testing, are all necessary components insuring the quality of the delivered product.

The tester’s role is to provide the information that tells the team whether or not what they say to themselves about the product, is true. When they discover it is not, then the team can have a conversation about what ought to be done about it. Sometimes, this will lead to immediate changes to the product code. Sometimes, nothing more than revising understanding or expectations is enough.

Lesson 2: Testing Is An Empirical Discipline

As I’ve floridly described in a previous post, testing is a practice that is focused heavily on comparing what we think the world is like, to what we actually find when we investigate it. This is similar to the way a scientist works, in one very important way: it begins with questioning. Every good tester will learn to question everything (though, as lesson 7 advices, not always out loud). Among the questions a tester should be asking (of both his team and himself), the first and most common will be, “how do you know?”.

The $20 word encapsulating this particular question is “epistemology”, which in philosophical circles, just means the method by which one justifies one’s beliefs. Lesson 16 calls testing “applied epistemology”. What is meant by this, is that testers take the philosophical study of knowledge (in particular, empirical knowledge), the the scientific study of cognitive psychology, and the discipline of critical thinking, as tools that not just enable the practice of software testing, but enable its excellence. As the authors put it in lesson 17: “Lots of people who never studied these subjects have done good testing. But if you want to be better than good, learn this stuff.”

Lesson 3: Testing Is An Evaluative Discipline

Lesson 24 makes the simile to scientific practice explicit: “All tests are experiments performed to answer a question about the relationship between what a product is and what it should be.” However, that assertion should make something else very clear. Testing is not just discovery, but evaluation.

The right half of the relationship in that statement is answered by ongoing conversations between the members of the development team and their customers (product teams, clients, etc). The left half is discovered by the tester. In the gap between what the product is, and what it should be, lies a normative judgment about it. There is, in fact, a right way and a wrong way to be the software. The degree to which the relationship is consonant, is the degree to which either the product is good for the purpose that has been set out, or the purpose itself is achievable.

For a software product, purpose is the value it provides to the customer. That value is constituted in whatever problem it was built to solve. The degree to which it solves that problem is the measure of its quality. The higher the quality, the tighter the relationship between what the product is, and what the product should be. Or, to put it much more succinctly, we can quote lesson 62: “Quality is value to some person”. Testing is the practice that tells us whether the product is achieving its purpose.

Lesson 4: Typically, Automated Testing Isn’t

Reiterating an important point from my first section, “Lessons Learned” asks this question in Chapter 5: “The role of testing is to gain information. What information is your automation providing?” This question is difficult to answer for most, because test automation wasn’t implemented to provide information. It was implemented as a cost-cutting measure. Often on the misbegotten view that testing is a low-yield activity in the development process, that is generally not worth its cost. This, in turn, is typically because those shops don’t know why they’re testing.

Chapter 5 goes on to answer its own question: “Use automation when it advances the mission of testing.” What is the mission of testing? To gather and disseminate information that is necessary for judging the quality of the product. Or, to put it more succinctly, as I like to say, “automation is a tool, not a goal.” If the tool is not getting you to your goal, stop using it. Lesson 106 reinforces this view: “A test tool won’t teach your testers how to test. If your testing is confused, test tools will reinforce the confusion. Fix your testing processes before automating them.”

To be fair, good unit tests very often do help to get a project to its goal. However, there are many forms of so-called automated testing that do not. Chapter five’s introduction suggests that one way to avoid the mistake of “automation first”, is to first figure out what you want to test: “Design your tests first, before deciding which to automate. This prevents you from falling into the trap of automating tests that are easy to automate, but weak at finding defects.”

What’s interesting is that most of the Lessons in this section of the book are suggesting uses of automation that aren’t actually testing functions. They’re about acceleration, extension, scale, and volume. They’re about ways to use automation to automate what at tester does, not to actually test. To put it another way, automation is about cybernetically extending a human tester. If you have an application for that kind of extension that works, it can be enormously valuable. However, ultimately, Lesson 106 has the last word: “If you buy [a test tool instead of hiring a tester] you may get rid of your testers only to end up with a bunch of busy tool users, a lot of activity, and little or no bug finding. Where’s the value in that?”

Lesson 5: Testers And Programmers Are Not Adversaries

One of the oldest myths in software development, is the depiction of the cackling tester who makes it his life’s mission to make the programmer cry. It even has its own famous Tik-Tok meme now. As funny as this is, it’s completely the wrong way to think about the relationship between the programmer and the tester.

Ultimately, programmer and tester are both there to produce a quality software product. Their roles are not adversarial, they are complementary. To foster this complementarity, mutual respect is necessary. Mutual respect elevates the quality of the relationship between developer and tester, and in so doing, will necessarily elevate the quality of the product.

Lesson 6 counsels the tester to “run with the programmers”, and Lesson 150 urges tester to “understand how programmers think.” It even goes so far as to say “The best way for you to learn how to interact with programmers, is to become one and to work with other programmers as your peers for a while. Come back to testing after you’ve written production code, had it criticised, condemned, and praised by testers, users, managers, and fellow programmers.” While I don’t go quite that far, the implication is clear: nothing is more effective at fostering a sense of shared mission, than walking a mile in your colleagues shoes.

But there is a deeper point to be made here, as well. Namely, found in lesson 39: “You can’t avoid bias, but you can manage it”. One of the insights I’ve gained in my years as a tester, is that you can leverage complementary biases, for the sake of quality. Lesson 39 goes on to list a number of common cognitive biases: Assimilation, Confirmation, Primacy, Representativeness, and so on. The lesson focuses on the testers own awareness of these biases, and treats them as potential hazards. But this is, I think, a mistake. Since programmers and testers are both susceptible to the same biases, but are susceptible to them under different conditions and motivations (see lesson 150), the biases marry up, in the interaction between software and tester.

The best example of this, can be found in confirmation bias. Programmer is eager to see his functionality work. He iterates over his code numerous times until he gets an outcome he is hoping for. Shazam! Success! Commit. Push. The tester, equipped with not just the requirements, but the boundaries and corners of those requirements, iterates over the functionality, until he gets an outcome he is hoping for. Shazam! Bug! Cut. Publish.

You can see this process actually playing out in the Tik-Tok meme. This process repeats as often as necessary, until you get to a point where it’s more costly to keep iterating, than to deploy. In other words, until you’ve reached a point of practical quality. On this model, therefore, the developer should actually be gratified to see test reports with bugs in them. It means there is still an opportunity available for further improving the quality of the product. A goal you both seek.