10 KiB
What is a test?
To put it succinctly, a 'test' is an attempt to answer a question. However, only certain kinds of questions. Two kinds, to be precise: discovery, and comparison. In the former case, I begin with no (or few) preconcieved notions, and seek information with which to form an opinion about the object of my attention. It is the "what if?" question. In the latter case, I begin with an expectation and I seek information that either confirms or denies my expectation. This question challenges assertions or assumptions. It is the "is that so?" question.
In both cases, I am seeking a certain kind of knowledge when I ask these questions: empirical knowledge; facts about the actual state of affairs in the real world. However, the method by which I obtain this knowledge -- the way in which I attempt to answer the questions -- will depend on the type of question. The "is that so?" questions -- comparison questions -- are best answered by a method we familiarly call experimentation, while the "what if?" questions -- discovery questions -- are best answered by a method of exploration. When we act out either the method of experimentation or the method of exploration, we are engaging in a test. Thus, when we think of a "test", we should be thinking of an activity, not an artifact.
Our task, as testers, is to be sure we're engaging in these activities effectively. To do that, we should also be asking ourselves questions about questions. What are the questions that matter to this particular piece of work? What is the best way to ask those questions? What sorts of answers might I expect to get? What is the best way to deal with answers I don't expect? What are the implications of answers I do expect? What's more, each of these two methods, experimentation and exploration, come with their own set of risks and best practices testers should be aware of. Keeping them in mind will help you ask and answer those questions about questions.
Experimentation
As the label implies, experimentation is the process of hypothesis testing. There is a load of good literature on the topic of hypothesis crafting, and experiment design. For our purposes, a hypothesis is a question about the application under consideration, stated in the form of an assertion. For example, "Users who have previous business card orders can see them in their order history", and our task is to answer the question "is this so?". This may seem like a straightforward problem. But, as Karl Popper has explained at length in Conjectures and Refutations, if we come at this problem from the wrong direction, we may not actually answer our question -- and even worse, may not realize that we haven't actually answered it. The way he put it:
Every genuine test of a theory is an attempt to falsify it, or to refute it. Testability is falsifiability; but there are degrees of testability: some theories are more testable, more exposed to refutation, than others; they take, as it were, greater risks.
But why are the only "genuine" tests, ones in which we attempt to falsify an assertion? Well, because confirmation bias will delude us into thinking we've succeeded, when we may not have. As Popper puts it, again:
It is easy to obtain confirmations, or verifications, for nearly every theory -- if we look for confirmations... Confirmations should count only if they are the result of risky predictions; that is to say, if, unenlightened by the theory in question, we should have expected an event which was incompatible with the theory -- an event which would have refuted the theory.
In other words, the most effective test of our assertion would not be "is this so?", but rather, "is there some way I can show that this isn't so?". This helps us avoid the problem of confirmation bias, and helps to keep our claims about the health of the application tentative and attenuates our confidence to a level more appropriate to our actual understanding of the application.
This is a difficult attitude for many people to adopt. The reason is, because it requires one to dwell in a continuous state of some degree of uncertainty. Why is this? Because falsification does not justify positive claims. Applying this to our example, "Users who have previous business card orders can see them in their order history", falsification will not prove this to be true. It will only prove (in the best case scenario) that we cannot show that it is false. But there could be a context within which this assertion is false (we just haven't found it yet).
As we can see, hypothesis testing -- experimentation -- requires not only knowing what questions to ask, but also knowing how best to ask them. What we want to accomplish, ultimately, is to narrow the delta between the model of the application in our minds, and the actual application in reality. The better the questions, the narrower that gap will be.
But what do we do when we don't have anything we're trying to compare? How do we discover more about the application in the absence of a specific question to ask? We ask a different kind of question...
Exploration
Interestingly, there is very little literature on the topic of exploration, outside of software testing and travel writing. As much as we romanticize the idea of explorers and adventurers, it seems very few of us are willing to take on the role, and talk about it (at least, in print). The two best books on the subject come from James Whittaker and Elisabeth Hendrickson. Hendrickson applies precisely the metaphor of tourism to exploratory testing in her preface, but is careful to point out that meandering tourism is not quite enough:
When I travel, I always reserve at least a little time to wander the back streets and find hidden gems. I escape the tourist areas to find the restaurants where only locals go and the shops that cater to everyday needs. It is in these meanderings that I begin to understand the local culture and become acquainted with the real character of the place. It’s true of software as well. If you want to learn about the real capabilities and limitations of your software, you have to navigate off the beaten path. Yet there is a difference between wandering at random and real exploring... If you meander without direction or purpose, you’ll spend much time wandering for precious little insight: you’ll be lost.
To employ another adventure metaphor, the difference between the meandering experiences of the software user, and the careful explorations of the software tester, is the difference between the tourist and the woodsman-tracker. I'll let Hendrickson expand on this:
You watch carefully, looking for subtle clues about where there might be a nest of bugs lurking. Observing is crucial: the better you are at observing, the more you’ll learn. It’s also harder than it sounds. You have to look past what you expect or hope to see in order to see what’s really happening.
But why are we exploring? This gets us back to the thesis of this post. There are questions we want to ask, and that we want to find answers to. The overall goal is learning. Exploration that has no initial focus in mind may or may not provide us an opportunity to learn what we want to learn. As such, while it begins as an open-ended process, effective exploration will rapidly narrow the range of possibilities, and focus your effort. This is what Hendrickson calls "steering":
You use your curiosity, fueled by what you’ve learned so far, to suggest the next most interesting piece of information to uncover. Steering while focusing on the most important information to discover is one of the core skills of a master explorer.
What is an "interesting" piece of information? That which will answer a question. Where does this question come from? From the results of your initial exploration combined with your curiosity. How does that initial exploration begin? Hendrickson suggests what she calls the "charter" approach. This is analagous to writing a feature story, only it is designed to focus your "expedition" and prepare it to succeed (in the same way that Jefferson's charter to Lewis and Clarke did for them). Similar to an Agile story, Hendrickson suggests a template of the form:
Explore: {target}
With: {resources}
To Discover: {information}
Her specific model isn't the point, however. The point here, is that the charter, like the story, confines us to a finite space, and gives us a goal. There is something else that needs to be highlighted, that Hendrickson doesn't really address, though. Using her template, What if we had a charter that read:
Explore: the northwest territory
With: survival gear, weapons, excavation equipment, and a team of men
To Discover: Gold, coal, and other precious metals
Immediately, something stands out. The need for technical expertise that extends beyond just being able to survive in the wilderness, and track our position. We need to know at least a little something about geology, about mining, about the local politics and tribes, about team management, and perhaps other skills like smelting or curing. Without a basic remediation in these skills, I might wander the wilderness of Missouri and Montana for a decade, and never find anything, even with an explicit charter.
Hendrickson's example in the text, is a simple security exploration, in which she even includes specific details about what strings to insert into fields for a SQL injection. So, she seems at least partly aware of the need for this kind of out-of-band expertise, but didn't address it explicitly.
This is a bit off my point, however. The reason I am raising this issue, is because this (as well as the segment on experimentation) raises a new essential question, closely related to what is a test. Namely, what is a tester? What is her skillset? What should he be expected to know? Whatever that is, it is beyond the scope of this article, and given how long I've gone on already, I think I'll make that my topic for next time.