old-blogs/new-testing-blog/test-automation-principles.md

This document is an initial attempt to describe the responsibilities and commitments that the Test Automation team will be making, to the Architecture principles being drafted by Cláudio Ferrete and his team. With regard to automated testing, our team's first core principle should be Designing For Testability, in a contract-driven approach to development.

"Testability", in the context of our tests, essentially means that what we say about our applications should be testable hypotheses. In order for that to be possible, our “contracts” (here at MOO, they are essentially acceptance criteria), need to adhere to the following criteria:

• Consistent - criteria should not contradict each other, for example.
• Complete - if the feature under development includes things like special error messages, then it should be found in requirements
• Unambiguous - “fail gracefully” is pretty ambiguous.
• Quantitative - a requirement like “fast response time" can not be verified
• Verifiable in practice - it should be feasible not only in theory but also in practice with limited resources

"Testability", in the context of our applications, means that the product about which we are making testable hypotheses, should be engineered to facilitate the testing of those hypotheses. To that end, we should begin to judge the products we generate each sprint, for their "testability" according to the following criteria:

• Controllability: The degree to which it is possible to control the state of the application, as required for testing.
• Observability: The degree to which it is possible to observe (intermediate and final) test results.
• Isolatability: The degree to which the application can be tested in isolation.
• Discreteness: The degree to which the tested application has a single, well defined responsibility.
• Understandability: The degree to which the tested application is documented or self-explaining.
• Automatability: The degree to which it is possible to automate testing of the application under test.
• Heterogeneity: The degree to which the use of diverse technologies, test methods, and tools in parallel are required for testing.