6.5 KiB
Desperation At Scale: Nobody Is Coming To Save You
This week, I attended Test Automation And Continuous Delivery At Scale, a one-day class hosted by The Ministry Of Testing, and headed by Noah Sussman and Dr. Jess Ingrassellino.
The summary on the Ministry of Testing site was enticing enough, and the name Noah Sussman certainly caught my attention, but the "At Scale" in the title was what had really convinced me I needed to attend this class.
Like a moth to the flame, I ran into this class hoping, at last, I would find others who've had to grapple with the hairy and barely tangible problems of testing a patchwork quilt of an application, that's been divided up and dispersed amongst disparate internal teams like so much beef, in a lion's den.
I yearned for a well structured curriculum chock full of dense and important principles and critical technical implementation details that would load up my intellectual tool belt with everything I needed to solve all the test automation challenges facing me any my automation team.
Of course, that's not really even close to what I got. Which is not to suggest that the class was without value. On the contrary, what I did take from it, was well worth every penny invested. It's just confounding how the value I think I'm going to get from these events is never really the value I end up with.
The Fog Of War
Jess and Noah are highly skilled and experienced, and earnestly worked to provide the room with as much insight and expertise as they could muster. Jess did a terrific job of encouraging group discussion, and Noah was chock full of fascinating concepts, and technical tips. However, I definitely got the sense that neither really had a clear idea of what it was they were trying to teach us.
Jess opened the class by surveying the room for an open-ended list of test automation problems experienced by class participants. Many of the most usual suspects of testing ended up on that list: "too many tests", "flakey tests", opaque frameworks, difficult relationships with developers, and so forth. She then promised the class that these "pain points" would all be addressed in the day's lectures. Indeed, most of them did end up being topics of discussion throughout the day (though a few, only tangentially).
But rather than focusing the class, I think this approach actually fundamentally changed the class into something it originally was not. Because, with the exception of a few of Noah's lectures, almost all of the rest of the class seemed to me like on long digression from the title of this class. We spent a great deal of time discussing inter-disciplinary relationship issues, frustrations with product and business team members, what it means to be a tester, how the tester mindset is different from the developer mindset, why testers are often treated like second-class citizens, and other classic favorites on the Woeful Tester Top 40 List.
The Nutty Professor
Noah did his level-best (though it seemed to me a somewhat scatter-brained attempt) to introduce some essential philosophical concepts that lie at the heart of "scaling" any software development endeavor. He opened abruptly with a brief explanation of "Oracles" as they're applied to testing. This is something I thought might be really interesting, as it's a favorite buzzword in "context driven" circles, and I was intrigued to see how the idea could be brought to bear in the implementation of automated tests in a complex environment. But, just as abruptly as the concept was introduced, he terminated it. Almost nothing was said about Oracles after this.
Another concept Noah spent a great deal of time on, was the problem of Intractable Complexity. His description of the problem is a bit different than the understanding I was accustomed to, however. He described, in the abstract, a "complex" system, and argued that the amount of detail needed to describe this system is in fact infinite, because the layers of specificity are effectively infinite, and because "complex" systems are "living" systems, in the sense that every interaction with them changes them in some fundamental way. I could vaguely envision how this could apply to large scale test automation implementations, and it was intriguing to ponder the challenge we face as test automators through this sort of fractal mental lens. But the thin mental strings tethering this concept to anything real were tenuous at best.
The last conceptual presentation for the morning, dealt with epistemology. Attempting to link the presentation to the concept of Intractable Complexity, Noah illustrated in graphics how the domain of the knowable is necessarily finite, and the domain of the unknowables ("unknown unknowns") is necessarily infinite.
To hammer home the idea of combinatorial explosion, as it applies to software development problems, we were taken through an exercise meant to emulate the problem posed by the Abelian Sandpile Model. However, the boards were drawn too large, and the rules of the game were not explained nearly well enough to make any sense of it. It took me a day of research after the class, to understand what it was he was trying to teach us. For that, I suppose, I am quite greatful.
As Noah's "big-ideas" discussions grew more and more abstract, his alternating "tips" presentations detailing the many clever uses of command-line tools like curl, xmlstarlet, and spark, made him seem progressively more schizophrenic, and left me with a serious case of mental whiplash.
Noah's afternoon presentations were almost entirely devoted to the hands-on demonstrations of command-line tools for extracting useful information out of development and ops tools like git and netcat. However, he did attempt one last very brief prescriptive conceptual presentation, specifically on the topic of scaling, in which he argued that the three main challenges of any scaling effort are communication, configuration, and performance. He offered one great insight during this presentation: tools like Jenkins and Github are, fundamentally, communication tools. They are designed to automate the coordination of efforts between engineering teams, which is ultimately a communication task.
Rallying The Troops
Periodically interspersed between