From 19 till 21 April 2013, the third peer conference of the Dutch Exploratory Workshop on Testing was held in Driebergen, The Netherlands. In a talk I did during that conference, I made a slightly emotional appeal to start looking at software testing skills, in order to explore the question of what makes a good tester.
The particular skill I mentioned in my talk was reading. Some time ago, I did an assignment in which the reviewing of functional documentation played a major part. In preparation for this assignment, I was asked to take a close look at chapter 15 of the book TMap Next (Koomen, 2006). TMap Next represents a test management approach that is widely used in the Netherlands. Chapter 15 is on ‘Toetstechnieken’ which is translated (in the English edition) as ‘Evaluation techniques’. It was suggested that the information in the chapter would prepare me for the task of reviewing functional documentation.
Building a mental picture through reading
While working on my assignment, reviewing the documentation, I found that I was conducting an investigation into the testability of the design. Now testability is something that may have different meanings depending on who you ask. Testability for me meant that the design, the large set of documents that was placed before me, had to be internally consistent and consistent with relevant standards and related documents. Furthermore the structure of the design had to be complete and correct. This, among other things, meant that flows through the design could be followed without gaps, disconnects or open ends. A testable design, in my opinion, was a design based on which tests could be created without leaving too much room for interpretation and thus ambiguity. I was not actually looking for the functional correctness of the design with regards to user requirements. For several reasons, that would have severely complicated the scope of my review.
It turned out that, when reviewing for consistency and structure, many things need to be considered. Based on nothing more than words and diagrams it should be possible to create a mental model of the system and operate that system within the mind. We may have a model of the actual system in our minds. Such a model may contain a hierarchical structure, interacting components and flows of information. So concepts of systems need to be discovered and checked with what’s on paper.
But we may also build different models, based upon the text. Such are models of the organisation, models of how the organisation perceives reality, and models of how the organisation deals with its knowledge, how it describes what’s meaningful and special. All of these models can be investigated through interacting with the text.
Testers as first-rate readers
Reviewing documentation is a process of questioning and evaluation. In that respect it is like software testing, which is the art of skillful investigation through experimentation. A greater awareness of how we read, can aid this investigation. There are models, such as SQ3R (Survey, Question, Read, Recite, and Review, Robinson, 1946), that emphasize the investigative part of reading. There is also the famous book on how to read a book (Adler, 1940) which touches upon many of the themes that are important in testing, such as analytical reading and critical reading, but also prejudice and judgment.
Reading at a more detailed level can provide even more information. A heuristic such as ‘Mary had a little lamb’ introduced by Gerald Weinberg and Donald Gause (Weinberg, 1989) focuses on how the meaning of a sentence changes, whenever a different word is stressed.
The most fundamental skill that aids the creation of a mental model from paper, is reading. Actually – considering the fact that software testers are continually confronted with written material – reading should be a skill with which the average tester is liberally bestowed. We should be first-rate readers.
So why is the art of reading missing in the TMap methodology?
It is hard not to notice that chapter 15 of TMap Next does not mention reading at all. If we extremely summarize the contents of this chapter, then what is said about reviewing is that it can be done by one or more persons. The chapter focuses on the review process, using the IEEE Standard for Software Reviews (IEEE Std 1028-1997) as a starting point. It tells us something about the roles, criteria and procedures for the review types inspection, walkthrough (Waldstein, 1974, though, for some reason, not referenced in TMap Next) and reviews.
It may be so that the authors of TMap Next simply assumed that reading skills are omnipresent in software testers. What I did notice over the years is that people who are designated to review documentation can fail horribly because they absolutely do not know how to read usually rather complicated documentation. So at the very least reading is a cognitive skill that should be fostered carefully. Somewhere in the past I guess I believed, in a period of ill justified innocence, that all testers possessed those skills that they naturally require. I do not believe that anymore.
Another starting point for this chapter could be that not reading skills but the processes described in the chapter, are the way to handle reviewing. Here my main struggle is that I saw review processes in action (though not anything identical to the processes described in chapter 15) and I never once felt that the processes themselves were the reason why the results were as they were. The process suggests that actors (read: humans) are aligned to accomplish a task together. Chapter 15 does not shed even the dimmest of lights on the capacities of those actors or the circumstances under which they interact. Maybe it assumes that the actors are all the same and they only need to be switched on and off at the right time, in the right room, in the right combination. But even that basic and dreadfully incorrect assumption is not mentioned in the chapter. Without these fundamental assumptions, the process is a sham.
What it boils down to is that chapter 15 had better not be used in software testing and had better not be taught to testers as an introduction to or an elucidation of reviewing. As far as I am concerned inflicting the information contained in chapter 15 on testers, should be considered harmful. The main reason is that it is impossible to verify what is described in chapter 15. The second reason is that whatever is described in chapter, is not related to the craft of software testing, because we know that software testing is about things such as reading. We should realize that testers, when reviewing, have far more pressing problems than figuring out the minimum number of warm bodies that is needed in order to officially commence the review process.