On Tuesday January 21st we had a DEWT meeting to discuss the follow-up on our article on the TestNet Autumn Event on context-driven testing (CDT). The goal was to create an outline for an article to explain what we think CDT is. Unfortunately we did not reach a consensus on anything about the article: contents, audience, style, … There were just too many interesting possibilities.
So the discussion kept going throughout the evening among the participants: the DEWTs Ruud Cox, Jeanne Hofmans, Joris Meerts, Ray Oei, Huib Schoots, Peter Schrijver, Joep Schuurkes, Jean-Paul Varwijk and guest Jan Jaap Cannegieter. And at the end of the evening with no consensus in sight, we came to a simple solution consisting of two parts. Firstly, write an article about the discussion we had, and secondly, something you can read about at the end of this article as that’s what we came up with it.
Elements of CDT
One of the first problems we encountered is that there are just too many things to explain about context-driven testing to fit into one article. For instance, it makes little sense to explain CDT without talking about its history and comparing it to the other schools of testing and their methodologies. One important question in this regard is if software testing is more like manufacturing or more like problem solving. Is it more like building a house or more like building the Betuwe-route?
The answer to that question determines how much value you see in documents and prescribed processes, in heuristics and craftsmanship, in stories and in metrics. To explain CDT, we’d need to describe how CDT thinks about all of these – and then some.
And that’s not even everything we came up with! Our quick list also contained project management and CDT, common sense in testing, people and using the brain power of the tester. So that was a first problem we ran into: there are just so many things to explain about CDT if you want to do it well…
All those possibilities
Another question we tackled is who do we want to reach, how and why? If there is so much to explain, do we want to write an article, a series of articles or perhaps even a compendium? And do we want to write for fellow testers, for project managers, for clients that hire testers? These all require different arguments, a different story to capture their attention.
What style do we want to use? We can focus on explaining context-driven testing. We can critique the other schools and their methodologies from the perspective of CDT. Or our main goal can be to intrigue and make curious. Which is related to the question: how do we want to hook our audience? Do we want to draw the reader in because he/she is unhappy with how testing is being done now? Or because context-driven testing has interesting ideas about what testing is and how to do it? What is the selling point of context-driven testing? Is it better? Is it more fun? Both?
Perhaps we should explain by drawing an analogy with cooking, chefs and using recipes?
The pitch of Situationeel Testen
As we weren’t really getting anywhere, we asked Jan Jaap Cannegieter (CTO of SYSQA) how he describes situational testing when he is making a pitch to a client. Situational testing has been developed by SYSQA with context-driven testing as one of the inspirations. (You can read more about it in this blog post by DEWT Joris Meerts.) Jan Jaap began by explaining that for him situational testing is a means to move the Dutch testing world in the direction of context-driven testing. It moves away from the idea that one testing method is all you need. Instead, situational testing says you have to choose your testing method based on the situation. So in some situations, you will use detailed test scripts, in others testing tours and in yet other situations free-style exploratory testing. Situational testing also focusses on software as something that needs to have value for the customer. As a result testers are encouraged to use multiple sources of information and not just restrict themselves to the existing documentation.
Even with this small but significant step away from the classical testing methodologies towards more flexibility, Jan Jaap has encountered resistance. According to him, situational testing is not for every tester. Some testers see no reason to explore beyond the testing methodologies they are already comfortable with.
We learned several things from Jan Jaap’s story. Selling something is different from explaining something. Selling means convincing someone to give you an opportunity. It means knowing who you’re selling to and catering your story to that person. Most likely, you won’t get a lot of time to convince someone. So you leave a lot of things out and try to hook your audience with a selection of aspects of the thing you want to sell. For example, Jan Jaap’s pitch summarised above is to convince a client to let SYSQA use their situational testing in a project.
So that left us with the question: do we want to explain context-driven testing or do we want to sell it? And although we see value in having a pitch to sell CDT to different people, it didn’t feel like the right way to go. As context-driven testers we want to be complete, we want to give an honest and full view of CDT, not a view that is custom tailored to convince a specific someone.
Problems with explaining CDT
All the previous discussions led us to the question: why do we find it so difficult to explain context-driven testing?
One simple problem is the usage of words like ‘heuristic’. Heuristics are a big deal in context-driven testing and we’d like to explain why. However, before we can do that, we need to make sure that when we say ‘heuristic’, it’s clear what we mean. Perhaps the reader doesn’t know what a heuristic is or perhaps he defines it differently. So before we talk about heuristics and testing, we need to explain what we mean with ‘a heuristic’. However, that’s like a bait-and-switch. The reader wants to know about context-driven testing and instead of explaining CDT, we start talking about the definition of a heuristic…
Another problem is that some people we want to reach, have a to us somewhat naive idea on what testing is. They’re a client or a project manager who want testers to produce a test plan, test cases and defects, because that’s they think what the job of a tester is: to produce these artefacts. An anecdote that was told during the evening was that apparently a manager once answered the question “Why do you need a tester?” with “Because we have three developers.”
A related problem is that we may have a different idea on what testing is.
Let’s compare software development to taking a road trip. Some people seem to think of testing as cruising down the highway. We know where we’re going and we know how we’re going to get there. The job of the tester is to check if we stay on the road and to check if the driver adheres to the road signs. So the tester’s role is that of a safety measure. He keeps a watchful eye as we drive to our destination.
Context-driven testers see software development quite differently. To us it’s more like off-roading. We need to get from A to B over difficult terrain. Hopefully we can follow a trail for most of the way, but perhaps we can’t. And even if there is a trail, there may be danger ahead. If so, we want to know in time to be able to decide if it’s better to steer off the trail. We could also lose the trail. Or at some point the trail may no longer lead to the place we want to go to. That’s why you need a tester: to scout the route ahead.
Now imagine two people talking about how to prepare for a road trip with each of them having such a different image of a road trip. That discussion would lead nowhere. One would talk about the snacks and drinks to bring along; the other how many shovels, ropes and pick axes. And that’s what happens in a lot of discussions about how to do good software testing: shallow agreement on what software testing is and as a result a discussion that never really gets started. If you want to have a fruitful in-depth discussion, you’ll first have to come to a good understanding of what the other person means by ‘software testing’.
Next, there’s the question of why people adhere to factory-school methods. We don’t think it’s because they investigated all the methods and schools and then decided that the factory school is the way to go. It’s simply the case that the factory school is what most testers are taught, it’s what they’re expected to use and it’s what everyone else seems to be using. So before we can present context-driven testing as an alternative to what they are used to, we need to show that there actually is an alternative, that there are different views on what testing is. It’s like explaining to fish that there are alternatives to living in water.
Finally, there’s the problem of the difference between what context-driven testing is and how we talk about it. It’s easy to define context-driven testing by saying how most of all it’s not like the other schools of testing, by explaining how the context-driven school disagrees with what other people are thinking or doing. Especially when you have James Bach and Michael Bolton laying out the arguments for you. It’s too much like an agile evangelist telling you that the most important aspect of agile is that it’s not waterfall. That may be true, but it doesn’t tell you why agile is better or more fun or whatever. It doesn’t tell you what the strength of agile is.
Context-driven testing has been around for a while, for more than a decade. So perhaps it’s time to change the way we tell our story. To focus less on how context-driven testing disagrees with other views on testing such as the factory school. To focus more on the strengths of context-driven testing and on the stories of people that consider themselves context-driven testers.
So this is just a long introduction
At this point it was time to wrap up the evening, but we still didn’t have a plan on what to do.
Ideas ranged from writing a compendium on context-driven testing to ‘something with experience reports’. We considered writing an article describing several different contexts and explaining the strengths of context-driven testing for each of the contexts.
And at that point we returned to the question: what is context-driven testing to us?
Eight DEWTs are present in this meeting. Each of us considers him- or herself a context-driven tester. Why? Is it because we believe context-driven testing is a better way of testing? Because it’s a more fun way of testing? Both? Perhaps there is another reason?
And then we realised that maybe the best way to present and explain context-driven testing is for each of us to share our personal story and experiences. To have each of the DEWTs write a post answering one simple question “Why I am a context-driven tester”.
And sharing those stories is exactly what we will be doing the coming months.