On October 31, 2013 the TestNet Autumn Event took place in Nieuwegein (the full program of the event is displayed below this article). The members of the Dutch Exploratory Workshop on Testing (DEWT) were present at this conference. The theme of the event was ‘Exploring context-driven testing – a new hype or here to stay?’. In this article we want to share our experiences of the event with the TestNet community.
As a starting point it is important to know what DEWT means by context-driven testing. For this we would like to refer to the article by Huib Schoots in the TestNet News Fall Special October 2013 with the appropriate title ‘What is context-driven testing?’.
Let’s start with the theme of the conference. A whole day of context-driven testing! Apparently the time had come for the Dutch testing community to set aside an entire day to examine this subject and to share experiences. Something we as DEWT applaud. From the considerable turnout we can conclude that there were many people who are curious and want to learn, while for a part of those present context-driven testing was still uncharted territory. This is already a success in itself.
How DEWT prepared for the conference
In preparation for the TestNet event, DEWT came together on Wednesday, October 23, along with James Bach, to talk about “What is context-driven testing?’ and ‘How to recognize a context-driven presentation?’. A report of that meeting and a list of context-driven presentation heuristics can be found on Huib’s blog.
An important starting point for a presentation on context-driven testing is presenting a story based on experiences. One of the principles in the context-driven testing community is ‘There are no best practices’. Many of the speakers indeed picked this up by sharing experiences. Without stressing how something should be done in a particular context, the speakers told how a specific problem was addressed, with parts that were good, parts that did not go well and what lessons were learned. Gitte Ottosen, Jos van Rooyen and Rik Teuben told a story about an approach that worked and an approach that did not work. Wim ten Tusscher even merged elements from his personal life into his experiential story. Tim Koomen talked about best practices as a guide. These speakers were able to put in perspective the absolute character that best practices sometimes have, and regard them as practices that may or may not work in context.
There was also a number of presentations where attendees were invited to participate in a discussion. Attending a presentation should be a matter of active participation, in which it is allowed to challenge each other. TestNet could be a good stage for such a presentation style, in which a provocative point of view may provide clarification of the subject. A fine example of this was Eibert Dijkgraaf’s presentation. Eibert conducted a ‘naked session’, therewith referring to the fact that he did not use slides. After a brief introduction, a discussion was held in which all participants could take part.
An important aspect of context-driven testing is to use knowledge from other disciplines, even if this knowledge does not appear to be related to software testing at first sight. We want to stress ’at first sight’ because we think input from other disciplines is an important part of testing. In her keynote ‘Building a Thinking Portfolio’ (PDF), Karen Johnson did not focus on testing processes, techniques or technical means. She highlighted the human side of testing and talked about how we think and analyze. Her talk was yet another experience report, in which she told how she was fascinated by the subject and how it is important for testing. Rikard Edgren (Curing Our Binary Disease, PDF) talked about a theory from the social sciences and explained with appropriate examples what this theory means to testing.
Arjen Verweij’s story was an experience report of his approach to studying (context-driven) testing. One of his strong point was that he only mentioned subjects that he had actually studied. Oftentimes it is not easy for a listener to gather what the speaker is referring to exactly. Arjen was able to pinpoint his references and his experiences with them. A study approach should not, however, turn into a recipe that will make you a context-driven tester, within a pre-determined time frame.
With this, perhaps unintentionally, an interesting question is raised: can one become a context-driven tester? Or more generally, can one become a tester of a particular school? Or are you, from the first day of your career, already a tester belonging to a particular school of thought, irrespective of your knowledge, skills or experience, because you carry with you a certain worldview? We like to leave the answer to that question, and its implications open for discussion.
Aspects of the presentations that satisfied the context-driven presentation heuristics to a lesser degree
There is still a lack of understanding of certain aspects of context-driven testing , a number of which we would like to discuss here. One such aspect is the notion of context, which is often seen from limited perspective. Examples of this are:
- context is the assignment that is given to you by the customer, or
- context is the development method that is used.
To us, context is not something that is static and pinned-down, but dynamic: everything that happens in a project and everything that affects the project is part of the context and cannot, by definition, be laid down in a list. In order to deal with this, it is important that the tester understands what context is, that he is able to observe and analyze his context and that he has the skills to adapt to an ever-changing context. Talking about frameworks within which you should always work can keep you from being receptive to the ever-changing context.
The mixing of schools is another common misconception. A number of speakers, such as Wim Ten Tuscher and Eibert Dijkgraaf, argued for an approach that consists of a mixing of schools in which you can choose, depending on the context, for one or the other school. From the context -driven perspective the verdict on this approach is very simple: it cannot be done! It is noteworthy that some speakers think it possible. The reason for the misunderstanding is that people confuse artifacts with paradigms. In every school you can use testing techniques, such as described in, for example, TMap or ISTQB. Just as well you can use, in every school, session-based test management, which was introduced by the context-driven school. However, this is not a mixing of schools, but the use of artifacts from different schools. A factory school tester who uses context-driven artifacts, is not, all of the sudden, a bit context-driven. And a context-driven tester who makes use of artifacts that have been described by the factory school is not suddenly to a certain extent a factory school tester. The answer to the question of what testing is or what it comprises, does not change by mixing artifacts from different schools.
Another conspicuous point are some unsubstantiated claims, such as ‘context-driven testing can not be applied in complex chain testing’ or ‘in a certain situation it is best to take an approach from a particular school’. When confronted with these claims our advice is to use three powerful questions from James Bach, to stimulate critical thinking.
- Huh? (Do I really understand what is being said?),
- Really? (How do I know it’s true?) and
- So? (Is this the only solution? So what?).
Unsubstantiated claims are often not based on experiences – let alone thorough research – but on assumptions that cannot be clarified or traced. If these assumptions are not met with critical examination, it is not possible to interpret them.
What is also striking, is that context-driven testing is sometimes taken lightly. It is reduced to its (context-driven) principles, which then take on a universal character. Also, context -driven is sometimes equated with common sense. It is argued that context-driven testing is merely based on semantic discussions, or that it is something that everyone does but about which a group of people likes to make a lot of fuss. Here the context-driven community and DEWT in particular learned a valuable lesson, because apparently we have not been able to elucidate the importance of the paradigm, the context-driven semantics, what common sense means to context-driven testers, and why we love to talk about testing. Clearly, these are lessons learned for us.
What can be improved by the context-driven testing community, and specifically DEWT?
A frequently heard comment was that after a number of presentations, people still did not know what context-driven testing really is, or that, as previously stated, context-driven testers are a group of people who make things needlessly complicated. DEWT sets itself the task of explaining in a better way what context-driven testing is – without simplifying it. We want to pick up this glove immediately:
- the first step is to share our experiences of the TestNet Autumn Event, and
- the second step is an article for the TestNet News explaining context-driven testing.
Additionally, we are open to questions and discussions.
Looking back at the event, we, as DEWT, can say that it was an informative day, during which there was room for different voices and visions. We hope that some testers noticed that context-driven testing suits them and that they would like to know more about it. In that case, do not hesitate to get in touch with the DEWT members. And if you want to continue the discussion on context-driven testing within your company, or want to organize a meeting for your colleagues, we are here to help.
The pictures in this article were taken by Rik Marselis.