Why I am context-driven – Ray Oei …. I think…..

This is my blog in the series “Why am I Context-driven”. In this series every member of DEWT writes a personal story on why he or she is context-driven.

That is the line the contibutions have started in this series. My first thought was to change that in: “Am I Context-driven?”. Because there are a lot of questions that need to be answered to explain that…
What /is/ Context-driven?
What does it mean to /be/ Context-driven?
What does it mean to be /part of/ the Context-driven community?
What means /the/ in the sentence ‘the Context-driven community’?
What is a community?
What is context?
Is there a reason to write context with a capital?
What is…?

And I could go on for a while.

When I started testing, by accident in a way, there were a lot of questions running through my mind. At my first serious testing challenge it could have looked a bit like this:

How /should/ I be doing /that???
– who decides what is good?
– who’s opinion should I take into account?
– who has the answers?
– what does my manager want from me?
– in what way can I be wrong?
– how am I going to prevent to be wrong?
– what is wrong about being wrong?
– where can I learn to do it right?
– why would I supply that?
– why wouldn’t I supply that?
– what is the problem (here)?
– why would I bother?
– why would anyone bother?
– what is right anyway?
– what is it what I want to say?
– what is it what I mean?
– what is it what I want?
– I don’t want to fail…..
– …….
– what /is/ failing anyway?
– who would care?
– …
– wait, …. do I see a pattern here??????

I have written before what led me to this point: here. And why I started the discussion what led to DEWT (here). So I am not going to re-itereate that.
But that still doesn’t answer any of those questions. Some of the critics (here and here both in Dutch) find most of what we (“what is /we/?” “Who are /we/?”) vague or obvious or not interesting. That is ok. Despite of what some of us (“who is /us/?”) say or seem to imply: there is no ‘best practice’ of being ‘context-driven’. Some of the discussions have culminated in a fight over ‘wrong’ and ‘right’. In essence – I think – it is about ‘following the rules’ versus ‘investigate and see what can be learned’. The latter is something that appeals to me the most, being from a scientific background. So many times questions like: “is it useful?” “What information does that provide?” “To what purpose should I be doing this?” have gone through my mind at some time.

So why write this in the first place? One of the important foundations for this community is to share. Share knowledge. Share experiences. Share thoughts. And discuss them. Learn from them. Change your ideas when information compels you to. Be critical. Accepts that others might be (very) critical. Try to learn from that too. About the craft. About yourself. About the world. And use that in your work.

This community helps me to do that. With people who expect critical thinking. Who don’t provide the “right” answer. Who provoke (which I wouldn’t do that easily). Who get me thinking. Do I agree with all of what is said? Does that really matter? To whom? And sometimes I am doubting whether I am ‘context-driven’. But do I have to be?

I can see why some people have problems with ‘us’. And I think that is something we should learn from. We can be overbearing at times. Know it all’s. Seem to be a closed group. And sometimes it is difficult to separate the message from the person. I should not speak for others. I myself don’t really like being too direct. Or at the front of the stage. It’s uncomfortable, to me. So am I still context-driven then? Are people hesitant to join the community because it feels too much that you can only belong if you are like ‘that’? (“What is /that/ exactly?”). We also need to be self critical. We can (and should!) learn and sharpen our ideas when we are confronted with other opinions. Are we vague? We don’t intend to (no really: we don’t!!) but we are perceived that way sometimes. So we should work on that. Well… no, I should work on that.

I try to learn. I share what I have learned by teaching others for free (here and here). I coach testers for free (here). They learn from me, I hope. And I learn from them. What I do right ánd what I do not so right.
So what does it mean to be context-driven? Can you only be context-driven when you call yourself context-driven? I know a few very good testers who explicitly do not consider themselves to be ‘context-driven’. Are those people less as testers? I don’t think so! /I/ consider them to be a part of the ‘context-driven’ philosophy. They are critical thinkers to start with…

So it might be /that/ that appeals to me the most: ask questions. Investigate. Discuss. Try to find answers with the information you have. Re-investigate. Lean. Learn. Learn…

Posted in Context-driven testing, Why am I context-driven | Tagged , | 2 Comments

Why I am context-driven – Philip Hoeben

This is my blog in the series “Why am I Context-driven”. In this series every member of DEWT writes a personal story on why he or she is context-driven.

A natural phenomenon that appeals to me.

How obvious it may seem, first of all I want to point out that testing is part of a society where it functions. It’s not some form of alienated stand-alone discipline. One can notice a lot of similarities happening in the test world with things happening outside of the test world. It is not surprising that there is such a thing as a context-driven community, because it is a manifestation of what happens in society.

For instance, have a look at this article: http://www.theguardian.com/business/2014/may/06/oecd-pisa-tests-education-joy-of-learning. It’s about the PISA (Programme for International Student Assessment) tests, the introduction of a standardised curriculum, and the criticism on it. I don’t think I have to elaborate that these debates are also going on within the test world. Just think of the vivid discussions about the value of certifications.

Or have a look at: http://hackeducation.com/.  Audrey Watters who writes a lot about ed tech on her blog, elaborates inspiringly about the effects of technology in education. When reading her blogs it is not hard to see parallels with the test world. She points out on her blog “Robots and education labor” that education is about gaining skills and not job training and career readiness.

My own take on education and the ‘joy of learning’ is that I want to dive into areas of my own interest, developing my own learning path, shaping my ideas in a way I choose. I don’t have any need to give an answer somebody expects me to give, which can be falsified anyway, as has been demonstrated both within as outside of the test world. It’s much more important to show skills in reasoning behind the answer. The context-driven school emphasizes on showing skills and being able to explain your reasoning.

Solving actual test problems.

I value one model over another when that model gives more insight in a topic. When I started to read about testing, at some point, I realise which model I value most.

Why the context-driven model?

When you want to learn about testing you have to study what testing is about and what happens during testing, because testing only happens when somebody is testing. We have to study this ‘act of testing’. Looking at testing with the stance of a detached observer can give illusory pretensions of what testing is, since only artifacts are the objects of study (test cases, test techniques, test plans, test processes etc.).

You interact with the software by using and manipulating the software as a human being. Your whole history (e.g. knowledge, experiences) is brought to your testing.

Wow, isn’t that great!

The context-driven community focusses on the human aspects of software testing and studies phenomena like interpretation, understanding and observing. By addressing it and showing the importance, you can actually study it.

Improvements in testing

As said before, I value the context-driven model most. I am convinced that in the future problems will rise that cannot be addressed by the context-driven model. As long as I don’t know or have any model for that or as long as a new model is not relevant, I stick to context-driven testing.

Improvements in testing are not related to security, big data, mobile, more or less or less is more automation, TMMi level x or other tech trends. Improvements in testing are about refining or redefining a model that challenge ideas that became common, and fits better in the ever changing society. Context-driven testing, and for example also agile, didn’t come to life because of tech trends, but through a different view on a subject. I don’t think tech trends will help us refine or redefine a model for testing. I believe context-driven testing is currently the only model which redefines and refines software testing. And for the good!

That people thing and the community

Of course there is a community of people saying meaningful things. Those people that are willing to help without presenting a one size fits all solution. I enjoy being with them and like to read their stuff.

It’s a truly inspiring community!

Posted in Context-driven testing, Why am I context-driven | Leave a comment

Why I am context-driven – Simon ‘Peter’ Schrijver

This is my blog in the series “Why am I Context-driven”. In this series every member of DEWT writes a personal story on why he or she is context-driven.

Officially, I started with testing in 1997. Looking back to my career, I can say I started with testing when I started my professional career back in the eighties. All the jobs I did before 1997, I had to deal with some serious test work. I only did not realize I was doing that. That period perhaps made me the tester I’m today. I always look for problem situations, look for situations where things could go wrong, how the system did behave at that moment.
I learned testing from 1997 to 2003 by just using common sense. Next to looking where things are okay, I looked for problem situations. In that period I was working for a large telecom network systems provider and we were testing network systems and network solutions. We always had to introduce errors in the environment and look how the solution reacted, responded. We found some serious problems in these days.

I kept on going with this mindset for all my test assignments with other companies. The time I started as an independent tester, I needed to take care of my own education. I decided to go to conferences. There I came in contact with people like Michael Bolton, James and Jon Bach at STAR East in 2007. What they were talking about got me interested. Perhaps this was the push in the ‘right’ direction.

I got in touch with CDT when I registered for the CAST 2011 conference. This conference was organized James and Jon Bach with the theme “Context Driven Testing” and it was an epic event. Some of the presentations given there were great. Markus Gärtner gave a presentation about self learning that really triggered me to do more with my test career. I was already working on improving my testers skills, but that were certificate focused classes instead of skill training. From the moment I left the conference, I said to my self: “I want to be a context driven tester. What do I need to do?”

I joined Twitter to follow the stories of other testers. I started reading blogs and books. I tried to find means to use CDT at my work. That was not easy sometimes as I worked in an environment with classical trained people focused on processes with pre–scripted test cases and pass/fail graphs. Luckily, the projects I did, I was able focus on what was really important in the context of my stakeholders.

Let me give an example of a project I did some time ago. It was a replacement project. The development team wanted to built software using SCRUM. There was only one strange thing they wanted. They wanted to deliver the software to my acceptance test team in three big deliveries. My reaction to this was as follows:
“If you want to do Scrum as a development approach and you deliver the SW in three big deliveries to me, each containing 8 features (i.e. data streams for billing and fraud systems), why are you doing Scrum? I want for every feature a separate, dedicated delivery plan.”
The next question I got from the project team was, “what is your test plan, how many test cases do you have?” After some discussion I told them, “I have one test case. I want to see that the data stream on the test environment produces the same output compared to the production environment.”
By having this attitude, I convinced my stakeholders I was serving them well.

As a tester I look for solutions which serve my stakeholders. I learn new practices which are useful at that moment risking I will never use them again. That is a risk I’m willing to take.
Further, I share my experiences with people I meet at conferences and/or via my blog.

I like to end with a few lines some of you probably recognize (if you are a Trekkie).

Space, the final frontier
These are the sessions of the context driven tester
Its unchartered missions
To explore undiscovered areas
To seek out for bugs and new information
To boldly test where no one has tested before

I hope you will join me on my journey.

Posted in Context-driven testing, Why am I context-driven | Leave a comment

Why I am context-driven – Joris Meerts

We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.

T.S Eliot – Little Gidding

Any theory that can account for all of the facts is wrong, because some of the facts are always wrong.

Francis Crick

I was introduced to the context-driven community when Shmuel Gershon invited me to a soiree in Copenhagen, during the EuroSTAR conference in 2010. This invitation was based on my work on the history of software testing, the importance and purpose of which I was largely unaware of at that time.

I do not claim to be a context-driven tester. However, I am drawn to view of testing that is entertained by the context-driven community. Currently, there is no other school of thought in software testing that provides the same kind of energy and excitement, willingness to explore and at the same time the willingness to confront ourselves with the conditions and predicaments of our craft.

Software testing is an act of investigation. This is the constant factor in software testing. The reason why we conduct this investigation, the means by which we conduct this investigation and the degree of success of this investigation differs from project to project. From this it follows that there is no single technique, no single approach, that can universally be applied. It also follows that an attempt to apply universally a single approach is bound to diverge constantly from the purpose of the software development effort. Regretfully this means that there are techniques or tools that I mastered and like, but may never apply again. It also means that there are other techniques and tools that I must learn in order to be able to do my next job.

The constant factor from project to project, is to develop an understanding of how to conduct an investigation and how to act in a way that is relevant to the project. This is by far the most difficult part of software testing. Software testers have to develop a sense of awareness and ways of reasoning to assess themselves, their team, the project, the company, the customers and, perhaps, taken to the very extreme, the society in which the software plays a part. Context-driven testing studies the ideas and tools that aid reasoning, observation and experimentation. As far as I am aware, the school of context-driven testing is the only school in software testing that actively pursues the application of these academic areas in software testing.

On the other hand, context-driven testing may very well be a product of its age, just as methodology-driven testing is a product of a firm belief that industrialization and standardization will eventually lead to success in software development. The age in which, in my opinion, the roots of context-driven can be found, spans the 1940s to the 1970s. As far as I have been able to assess, in this age the seeds of developments in rather young scientific disciplines were sown; disciplines that influence software testing today. Parts of systems theory were developed by Kenneth Boulding in the 1950s, cybernetics by Norbert Wiener in the 1940s, information theory by Claude Shannon in the 1940s, game theory by John von Neumann in the 1940s, complexity theory by William Ross Ashby in the 1950s, system dynamics by Jay Forrester in the 1960s and social systems theory by Talcott Parsons in the 1950s. It is my personal feeling that, in order to meaningfully contribute to software development, software testing should study the aforementioned areas first, and only after that start fussing about condensing the outcomes of experiments (‘test cases’) into simplistic binary representations.

There are many other developments in this age, particularly in cognition, language (Chomsky comes to mind), art and anthropology that may prove useful in software testing.

In the Netherlands, whenever there is a discussion of ‘European testing’ versus ‘American testing’, contrasting methodology-driven testing and context-driven testing, I like to jokingly elucidate that actually context-driven testing is fundamentally an Austrian invention. As ‘evidence’ to support my case I use the books Against Method by Paul Feyerabend (1975), General System Theory by Ludwig von Bertalanffy (1968) and the Logik der Forschung by Karl Popper (1935), books that contain ideas that influenced context-driven testing. As you can guess all of these authors were born in Austria.

I should probably mention that I have not read Popper’s book. Naturally, the book is in my library, because I feel that the chapters on falsifiability, empiricism, testability and probability will some day serve me well.

Much of this comes down to an understanding of what software actually is. I feel that software testing as a craft should develop such an understanding, not on the basis of wrought iron processes or convoluted models, but out of a naturalistic investigation, by looking at software as it is. This is the capability that we lost in the past couple of decades. I hope context-driven testing can regain it.

Posted in Context-driven testing, Why am I context-driven | Leave a comment

Why I am context-driven – Jeanne Hofmans

This is the third blog in the series “Why am I Context-driven”. In this series every member of DEWT writes a personal story on why he or she is context-driven.

After graduating from my study in Software Technology I started to work as a tester. The reason? Although I liked to program, I enjoyed designing and testing more. Agile development was not yet a common practice and I knew that solely programming at my desk would not make me happy. I simply loved (and love) talking to other people: discussing strategies and questioning features. As a tester I could do just that. In my opinion testers play a central role in projects. As a tester I would need to communicate with designers, with the business, with developers, with project management, with QA, etcetera. Also I liked the act of testing. When other students had finished their work, I could not resist to ask: may I? After which I tried to ‘break’ their program. So I decided to make testing my career.

Choose a job you love and you will never have to work a day in your life. – Confucius

On the job, working for several clients, I really learned a lot about testing. I saw how some things worked and some things did not. Sometimes paper and processes seemed to prevail over people and skills. That bothered me. For example when a risk analysis had been performed but was not actually used. Why spend time on the risk analysis in that case at all? I experimented with performing risk analyses and the way to implement a risk based strategy. One day I was discussing this with my colleague Ruud Cox. After that in-depth discussion he asked me if I would like to attend DEWT once. So I did.
At DEWT I found a group of people passionate about testing. Discussing all kind of aspects related to testing appealed to me. So I joined DEWT. At DEWT I learned about context driven testing (CDT). I really liked some of the principles of CDT, like the focus on people, skill and also playing with the product.

  • People: Jerry Weinberg’s definition of quality: “Quality is value to some person” clearly indicates the importance of people in quality and testing. Products are made by people and are to be used by people. Thus understanding people and interacting with people, asking questions, makes us better testers.
  • Skill: Several skills are of importance to testers. From general communication skills to being able to use several heuristics and test design techniques (as seen suited!)
  • Play: The ability to play makes us, people, different from a computer. Playing with the product helps us understand the product and find bugs that we otherwise would not find. Playing is also about fun.

Pleasure in the job puts perfection in the work. – Aristotle

Basically all of these are about continuous learning. Learning by reference (discussing with peers and people having other viewpoints and backgrounds), learning by exercising, and learning by reading. Once you are more aware of possibilities and pitfalls, you can make better choices for your testing. As every project is unique, the solution to testing is also unique. And that is why I am a context driven tester.

Posted in Context-driven testing, Why am I context-driven | Leave a comment

Why I am context-driven – Jean-Paul Varwijk

This is the second blog in the series “Why am I Context-driven”. In this series every member of DEWT writes a personal story on why he or she is context-driven.

“There are many possible answers to questions you might have and the best answer is not immediately or easily available. It is a situation that is complex enough to require the assignment of a specialist…(engineer, developer, architect, software tester, etc.)…to assess the question and organize recourses as necessary to find an answer”.         Software exploration

Several years before the turn of the century my role as a user required me to take a closer look at evaluating software that was to be “Euro giro” ready. Looking back now it was then that the foundation for my software testing mind-set was initially formed. At first I was being a typical unconsciously incompetent user and thought testing is actually nothing more than doing what I always do but only on new software. Nicely following Dunning-Kruger’s initial success made me feel I was on the right track. After a while the IT project manager nicely added some perspective to this by having me execute some ‘real’ test cases which all miserably failed on some features that I before had found acceptable. Now being consciously incompetent I took some time to study a few software testing practices and got some advice from in-house software testers. Soon I was able to deliver nicely scripted test cases, report on coverage and test results against acceptance criteria. For a user I was doing nicely and since most other users disliked the extra activity next to their regular work I was happy being the departments’ acceptance tester on and off for a number years.

Changing Jobs

In 2004 the department I was working for faced a major reorganisation and I was told that I was among the 40% of people eventually either re-assigned or laid off at some point in the next 24 months. I chose not to wait for someone to tell me how my future would be but contacted HR myself and requested to participate in a job reassessment program. Around a third into the program I realised that the software testing part of my work was probably what I had liked the most in my current work so why not do that as an actual career. Luckily for me right then and there an opening to become a software tester at our IT department was available and three interviews and an assessment later there I was ready to step into my new career.

Learning traditional testing

I started with a four-week master class which included getting TMap and (eventually) ISTQB certified, learning testing techniques, test management and real testing of Tax Office software. Being primed I felt that now being consciously competent I was ready to take on any software that would come my way. Boy was I in for a number of quick disappointments. First applying test techniques to actual documents and software was much more difficult than applying it to any training material I encountered until then. Second I was spending a large amount of time on writing about testing and not actually doing any testing. Doing a Product Risk Analysis still seemed pretty useful but making a 50+ pages Master Test Plan, Functional Acceptance Test Detailed Test Plan, Integration Testing Detailed Test Plan and User Acceptance Test Detailed Test Plan, Test Stage Reports, Final Test Report and a Release Advice absorbed at least as much time as testing the product. I could, and still do, to some extend understand that you want to assure that no steps are missed or lightly discarded but every document turned out to become a copy paste ritualistic window dressing exercise without much added value. I still loved the software testing and started looking for ways to cut corners on documentation and still tell what the stakeholders wanted to know. I moved towards more agile projects and enjoyed the more active approach of working together with developers, business analysts and users and appreciated the shorter feedback loop. Even so I was still searching to find a better way to test software.

Becoming a software tester

LessonsLearnedOctober 2009 marked a turning point with Immune IT organizing a Cem Kaner Week. I participated at the last presentation in which Cem spoke about Exploratory Test Automation and afterwards gave away signed copies of Lessons Learned in Software Testing – A Context-Driven Approach. This got me thinking. Software testing is not a separate activity in software development. Software testing is an part of software development and by bringing skilled testers into the mix it can help to produce better software on the one side and deliver meaningful information about the quality of the software on the other side. Yes you may identify different roles, skill sets and specialisms but only the complete set of activities joined, executed concurrently and in awareness of socio cultural and behavioural dynamics delivers a potentially successful software solution to a business problem. To me this means that I do not only need to know how to apply testing techniques, to understand software but I also need to understand how personal and business needs and wants translate into explicit and implicit requirements. How participants in software development use and translate these requirements. What the choice of development approach means. How software interacts with people. How they make decisions. How to convey information, etc., etc… In my opinion the only holistic approach to software testing that takes this whole diversity of attitudes and environments into account is the approach as described in by Context-Driven Testing and the fast amount of articles and presentations of its followers.

More than a practice

Context-Driven Testing is more than just another practice. To some part it is a paradigm, a school of testing. To a larger part it is an approach to software testing, of how to do your work and finally, at least to me, to the largest part it is a community of testing practitioners sharing experiences, ideas, problems, and solutions through blogs, (peer) conferences, workshops and by challenging each other on their views.Passion

Image | Posted on by | Tagged | Leave a comment

Why I am context-driven – Huib Schoots

This is the first blog in the series “Why am I Context-driven”. In this serie every member of DEWT writes a personal story on why he or she is context-driven.

“Because testing (and any engineering activity) is a solution to a very difficult problem, it must be tailored to the context of the project, and therefore testing is a human activity that requires a great deal of skill to do well. That’s why we must study it seriously. We must practice our craft. Context-driven testers strive to become the Jedi knights of testing”. James Bach

legojediMy first steps on the testing path were in a happy-go-lucky way. I didn’t know much about testing and common sense helped me to do my work. Next I learned about structured testing methodologies. As a tester and later as a test manager, but also as a teacher, I observed many people test and learn to test. I struggled with the testing I did and saw others do. I started doubting the testing I was taught and teaching myself: it just wasn’t right… There is much more to testing that wasn’t told in the common testing classes. And why weren’t we testing actual software in these classes?

I have been trainer of factory based methods for many years. After telling the same story over 10 times, you get the chance to really think about it. I remember one day when I drove home, I thought about what we did in class. A three-day class where we never actually tested anything. We only did exercises on paper like applying a test technique on an over-simplified example written out on paper.

I watched many people doing testing. But what were they doing? They were applying standards and often a happy-go-lucky approach was used. Not using test techniques, because they are too difficult to apply. And why were people struggling in their testing projects? Testing has a bad name in many organisations and testing is often undervalued. Testing is often devalued as “pushing buttons” and “everybody can test”. Many people use testing to get into IT. And looking at our industry, it can be done easily. Nine out of ten job interviewers do not know how to recognize the difference between fake and real testers. That is why so many fake testers are still running around without being detected. Do you remember your last job interview? Did you have to do actual testing? Or were they only asking questions? In the movie business even actors like Brad Pitt have to audition for a role. The director wants to see them do the character they have to play in the movie…

On foreign blogs I started reading about agile and context-driven testing. I found there is much more to testing then I could guess. I learned new, exciting and interesting aspects, which I hadn’t considered before like critical and systems thinking, heuristics, the art of asking good questions, safety language, models, credibility, diversity, storytelling, social sciences and exploratory testing. Some of these ideas where considered controversial by my colleagues.

Polder models in testing?

polderTesting standards are created by many people around the world. And if many people work on a standard or methodology it becomes an average, something everybody has to agree with. Everyone working on it, wants their thing in there. The standard has to fit every context easily. In the Netherlands we call that a polder model. Consensus rules and nobody gets what he really wants. These standards are made for an average situation or project. But is there such a thing as an average project? I think not! Projects are unpredictable, often complex and depend on humans. Skills are absolutely required to deal with difficult, complex and ambiguous problems.

Within context-driven testing people are the most important part of projects. Skills matter and they are trained by context-driven testers. Context-driven testing promotes skepticism, thinking and advocates problem solving instead of merely applying standards.

Factory approach or research and development?

Software development is often compared with product manufacturing in which standards and best practices are successfully used. But to me software development is more like research and development. No mass production but creating a product that solves a very specific problem. Testing can be also be seen as a form of research: investigating the system and finding information about it. In research critical thinking is important. Collecting, analyzing and interpreting information requires critical thinking skills. Critical thinking to me is about thinking (critically) about your own personal thinking. Framing your own assumptions and using this to try to remove bias and hopefully clarifying your thoughts with reasoning.

Why is it important to me…

PassionI am a passionate person and I always want to be the best or do the best work possible. I refuse to do bad work. I have a high standard and ethics are important to me. Context-driven testing gives me that. The community is awesome. We share and learn together, we challenge each other and we practice continuous learning.

When I look at people working in IT, many are doing a 9 to 5 job. Not learning and applying stuff others tell them to do. They take bullshit from managers and project managers who do not know anything about testing, but still telling them what to do. Like writing or counting test cases, creating worthless metrics and writing bulky documents. They are happy to do what they believe are best practices. That is not for me. I need challenge and I want to do the very best possible in any situation. Considering that projects are always different, testers need to have many skills. I wrote a series of blog posts in 2011 on how to become an expert tester and what makes a good tester here, here and here.

The seven basic principles of the Context-Driven School tell us that people, working together, are the most important part of any project’s context. That good software testing is a challenging intellectual process and only through judgement and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

thinking fast and slowContext-driven testing gave me a lot of new challenges in testing. It brought me a lot of fun and gave me new insights to improve my understanding of the craft like social science. Studying social science gave me better understanding about people and biases. Reading books like thinking fast and slow helped me understand how the human brain works and made me aware how people make judgements and come to conclusions. I learned about qualitative and quantitative research. This gave me more knowledge about dealing with ambiguity, situational specific results and partial answers. Context-driven testing also brought me ways to cope with the difficulties I struggled with years ago. Exploratory testing made testing more fun and heuristics help me to guide and structure my testing to be able to do fast and efficient testing. I became aware that science is important. It gave me critical thinking and that helps proving the theories we have about the product. Try to prove yourself wrong instead of proving yourself right. Context-driven testing appeals to me because it puts the tester’s mind at the centre of testing instead of documents, process and standards.

Context-driven testing made my testing more personal. Not doing stuff everybody does, but it encouraged me to develop my own style. It is a mind set, a paradigm and a culture. It is not only about what I do, it is more about who I am!

Posted in Context-driven testing, Why am I context-driven | 5 Comments

Report on the discussion about the contents of an article titled ‘What is CDT?’

On Tuesday January 21st we had a DEWT meeting to discuss the follow-up on our article on the TestNet Autumn Event on context-driven testing (CDT). The goal was to create an outline for an article to explain what we think CDT is. Unfortunately we did not reach a consensus on anything about the article: contents, audience, style, … There were just too many interesting possibilities.

So the discussion kept going throughout the evening among the participants: the DEWTs Ruud Cox, Jeanne Hofmans, Joris Meerts, Ray Oei, Huib Schoots, Peter Schrijver, Joep Schuurkes, Jean-Paul Varwijk and guest Jan Jaap Cannegieter. And at the end of the evening with no consensus in sight, we came to a simple solution consisting of two parts. Firstly, write an article about the discussion we had, and secondly, something you can read about at the end of this article as that’s what we came up with it.

20140121 DEWT Meetup - Artikel Wat is CDT- - 2

Elements of CDT
One of the first problems we encountered is that there are just too many things to explain about context-driven testing to fit into one article. For instance, it makes little sense to explain CDT without talking about its history and comparing it to the other schools of testing and their methodologies. One important question in this regard is if software testing is more like manufacturing or more like problem solving. Is it more like building a house or more like building the Betuwe-route?
The answer to that question determines how much value you see in documents and prescribed processes, in heuristics and craftsmanship, in stories and in metrics. To explain CDT, we’d need to describe how CDT thinks about all of these – and then some.

And that’s not even everything we came up with! Our quick list also contained project management and CDT, common sense in testing, people and using the brain power of the tester. So that was a first problem we ran into: there are just so many things to explain about CDT if you want to do it well…

All those possibilities
Another question we tackled is who do we want to reach, how and why? If there is so much to explain, do we want to write an article, a series of articles or perhaps even a compendium? And do we want to write for fellow testers, for project managers, for clients that hire testers? These all require different arguments, a different story to capture their attention.

What style do we want to use? We can focus on explaining context-driven testing. We can critique the other schools and their methodologies from the perspective of CDT. Or our main goal can be to intrigue and make curious. Which is related to the question: how do we want to hook our audience? Do we want to draw the reader in because he/she is unhappy with how testing is being done now? Or because context-driven testing has interesting ideas about what testing is and how to do it? What is the selling point of context-driven testing? Is it better? Is it more fun? Both?

Perhaps we should explain by drawing an analogy with cooking, chefs and using recipes?

The pitch of Situationeel Testen
As we weren’t really getting anywhere, we asked Jan Jaap Cannegieter (CTO of SYSQA) how he describes situational testing when he is making a pitch to a client. Situational testing has been developed by SYSQA with context-driven testing as one of the inspirations. (You can read more about it in this blog post by DEWT Joris Meerts.) Jan Jaap began by explaining that for him situational testing is a means to move the Dutch testing world in the direction of context-driven testing. It moves away from the idea that one testing method is all you need. Instead, situational testing says you have to choose your testing method based on the situation. So in some situations, you will use detailed test scripts, in others testing tours and in yet other situations free-style exploratory testing. Situational testing also focusses on software as something that needs to have value for the customer. As a result testers are encouraged to use multiple sources of information and not just restrict themselves to the existing documentation.

Even with this small but significant step away from the classical testing methodologies towards more flexibility, Jan Jaap has encountered resistance. According to him, situational testing is not for every tester. Some testers see no reason to explore beyond the testing methodologies they are already comfortable with.

We learned several things from Jan Jaap’s story. Selling something is different from explaining something. Selling means convincing someone to give you an opportunity. It means knowing who you’re selling to and catering your story to that person. Most likely, you won’t get a lot of time to convince someone. So you leave a lot of things out and try to hook your audience with a selection of aspects of the thing you want to sell. For example, Jan Jaap’s pitch summarised above is to convince a client to let SYSQA use their situational testing in a project.

So that left us with the question: do we want to explain context-driven testing or do we want to sell it? And although we see value in having a pitch to sell CDT to different people, it didn’t feel like the right way to go. As context-driven testers we want to be complete, we want to give an honest and full view of CDT, not a view that is custom tailored to convince a specific someone.

20140121 DEWT Meetup - Artikel Wat is CDT- - 1

Problems with explaining CDT
All the previous discussions led us to the question: why do we find it so difficult to explain context-driven testing?

One simple problem is the usage of words like ‘heuristic’. Heuristics are a big deal in context-driven testing and we’d like to explain why. However, before we can do that, we need to make sure that when we say ‘heuristic’, it’s clear what we mean. Perhaps the reader doesn’t know what a heuristic is or perhaps he defines it differently. So before we talk about heuristics and testing, we need to explain what we mean with ‘a heuristic’. However, that’s like a bait-and-switch. The reader wants to know about context-driven testing and instead of explaining CDT, we start talking about the definition of  a heuristic…

Another problem is that some people we want to reach, have a to us somewhat naive idea on what testing is. They’re a client or a project manager who want testers to produce a test plan, test cases and defects, because that’s they think what the job of a tester is: to produce these artefacts. An anecdote that was told during the evening was that apparently a manager once answered the question “Why do you need a tester?” with “Because we have three developers.”

A related problem is that we may have a different idea on what testing is.
Let’s compare software development to taking a road trip. Some people seem to think of testing as cruising down the highway. We know where we’re going and we know how we’re going to get there. The job of the tester is to check if we stay on the road and to check if the driver adheres to the road signs. So the tester’s role is that of a safety measure. He keeps a watchful eye as we drive to our destination.
Context-driven testers see software development quite differently. To us it’s more like off-roading. We need to get from A to B over difficult terrain. Hopefully we can follow a trail for most of the way, but perhaps we can’t. And even if there is a trail, there may be danger ahead. If so, we want to know in time to be able to decide if it’s better to steer off the trail. We could also lose the trail. Or at some point the trail may no longer lead to the place we want to go to. That’s why you need a tester: to scout the route ahead.
Now imagine two people talking about how to prepare for a road trip with each of them having such a different image of a road trip. That discussion would lead nowhere. One would talk about the snacks and drinks to bring along; the other how many shovels, ropes and pick axes. And that’s what happens in a lot of discussions about how to do good software testing: shallow agreement on what software testing is and as a result a discussion that never really gets started. If you want to have a fruitful in-depth discussion, you’ll first have to come to a good understanding of what the other person means by ‘software testing’.

Next, there’s the question of why people adhere to factory-school methods. We don’t think it’s because they investigated all the methods and schools and then decided that the factory school is the way to go. It’s simply the case that the factory school is what most testers are taught, it’s what they’re expected to use and it’s what everyone else seems to be using. So before we can present context-driven testing as an alternative to what they are used to, we need to show that there actually is an alternative, that there are different views on what testing is. It’s like explaining to fish that there are alternatives to living in water.

Finally, there’s the problem of the difference between what context-driven testing is and how we talk about it. It’s easy to define context-driven testing by saying how most of all it’s not like the other schools of testing, by explaining how the context-driven school disagrees with what other people are thinking or doing. Especially when you have James Bach and Michael Bolton laying out the arguments for you. It’s too much like an agile evangelist telling you that the most important aspect of agile is that it’s not waterfall. That may be true, but it doesn’t tell you why agile is better or more fun or whatever. It doesn’t tell you what the strength of agile is.
Context-driven testing has been around for a while, for more than a decade. So perhaps it’s time to change the way we tell our story. To focus less on how context-driven testing disagrees with other views on testing such as the factory school. To focus more on the strengths of context-driven testing and on the stories of people that consider themselves context-driven testers.

So this is just a long introduction

At this point it was time to wrap up the evening, but we still didn’t have a plan on what to do.
Ideas ranged from writing a compendium on context-driven testing to ‘something with experience reports’. We considered writing an article describing several different contexts and explaining the strengths of context-driven testing for each of the contexts.

And at that point we returned to the question: what is context-driven testing to us?
Eight DEWTs are present in this meeting. Each of us considers him- or herself a context-driven tester. Why? Is it because we believe context-driven testing is a better way of testing? Because it’s a more fun way of testing? Both? Perhaps there is another reason?
And then we realised that maybe the best way to present and explain context-driven testing is for each of us to share our personal story and experiences. To have each of the DEWTs write a post answering one simple question “Why I am a context-driven tester”.
And sharing those stories is exactly what we will be doing the coming months.

Posted in Context-driven testing, Meeting reports | Leave a comment

DEWT4 Afterparty

DEWT4-participants-v4 From left to right: Jeanne Hofmans, Rob van Steenbergen, Jurian van de Laar, Peter Simon Schrijver, Jean-Paul Varwijk, Bernd Beersma, Huib Schoots, Arjen Verweij, Zeger van Hese, Joris Meerts, Markus Gärtner, Bart Broekman, Angela van Son, Pascal Dufour, Ard Kramer, Jeroen Mengerink, Kristoffer Nordström‏, Philip Hoeben, Daniël Wiersma, Joep Schuurkes, Duncan Nisbet, Eddy Bruin, Wim Heemskerk, Ruud Cox, Richard Scholtes, Ray Oei.

Below are a number of links to DEWT4 reports produced by participants. If you want to get an impression about what happened during this peer conference, then you’ve got to read them all.

On behalf of all the participants I’d like to thank the AST for the grant which helped in making this conference a success.

Posted in DEWT4, Peer conference | Tagged | 3 Comments

DEWT Experiences from the TestNet Autumn Event

This article is a translation of two-part series of articles that was published in the TestNet News. The original articles can be found here and here. On behalf of DEWT, the author is Philip Hoeben.

TestNetOn October 31, 2013 the TestNet Autumn Event took place in Nieuwegein (the full program of the event is displayed below this article). The members of the Dutch Exploratory Workshop on Testing (DEWT) were present at this conference. The theme of the event was ‘Exploring context-driven testing – a new hype or here to stay?’. In this article we want to share our experiences of the event with the TestNet community.

As a starting point it is important to know what DEWT means by context-driven testing. For this we would like to refer to the article by Huib Schoots in the TestNet News Fall Special October 2013 with the appropriate title ‘What is context-driven testing?’.

Ray Oei and Peter Schrijver

Ray Oei and Peter Schrijver conducted a workshop at the TestNet Autumn Event 2013.

Let’s start with the theme of the conference. A whole day of context-driven testing! Apparently the time had come for the Dutch testing community to set aside an entire day to examine this subject and to share experiences. Something we as DEWT applaud. From the considerable turnout we can conclude that there were many people who are curious and want to learn, while for a part of those present context-driven testing was still uncharted territory. This is already a success in itself.

How DEWT prepared for the conference

In preparation for the TestNet event, DEWT came together on Wednesday, October 23, along with James Bach, to talk about “What is context-driven testing?’ and ‘How to recognize a context-driven presentation?’. A report of that meeting and a list of context-driven presentation heuristics can be found on Huib’s blog.

The presentations

An important starting point for a presentation on context-driven testing is presenting a story based on experiences. One of the principles in the context-driven testing community is ‘There are no best practices’. Many of the speakers indeed picked this up by sharing experiences. Without stressing how something should be done in a particular context, the speakers told how a specific problem was addressed, with parts that were good, parts that did not go well and what lessons were learned. Gitte Ottosen, Jos van Rooyen and Rik Teuben told a story about an approach that worked and an approach that did not work. Wim ten Tusscher even merged elements from his personal life into his experiential story. Tim Koomen talked about best practices as a guide. These speakers were able to put in perspective the absolute character that best practices sometimes have, and regard them as practices that may or may not work in context.

There was also a number of presentations where attendees were invited to participate in a discussion. Attending a presentation should be a matter of active participation, in which it is allowed to challenge each other. TestNet could be a good stage for such a presentation style, in which a provocative point of view may provide clarification of the subject. A fine example of this was Eibert Dijkgraaf’s presentation. Eibert conducted a ‘naked session’, therewith referring to the fact that he did not use slides. After a brief introduction, a discussion was held in which all participants could take part.

Jeanne Hofmans

Jeanne Hofmans presented at the TestNet Autumn Event 2013 on how to test a tunnel.

An important aspect of context-driven testing is to use knowledge from other disciplines, even if this knowledge does not appear to be related to software testing at first sight. We want to stress ’at first sight’ because we think input from other disciplines is an important part of testing. In her keynote ‘Building a Thinking Portfolio’ (PDF), Karen Johnson did not focus on testing processes, techniques or technical means. She highlighted the human side of testing and talked about how we think and analyze. Her talk was yet another experience report, in which she told how she was fascinated by the subject and how it is important for testing. Rikard Edgren (Curing Our Binary Disease, PDF) talked about a theory from the social sciences and explained with appropriate examples what this theory means to testing.

Arjen Verweij’s story was an experience report of his approach to studying (context-driven) testing. One of his strong point was that he only mentioned subjects that he had actually studied. Oftentimes it is not easy for a listener to gather what the speaker is referring to exactly. Arjen was able to pinpoint his references and his experiences with them. A study approach should not, however, turn into a recipe that will make you a context-driven tester, within a pre-determined time frame.

With this, perhaps unintentionally, an interesting question is raised: can one become a context-driven tester? Or more generally, can one become a tester of a particular school? Or are you, from the first day of your career, already a tester belonging to a particular school of thought, irrespective of your knowledge, skills or experience, because you carry with you a certain worldview? We like to leave the answer to that question, and its implications open for discussion.

Aspects of the presentations that satisfied the context-driven presentation heuristics to a lesser degree

There is still a lack of understanding of certain aspects of context-driven testing , a number of which we would like to discuss here. One such aspect is the notion of context, which is often seen from limited perspective. Examples of this are:

  • context is the assignment that is given to you by the customer, or
  • context is the development method that is used.

To us, context is not something that is static and pinned-down, but dynamic: everything that happens in a project and everything that affects the project is part of the context and cannot, by definition, be laid down in a list. In order to deal with this, it is important that the tester understands what context is, that he is able to observe and analyze his context and that he has the skills to adapt to an ever-changing context. Talking about frameworks within which you should always work can  keep you from being receptive to the ever-changing context.

The mixing of schools is another common misconception. A number of speakers, such as Wim Ten Tuscher and Eibert Dijkgraaf, argued for an approach that consists of a mixing of schools in which you can choose, depending on the context, for one or the other school. From the context -driven perspective the verdict on this approach is very simple: it cannot be done! It is noteworthy that some speakers think it possible. The reason for the misunderstanding is that people confuse artifacts with paradigms. In every school you can use testing techniques, such as described in, for example, TMap or ISTQB.  Just as well you can use, in every school, session-based test management, which was introduced by the context-driven school. However, this is not a mixing of schools, but the use of artifacts from different schools. A factory school tester who uses context-driven artifacts, is not, all of the sudden, a bit context-driven. And a context-driven tester who makes use of artifacts that have been described by the factory school is not suddenly to a certain extent a factory school tester. The answer to the question of what testing is or what it comprises, does not change by mixing artifacts from different schools.

Another conspicuous point are some unsubstantiated claims, such as ‘context-driven testing can not be applied in complex chain testing’ or ‘in a certain situation it is best to take an approach from a particular school’. When confronted with these claims our advice is to use three powerful questions from James Bach, to stimulate critical thinking.

  • Huh? (Do I really understand what is being said?),
  • Really? (How do I know it’s true?) and
  • So? (Is this the only solution? So what?).

Unsubstantiated claims are often not based on experiences – let alone thorough research – but on assumptions that cannot be clarified or traced. If these assumptions are not met with critical examination, it is not possible to interpret them.

Jean-Paul Varwijk

Jean-Paul Varwijk presented at the TestNet Autumn Event 2013 on focusing testing more on the results.

What is also striking, is that context-driven testing is sometimes taken lightly. It is reduced to its (context-driven) principles, which then take on a universal character. Also, context -driven is sometimes equated with common sense. It is argued that context-driven testing is merely based on semantic discussions, or that it is something that everyone does but about which a group of people likes to make a lot of fuss. Here the context-driven community and DEWT in particular learned a valuable lesson, because apparently we have not been able to elucidate the importance of the paradigm, the context-driven semantics, what common sense means to context-driven testers, and why we love to talk about testing. Clearly, these are lessons learned for us.

What can be improved by the context-driven testing community, and specifically DEWT?

A frequently heard comment was that after a number of presentations, people still did not know what context-driven testing really is, or that, as previously stated, context-driven testers are a group of people who make things needlessly complicated. DEWT sets itself the task of explaining in a better way what context-driven testing is – without simplifying it. We want to pick up this glove immediately:

  • the first step is to share our experiences of the TestNet Autumn Event, and
  • the second step is an article for the TestNet News explaining context-driven testing.

Additionally, we are open to questions and discussions.

Looking back at the event, we, as DEWT, can say that it was an informative day, during which there was room for different voices and visions. We hope that some testers noticed that context-driven testing suits them and that they would like to know more about it. In that case, do not hesitate to get in touch with the DEWT members. And if you want to continue the discussion on context-driven testing within your company, or want to organize a meeting for your colleagues, we are here to help.

The pictures in this article were taken by Rik Marselis.

Program of the TestNet Autumn Event 2013

Program of the TestNet Autumn Event 2013

Posted in Context-driven testing | Tagged | Leave a comment