Designing Tests: What’s the difference between a good test and a bad test?

By Mark Winteringham

The Software Testing Clinic is a safe environment for those who are interested in software testing to learn and enhance their testing skills. It also enables more experienced testers to learn and enhance their mentoring skills. At Software Testing Clinic events, we get asked a lot of good questions and not all of them we can answer in detail. This series explores some of the more common questions attendees ask.

Recently the question of what makes a test good or bad has come up, not only at Software Testing Clinic but at some of my own personal workshops that I’ve run. Personally, I don’t believe there is such thing as a good test or a bad test. Even if I run the most simple and shallow of tests, if it reveals a bug, helps me come up with a new test idea, or reveals some information that is new or useful to me, then it’s a good test. This doesn’t mean that I can solely rely on simple or shallow tests.

There are many different types of tests and all of them can be of value to a tester; I want to ensure that each one I decide to employ is of the highest quality I can muster. The quality of a test will determine the value of the test. A poor quality test tells me very little, something I already knew or worse, nothing at all. A quality test will help me continue testing, and share new and interesting information with the team. So let’s explore how to make a quality test.

The Role Of Testing

Before looking at how to create a quality test, it’s important to ask “why is there testing in the first place?” “What is the role of testing?” These can be slightly contentious topics (and ones for another article) but my view is that my role as a tester should be to learn as much about the product and project as possible. I want to discover as much useful information as possible to share with my team to help them be awesome. This information can tell me:

  • How a stakeholder(s) wants the product to work.
  • How it actually works.
  • How to come up with new test ideas.
  • Are there any bugs within the product.

As a tester, it’s my responsibility to go forth and find that information; something that the very excellent EvilTester promotes regularly in his books and talks. This responsibility means that I have the freedom to create any type of test I want, but I need to ensure that what I create is revealing information that my team can use to succeed.

Working With Risk

We need to determine what information we want and see how that information can inform our testing. Our products are subject to many different risks. For example, there are risks that could break the product, damage the reputation of your company, or cause injury to someone! As testers, the most important information we want to discover is information about those risks. A quality test is therefore focused on revealing information related to a risk because it is allowing you to be open all kinds of details. Whereas, if you are testing simply to find bugs, then you may miss information to assist in new test ideas, whether the product is meeting business expectations or even bugs that we don’t know are bugs until we share what we’ve learnt with others.

Before we create our test, we need to identify and discuss potential risks we are concerned about, as a team. As a product or business analyst shares their vision and requirements we can attempt to understand those ideas by discussing them and asking questions. These activities will help us to form ideas about the risks we want to learn about.

Forming Test Ideas

Once you understand the risks your testing is focused toward, and you have something in front of you to test, then it’s time to start thinking about your test ideas. Remember, a quality test is one that will reveal valuable information to you related to specific risks. Try to think of something specific you want to learn about your product. Creating questions in your head, or noting them down, is a valuable contribution towards coming up with ideas. For example, you could try using the five W’s H technique which allows you to ask questions such as:

  • What if I was to add bad data in?
  • How does this form send data to a service?
  • When I click on this button twelve times what happens?

Take your test ideas and think about how you are going to execute them against your systems under test. Let’s say you are testing for a risk that exists in the backend system when it is populating data in a database. What would happen and what would you learn by executing a test for that backend via the Graphical User Interface versus an HTTP interface? Each of these approaches will behave differently and return different information.

Think about the things you might be gaining or missing from testing with a GUI or a HTTP interface. For example, working on the HTTP interface might tell you that any bugs found are specifically coming from the backend, whereas if you did it via GUI, the bug could look like it was from a GUI technology such as JavaScript. If you were testing via the GUI, you might discover that certain values being returned from the backend were breaking the layout of the product.

Conduct The Test

It’s important to remember that when you execute a test, it is only as good as your observational skills. Crafting a quality test to reveal information is worthless if you don’t learn from it. If you are focused on specific behaviour, actions or pieces of information, you may miss other important details. This is a common error humans make in the form of a cognitive bias, which is also known as inattentional blindness, where an individual fails to notice unexpected events that are appearing in plain view. However, if you try to broadly take in lots of details at the same time, the information can become overwhelming and have you scrambling to take down notes or think of ideas for your next test. You need to strike a balance with how much information you want to be looking for or receiving.

Closing The Loop

We’ve looked at some key points to create a quality test, but how can you judge something that is subjective? This simple heuristic can tell you if your test is good quality:

“if you feel you are learning more about what you are testing, then it’s probably a good test.”

It should help you think of new ideas, new risks, and inform you of important details which you want to share with your team (both good and bad).

If you find that what you’ve learnt from your tests isn’t useful, it may be for a number of reasons:

  1. The test itself was poor or wasn’t executed correctly. It may be you need to re-evaluate it or run it again.
  2. You are repeating yourself to a degree and learning nothing new about what you’re testing.
  3. You’ve simply run out of ideas. This may be due to fatigue or exhausting all the test ideas you have.

What you learn from your test should determine your next activity. Take time to step back from what you have tested, examine what you have done and question it:

  • Are your tests at a level of quality that you are happy with?
  • How can they be improved?
  • What information could be missing?

Wrap Up

Good tests and bad tests do not really exist. It’s up to us as testers to create quality tests and determine what are the most suitable tests to discover information about specific risks. We can do this by questioning and discussing ideas with our teams. Additionally by discovering risks, forming test ideas around those risks, and being observant when executing them. Not every test will be of the same quality, or be structured in the same way, so you need to remain vigilant. It’s up to us, as we test, to question what we have done and what we have learned, to see if assumptions were made or if details were missed. We can always learn and improve to deliver better testing.

References

Author Bio:

Mark Winteringham is a tester, coach, mentor, teacher and international speaker, presenting workshops and talks on technical testing techniques. He has worked on award-winning projects across a wide variety of technology sectors ranging from broadcast, digital, financial and public sector working with various Web, mobile and desktop technologies. Mark is an expert in technical testing and test automation and is a passionate advocate of risk-based automation and automation in testing practises which he regularly blogs about at mwtestconsultancy.co.uk he is also the co-founder of the Software Testing Clinic in London, a regular workshop for new and junior testers to receive free mentoring and lessons in software testing. Mark also has a keen interest in various technologies, developing new apps and Internet of things devices regularly. You can get in touch with Mark on Twitter: @2bittester