What Do Tests Look Like?

By Mark Winteringham

The Software Testing Clinic is a safe environment for those who are interested in software testing to learn and enhance their testing skills. It also enables more experienced testers to learn and enhance their mentoring skills. At Software Testing Clinic events, we get asked a lot of good questions and not all of them we can answer in detail. This series explores some of the more common questions attendees ask.

The question we are exploring is: ‘What do tests look like?’ It challenges us to think about the activities we do as testers, the output that comes from those activities, and the effect that those outputs can have on how others view what we do as testers. It also demonstrates the different options which are available to us and the choices we have to make in testing when weighing the different styles for creating tests.

What’s Your Flavour?

So what do tests look like? Well, it depends. Tests can be created in many different ways and the different styles depend on the context you are in, for example:

  • Do I have to write my tests before or after execution?

  • What am I testing for?

    • Validation of requirements?

    • To learn about a feature?

    • To ensure we are building what the business wants?

  • How are we organising ourselves to create our product? Are we doing it iteratively or in phases?

Different approaches to creating tests have strengths and weaknesses and it’s good to be informed about them. It may be that you have the opportunity to choose what types of tests you want to create, so you can make an informed choice. Or it may be that you don’t have a choice. It’s good to know if there are requirements which have to be met due to needs from the business like regulations, metrics, or visibility in general.

So let’s take a look at a few different types of tests, how they are used and consider some of their strengths and weaknesses.

Test Scripts

These are probably the most common output of testing. They are usually a series of steps for someone to follow with sections to fill in observations. Below is an example: Test scripts are usually built before testing commences with a view to test specific requirements or assumptions as they are delivered by development.

So what value might you get from using test scripts?

  • For non-testers and those new to testing, test scripts can be a good starting point to learn about how the application works.

  • Test cases can be a desirable way of proving that your application has met any legal requirements.

And what about weaknesses or pitfalls to be aware of?

  • As test scripts are written up front there are added costs in generating scripts and maintaining them.

  • Tests scripts steps are explicit instructions to follow, which means whoever reads your script might misunderstand your step because you made assumptions when creating the script.

  • Test scripts retread the same path each time which means that some bugs won’t be found unless you do some other types of testing, which is a negative impact on your test coverage.

Acceptance Tests

Acceptance tests are more abstract, business focused, tests that are attempting to check that the team is developing what the business wants. They can be written in the same the format as test scripts but more high level so that non-technical users can interpret how the application is expected to work. However, they can also be written using a language known as Gherkin, which has gained popularity in the last few years.

Gherkin follows the format of Given…, When…., Then…. like the example below:

        Given I am a registered user of the product

        When I log in with my credentials

        Then I should be logged into the product

Sometimes testers get confused about the use of Gherkin and acceptance tests and fall into the trap of trying to use Gherkin to create test scripts. Acceptance tests should be treated as a guide for the team rather than a tool for validation. This means teams can use a few acceptance tests to guide them towards developing what the business wants and not create exhaustive tests using Gherkin.

So what value might you get from using acceptance tests?

  • Acceptance tests are best generated in a group with developers and business owners. The high level, abstract, nature of them means they are great for initiating discussions/collaborations.

  • Show how the app behaves when all the acceptance criteria are put together

  • Acceptance tests, especially in Gherkin, describe not only the rules of the application in the form of acceptance criteria but also the expectation of how those rules will behave when an end user interacts with them.

And what about weaknesses or pitfalls to be aware of?

  • Acceptance tests are often misunderstood and the format is used to create test scripts. Which means you don’t get the collaborative value from them (too many to talk through) and they don’t offer enough detail (they are too abstract)

  • Acceptance tests come from the business perspective and don’t cover all the technical risks there might be in the application. So other testing activities are required.

Exploratory Testing Notes

Unlike the tests above, exploratory testing notes are not created before testing begins, but during a testing session. They document what has been tested throughout an exploratory testing session.  For future exploratory testing sessions, testing notes can be used as a reference or even a guide to additional testing. Exploratory testing notes can also offer a tester freedom when they are doing their testing in terms of extra time available and choosing what to test. Creating good exploratory testing notes is a skill in itself.  

Exploratory testing notes don’t have to follow any particular format.They could be a series of notes in a notepad, they could follow a Session-based test management approach, or they might come in the form of a mind map like the one shown below:

Mind map source: How To Use Mind Maps To Develop Clarity With Your Software Testing Strategy

It’s worth experimenting with different note-taking approaches, and taking advantage of debriefs at the end of exploratory testing sessions to share your notes with others, and get feedback on clarity.

So what value might you get from using exploratory testing notes?

  • Documenting your testing as you’re doing it helps highlight what you are doing for your testing sessions beyond just the execution of ideas. For example, they can track session setup, bug investigation and bug reporting.

  • Your testing notes can help generate other testing ideas.

And what about weaknesses or pitfalls to be aware of?

  • There is a lot more responsibility on you to create test notes. You need to feel confident that your notes are written in a way so that if you had to look at them six months later you will be able to remember what you tested, whilst also not having exhaustive notes that take too long to write up.

  • Exploratory testing and test notes can be hard-to-sell as an approach to teams used to using things like test scripts.

The examples above are just a few of the popular options, but there are other ways in which you can approach documenting your tests. Some test plans or test strategies might even mix and match styles to ensure there is a good level of test coverage. It’s worth exploring and trying out each in your own time to see how they can help you.

The Test, The Whole Test And Nothing But The Test

So it’s worth noting that whilst it’s important to know about how we document tests we shouldn’t let them define us as testers.

Tests are an artifact of testing. They are an output of a testing process, and the testing process is complex with lots of different activities taking place. Consider a test that documents that you should test a button to ensure it’s green. If we dig into what testing activities are carried out around that test we might find activities such as:

  • Asking the Product manager why do we need a green button

  • Asking a developer about the code they wrote to create the green button

  • Setting up an environment to test the green button

  • Raising issues and questions if the green button turns out to be red

  • Writing a report about what has been learnt about the green button

  • Creating a series or reusable checklists

  • Reviewing the project to see what tasks can be automated.

The list can go on! There are so many activities that are not shown within a test step.

A good tester needs to ensure they have the skills to communicate and collaborate with a team and talk about what testing is and what they require to do good testing. Testers that rely on only documenting tests and test results are at risk oversimplifying a job that requires much more than writing and executing test cases. They also risk creating the perception that testing is easy. This view can lead to problems around your status in a team and may even encourage the idea that you could be replaced by test automation. Documented Tests are a tool and they should be used wisely to solve a specific problem rather than a replacement for critical thinking and communication.

If you have a question for the Software Testing Clinic team that you would like to see answered, get in touch with us at swtc@ministryoftesting.com ‚Äč

Author Bio:

Mark is a tester, teacher, coach, mentor and international speaker, presenting workshops and talks on technical testing and automation in testing. He has worked on award-winning projects across a wide variety of technology sectors ranging from broadcast, digital, financial and public sector working with various Web, mobile and desktop technologies. Mark is an expert in technical testing and test automation and is a passionate advocate of risk-based automation and automation in testing practises which he regularly blogs about at mwtestconsultancy.co.uk he is also the co-founder of the Software Testing Clinic in London, a regular workshop for new and junior testers to receive free mentoring and lessons in software testing. Mark also has a keen interest in various technologies, developing new apps and Internet of thing devices regularly. You can get in touch with Mark on twitter: @2bittester