Keep Experimenting

By Jan Jaap Cannegieter and Michel van ‘t Zant

The idea of test mobs (also known as mobbing or mob testing) [1] is derived from mob programming. Based on the definition described by Woody Zuill [2], we can define a test mob as:

a software testing approach where a whole team tests the same thing, at the same time, in the same space and at the same computer.

The main difference compared to scripted testing, exploratory testing, or bug hunts is that a team is testing the functionality together, at the same time, in the same room at the same computer. Testing approaches like scripted testing or exploratory testing in which team members test on their own or in pairs are becoming more common. Other test approaches like bug hunt s[3] involve end-users, developers and testers all testing the system at the same but often testing different things.

The Context

After being motivated to mobbing at two conferences (APRIL 2017 in Tel Aviv, Israel [4] and Test Automation Day in Rotterdam, The Netherlands [5]) we started to experiment with test mobs in their project.

The project was the replacement of a core system in a non-profit organization. The system supported the main process of this organization; from the purchase to delivery and billing. The testing was partly done by professional testers, however, there was a team of users that also carried out testing activities. These users were not professional testers but they were trained by us so they could participate.

The Different Variants

At first, the developers and customer representatives who were going to use the system (the users) were a bit sceptical about test mobbing. A probable reason for that was that the users and developers didn’t interact very much with each other. Apart from three business owners and some managers, users communicated with developers or the other way around.

But there was one thing that irritated them both, there were a couple of issues which were open for a very long time. These issues were ‘ping-pong’-ed between the project team and the users. They were going from Opening to the solution, then rejection of the solution, retesting, reopening et cetera. Despite their scepticism about the idea of a test mob they were prepared to do the first mob to see whether this could solve these issues.

Organising A Test Mob

In this first mob, a dozen long-lasting defects were solved. Most of the defects were solved by explanation; the developers explaining how a task could be completed in a different way than expected, or the users explaining how they do their work. The defects that needed adjustments in the system were processed right away. The striking thing was that some of the developers and users introduced themselves to each other at the start of the mob; they clearly didn’t know each other.

The users saw immediately how beneficial test mobbing was to their work. Actually, this mob was all about communication. Direct face-to-face. This mob was seen as a big success and we did more. After a while the developers and users started to contact each other directly instead of waiting for a mob. It wasn’t possible to quantify this but the change in attitude and cooperation was a big success.

The Second Test Mob Variant

The second variant of a mob was more like a bug hunt. We made a list of global test cases upfront and with around 6 users we tested these cases in pairs. Every pair took a test case. Sometimes we discussed upfront with all the participants what to look for in that specific test case. Along the way, we found out it was better to group some test cases, especially when the end state of one test case could be the starting point of the next. When a serious defect was found we discussed it at the end of the session.

Inviting Management

The team used this kind of mob testing approach several times. Higher-level management was invited for some mobs. This had advantages and disadvantages. The biggest advantage was that the manager got a very good idea about the progress and especially why we couldn’t go into production with that part of the system at that moment. This meant the manager was able to explain in the steering committee why going to production at that time was not a good idea, instead of the test manager.

The biggest drawback was that some of the managers started discussions about the process design, system design and way of working. These things had already been discussed and most could not be changed at that moment. Nevertheless, a manager would start these discussions, often with everybody involved. This slowed down the pace. [7] [JJC8]

It was decided not to invite higher-level management again. They were informed about the progress by means of reports and demos. Or they were asked to join during the discussion of the defects at the end of a mob session.

The Third Mob Variant

In the third variant, we tried to involve developers in the mob. This actually worked pretty well, but for this variant, the developers were not needed the whole time. The developers were present during the start where they demonstrated the part of the system for the mob. They joined the mob for the last half hour in which defects were discussed.

After discussing defects, the developers fixed the defect. This way of working gives the user a lot more trust in working with that system or a part of the system.[9] [JJC10] We saw here again that direct, face-to-face communication led to a lot of clarity and took away a lot of misunderstanding. In reflection, this was the biggest benefit of test mobs.

The Mob That Is Not A Mob

Another variant of a test mob was end-to-end tests, in which we tested the interaction between the different systems. We didn’t call these sessions a test mob but it was a valuable variant.

In this mob, we had representatives of all the underlying systems. One person, called the driver, ‘fired off’ a test case. The different representatives checked whether the case was well processed in the system they were monitoring. The biggest benefit was the confidence the representatives gained from knowing the systems were processing correctly. It took a lot of time because we had made a detailed test specification upfront, but everybody was very satisfied with the result.

We reused a part of the test specification for a demo of the system for all the managers that were involved. Because we were talking about the core system of the organization, this was actually most of the senior management of the organization. For them, we demonstrated the system using different end-to-end test cases. We knew upfront the test cases would pass, we had already tested them before, but that was not the goal of this variant of a mob. The goal was to show the organization that the system could support their process and we succeeded in this. Instead of the good-old release advice, mostly written by the test manager, this organization decided to go into production based on first-hand information. They saw that it worked and the decision to go into production was not even a conscious one. Actually, it turned out this decision was already made in their heads when one of the senior managers asked at the end of the session when this was going to be live.

A Promising Concept

We described why we started with test mobs, how we started, what variants we used and what our experiences were. Basically, we took something promising that we’d heard at a test conference and we just started with it. From that moment on, we kept experimenting, trying new things, finding out what worked in our context. We don’t advise you to do the same as we did, but we do advise you to start with test mobs when you think they can add value in your organization or project. And when it does, keep experimenting, keep trying new things. This way you will find what works for you.

Authors Bios:

Jan Jaap Cannegieter
Jan Jaap Cannegieter is now Principal Consultant at Squerist, he did assignments in requirements, testing, quality assurance, process improvement, Agile and digitalization. Jan Jaap is a well-known author of several articles and books in the Netherlands and is a regular (keynote) speaker on international conferences.

Michel van ‘t Zant
MIchel is a medior tester at Squerist, he did several assignments in testing in both traditional and in Agile contexts.

Test Mobs In Practice