How Do They Work & Should You Use Them

By Tim Martin

Crowd testing providers are providing a valuable service for many of their customers and testers. For both groups, they have the potential to offer big advantages over more traditional forms of testing. These are mainly a quick way of getting excellent test coverage for customers and a very flexible way for testers to earn money. However, there are also some fundamental problems. For customers, this is mainly a lack of dedicated testers and limited communication with testers. For testers, it is mainly the large amount of competition they will face. For a beneficial experience, customers and testers should have a good understanding of the information here.

How Crowd Testing Companies Work

Testing work for customers of the provider is distributed amongst many different testers who work remotely, usually from home. Testers are paid by the provider using a performance based pay system where they earn money for each approved bug and test case.

Tester Onboarding

For testers, signing up with a provider is a quick and easy process. One of the first things new testers do when signing up is set up a profile. Here things can be specified like devices they can test with and demographics such as country, gender and age. This allows testers to be selected for test cycles based on these criteria. New testers often have to pass an entry test to demonstrate an understanding of how to use the platform and some basic testing skills. When this is done they are eligible to take part in paid test cycles.

Test Cycles

Work is typically split into short test cycles which last for a few days. Surveys are used to find testers for these cycles who meet specific criteria that are not already specified in tester profiles. For example, these surveys may ask if they have an account with a particular bank or regularly use a car hire company. Selected testers receive invitations to test cycles which they can accept or decline. Invitations contain information about things like what should be tested, contact information for test cycle managers (who work for the providers) and payout rates. Sometimes a chat channel is provided for testers and test cycle managers to communicate openly. Sometimes there is no chat channel and testers can only communicate with test cycle managers.

Performance Based Pay & Rating For Testers 

Bugs, test cases, and other work items submitted by testers are created and managed using a bespoke system built into the crowd testing platform. They are approved or rejected by test cycle managers and/or customers. They can be awarded different levels of value, the higher the value the better the payout for the tester. Testers have a rating which is defined by these approved/rejected work items. Testers with better ratings are likely to be invited to more test cycles with better payout rates.

Pros & Cons For Customers

👍 Coverage: Providers have large numbers of testers available who can test with different devices, demographics etc. This kind of diverse coverage is something that is not possible using more traditional forms of testing. Here are a few examples of when this approach can be very useful: finding bugs specific to different mobile devices, getting usability feedback from a broad range of users, realistic performance testing, and finding and fixing translation errors. 

👍 On Demand: Testing effort can be purchased in small units, typically per test cycle. A wide range of criteria can be specified by the customer for each test cycle including the number of testers and test cycle duration. Testing work may only be needed intermittently at certain points of a project’s life cycle. So crowd testing could provide large cost savings where resources like staffing are concerned compared to using more traditional employment models.

👍 Speed: Testing is generally available at any time of the day or week. Many testers are available to participate in test cycles. They are incentivised to find bugs and run test cases quickly, so a lot of testing effort can be focused into a small amount of time for time critical projects. Compare this to more traditional forms of testing where very large teams like this are not available so the same amount of testing effort would take significantly longer.

👎 Lack of Dedicated Testers: It’s in the providers business interests to keep moving testers between different customers rather than letting them be more dedicated to a smaller number. Testers will generally have a limited amount of in depth project and product knowledge. This can lead to poor quality results. For example, bugs being logged that have little value for customers or omit important information and a lot of communication noise.

👎 Communication: It’s in the providers business interests to limit communication between customers and testers. This can cause misunderstandings and delays. Remote working comes with its own classic communication problems such as lack of face-to-face interaction, problems with supporting technology and different time zones. Also, testers and test cycle managers may have different levels of fluency with the common language being used.

👎 Security: Customers should think carefully about the risks of sharing sensitive information with large numbers of remote testers they don’t know much about. For example, behind the scenes information such as access credentials could leave them vulnerable to hacking. New product features they want to keep a secret may be exposed to competitors. A significant amount of effort could be involved in mitigating these kinds of risk.

Pros & Cons for Testers

👍Flexible Working Conditions: With the performance based pay model, testers are essentially free to work whatever hours they like. This has the potential to really improve their work-life balance. Work can be done remotely from the comfort of a tester’s home so no commuting is required. If a tester does not like something described in a test cycle invitation such as payout or type of work involved, they can decline it with few negative consequences. 

👍Easy to Get Started: Onboarding for testers is a quick and easy process, without much risk, open to just about anyone with an internet connection. No rounds of interviews or extensive background checks for example. Giving it a try to see how it goes is simple. Providers are very keen to have large numbers of testers available for deployment. They will try to give every new tester an opportunity to take part in paid projects and keep them involved. 

👍 Learning Opportunities: Testers are given opportunities to test many different products, for many different customers. This is a great chance (especially for people who want to get started in testing) to learn. Key learning areas might include what customers value from testers, tools, market forces and new innovations. A permanently employed tester or contractor may have to test the same things for long periods of time which can become very monotonous.

👎 Not Well Paid: I believe this is true for most crowd testers, at least in the developed part of the world I live in. Many people will accept lower pay in exchange for flexible working conditions. Testers can build their rating/reputation and get invited to better paid projects, but there is a lot of competition so this can require significant effort. Work for providers that require testers to work away from their home usually has much better payout rates.

👎 Competitive: Test cycles normally have a maximum number of possible tester participants. These slots fill up quickly, so testers need to read and accept invitations quickly if they want to participate. Inside a test cycle, it’s often a race like an environment where testers are trying to claim test cases and log bugs as quickly as possible. This can be frustrating because the effort has to be moved away from giving quality results to doing things quickly.

👎 Privacy: Participation and selection for many test cycles involves sharing sensitive personal information with the provider and its customers such as addresses, payment card details, social media account details, audio & video recordings of testers, device log information and copies of identity documents. Over time, testers can end up sharing this kind of information with a large number of different companies, organisations and people.

Are you interested in continuing the conversation? Go over to The Club and join your fellow community members engaging in this topic and others. 

Biography

Tim Martin has been involved with testing since 2004, having made valuable contributions to many different projects since then. He worked as a tester for crowd testing providers between 2014 and 2018. He has a passion for trying out new things, learning about new technologies, developing new skills and getting involved with the wider testing community. He is excited about the changing landscape of the industry.