4 Key Challenges of Mobile Testing

By Mush Honda

The growth of mobile technology is nothing short of phenomenal. The hardware has improved dramatically over the last few years, but it’s the software that truly unleashes the potential of devices. What’s really driving the incredible success of smartphones and tablets is the application market. According to ABI Research the mobile app market will be worth $27 billion in 2013 and that’s projected to grow further in the years to come.

A meteoric rise like this inevitably creates problems for developers, as they work out how to produce apps quickly and efficiently for a diverse set of devices. QA departments are left playing catch up as they endeavour to test apps properly before they are rushed out the door. Testing for the mobile market is a new challenge. It’s a fast-paced industry that’s constantly changing, and that throws up some unique problems.

The trouble with mobile testing

There are four key challenges that differentiate mobile testing. These are the main barriers to overcome if your mobile testing plan is to end in success.

  1. Fragmentation – Mobile apps are expected to run on a wide variety of different devices, often running a number of different OS versions. With Android, you also have to factor in individual manufacturer UI’s. The number of possible permutations is intimidating.
  2. Networks – The performance of a carrier’s network can have a huge impact on the user’s experience with an app. This is another factor beyond the hardware and software that, nevertheless, must be taken into consideration for testing scenarios.
  3. Usability – The way we interact with touchscreens is still evolving and clever user interface design is essential if you want to create an app that people can pick up and use intuitively. Look beyond the functional aspects of the app and consider usability.
  4. Tools – As mobile testing is still in its infancy the tools at your disposal are limited when compared to traditional testing. There is an evolving arsenal of tools available for mobile testers who know where to find them.

There’s a lot to consider in the planning stage. A winning testing strategy is going to include an optimal mix of devices for maximum coverage. It will include some real devices, some emulators, and perhaps some remote access devices delivered via a cloud service. There will also have to be a consideration of real world conditions. How will different locations and fluctuations in connectivity impact on the app? Consider how the app handles interruptions such as incoming calls, text messages, or even Facebook notifications.

Naturally, the testing itself will be broken down to cover four main areas – functional, performance, security and usability. The level of automation that can be achieved will be a key factor in making the project more manageable.

Adopt a pragmatic mindset

A pragmatic approach is always invaluable in testing. It’s impossible to cover every possible angle, but there are lots of practical tips worth bearing in mind.

Knowing where to focus your resources is a prerequisite for getting the best out of them. You can’t cover an exhaustive combination of all devices and OS versions, so consider the product and its intended use, and target the most important devices. If this is a mass market app then analyse what device has the biggest market share and focus on the OS version with the greatest penetration. Investigate what is popular with the demographic being targeted and you can extract more value from your testing.

Think about different screen sizes and produce a requirement checklist for interactions. You need to cover interactions through virtual buttons, as well as hardware buttons. How is text input handled? What are the navigational workflows? Gestures and touchscreen inputs throw up a whole new set of problems.

Keep a fluid structure

You aren’t going to create a complete testing plan and then just rinse and repeat until the app ships. Requirements will change during the course of the project and you need to plan in stages.

Start out using emulators to identify bugs and functional problems. You can move to real devices, and possibly remotely accessed devices, later when you begin testing in real-world conditions and considering security and network impact.

You will validate the quality of the product primarily through functional and usability testing, but be aware that new features may be thrown into the mix during development. Everything must be validated before it is released into the live product. Consider compatibility and the impact of upgrades.

Testing in the real-world

It’s always important to emulate the end-user as much as humanly possible and that means going way beyond optimal conditions.

What is the impact of low signal strength, 3G or 4G networks, and different Wi-Fi speeds? What happens when the network drops and you switch to Wi-Fi or vice versa? How does the app operate offline? Are there constraints with particular chipsets, or is memory being impacted in unexpected ways? You need to think about low-level resource usage and watch for buffering problems or garbage data generation. Security considerations include data encryption and multi-user support handling. What about the impact of low battery, or other phone functionality?

This is really just the tip of the iceberg for effective mobile testing. We can’t possibly cover all scenarios here, but this should give you a flavour of what to consider, and help you to create a solid mobile testing plan.

Author Bio

Mush Honda is QA Director at KMS Technology, a provider of IT services across the software development lifecycle with offices in Atlanta, GA and Ho Chi Minh City, Vietnam. He was previously a tester at Ernst & Young, Nexidia, Colibrium Partners and Connecture. KMS services include application management, testing, support, professional services and staff augmentation.