What Mobile Challenges Do Software Testers Face In 2016?

We asked the software testing community what mobile challenges software testers face in 2016.  Here are the best answers!

Unexpected pop-ups

Submitted by Jason Huggins, Tapster.io

It's hard to write a repeatable automation test script when at any given time, an ad or other alert-style window appears and obscures the screen below. This has been a problem since the creation of the GUI and is still a problem in 2016. (Ads are mostly the problem now.)

Solution:

Create a routine that is always running in a background thread and scanning the screen for a list of "usual suspect" pop-up windows. And if found, perform the actions necessary to remove it -- usually it's sending an "Escape" key event. You also have to make the main script flexible enough to know to wait if it gets "stuck" by a pop-up.

In The Wild Testing

Submitted by Daniel Knott

One of the biggest challenges software testers need to handle in 2016 is manual "In the wild testing". The problem that I see, is that many software testers, test mobile applications in the same way as they would test web or desktop applications, always on the same place with the same Internet connection. But mobile apps are not the same like web or desktop applications. They are used in a total different environment with different Internet connections. If a mobile app is tested only with a fast and stable Internet connection lots of problems will not be covered during the testing and development phase.

Testing the app in the wild may find problems with:

The light situation because of sun or rain.
With the sensors because they are interfered by other elements such as buildings, cars or other obstacles.
With the Internet connection. Due to the movement, uses have not always the same Internet connection and this has an impact to the mobile app under test. For example, the network switch from LTE to Edge may influence the UI or the information that will be processed between the backend and the mobile app.
The listed points are just some of the problems that will not be found in a lab situation.

Solution:

The solution for this problem is pretty easy. Software testers must test outside the office. Depending on the use case of the app, the environment may change. For example if you are testing a business app, the app is likely used when people are commuting to or from work by train or car. Or if you are testing a sports app, that is tracking your workouts and activities, then you must test during this situations, too.

It is key for every software tester to test in the environment where the customers will use the app. I recommend to every software tester, to pack some devices in a backpack and to leave the office to test the app in the wild.

When testing in the wild it is important that the app under test is using a crash reporting tool like HockeyApp or Crashlytics to record any kind of problems that may happen during the test scenarios. Besides that I recommend to have just a pen and a piece of paper with you to make notes when discovering a problem.

Fragmentation

Submitted by Maaike Brinkhof

There are soooo many Android devices out there that you can only test a small fraction of them. Add to that the different flavours of OS's (Samsung adds something, other manufacturers do so too) and you have many potential risks.

Solution:

I'd like it if only stock Android was used but hey...I can keep on dreaming I'm afraid. As far as I know, there's no simple solution. You just have to engage with your customer and talk about which devices / OS's must be tested (based on facts like most popular devices in your country etc).

Native vs Webview

Submitted by Zachary Borrelli

This has been a continuous problem that will most likely be around for a long time; the willingness to get an API ready and complete and not resorting to having to use web views and other clunky alternatives.

Testing a hybrid app in the sense that some parts are native and some parts are webviews is a particular pain in the butt. When using these two side-by-side you can pretty much always notice that native is faster so a webview will always look slower and buggy. There's almost always a problem with the API and a webview talking to each other.

Solution:

If you're going to build an app, make sure you have all the APIs you need ready and raring to go before dev starts or early enough that something can be done about it.

If not go for all webviews and keep the app looking consistent in one form or another. At least this way it won't be noticeably different as you traverse the app.

Mobile Test Automation

Submitted by Stephen Janaway

Mobile test automation isn't in a great place -- the market is dominated by either big player pushing sub-standard, expensive tools to 'automate all the things' or smaller open-source tools that are at the mercy of the mobile OS vendors.

Too many people still have an 'automate all the things' test strategy for mobile, not realising that doing this is even more fool-hardy than on desktop web. Mobiles are complicated and automating a lot via the app UI is not the best strategy at all.

Solution:

Focus on understanding your application and the web services it most likely uses. Automate those first. Apply a small, light-weight automated UI test set using whatever automation tool suits you best. Be aware that the automation tool you use may stop working because an OS vendor cut's access to APIs so have a plan B.

Commoditisation of Android

Submitted by Dominic Kua

As it becomes simpler and simpler for someone to walk into a mall in ShanZhen and order a few thousand phones made to their specification from off the shelf parts, we're seeing an explosion in small phone OEMs. These are finding their way into people's pockets from the well advertised (Wileyfox) to random Chinese brands that you can buy on Amazon or Ebay (Cubot, Coolpad etc). These may not come with Google services, or come with Cyanogen OS instead of stock Android, and often will have custom launchers and applications baked into the ROM.

The number of devices you can practically test against is becoming a vanishingly small fraction of the whole. Thus your coverage of potential markets is always declining, diluting the value of testing on hardware as opposed to emulators.

Solution:

There're two solutions which come to mind immediately.

First; give in. Abandon all pretence of testing against hardware and only focus on emulators, or something as generic as possible like a Nexus device. Accept that hardware issues and custom ROMs are going to cause issues and try and test using random failures of components and calls.

Second: have a supported platform list. Say you support handsets X, Y and Z fully and provide partial support on all others. Continue testing as fully as you can on the supported devices and attempt to answer customer queries as they come in.

Slow Automation Tools

Submitted by Jarod McBride

While it's only one small aspect of testing, not having fast reliable UI automation tools to perform automated checks is a huge pain. Tools like Appium and Xcode UI Automation try but fall short in the speed dept.

Solution:

I think the key to fixing this is working with the platform developers (Google, Apple, Microsoft, etc) to show that this is a desired function of testing and worth putting development resources into.

Mobile Last

Submitted by Stefan Papusoi

At least in my company they say they are promoting mobile first thinking but Mobile development remains the last one to be implemented/released/tested if ever. Testing sometimes is done very briefly because -- mobile is not that important then.

Solution:

I don't have a solution for upper management and politics. It's always about the money -- the more people you have working on stuff the faster you usually get it done. People can be redistributed after project to others.  So 3 people working on a project for 1.5 years can be moved to 6 people working on a project for 1 year which could include mobile development.

Immature Automation Tools

Submitted by Szymon Kazmierczak

Web testers have Selenium to cover their test automation needs. Selenium has been around for more or less 10 years. In Mobile space, majority of the tools have been around for less than 2-3 years! Not only that, there is no "go to" automation tool, they all have their pros and cons, thus fragmenting the market.

Solution:

More people should get involved in pushing forward the mobile test automation -- support major open source automation frameworks! For example, Appium is about to have it's 1.5 release, which is a major rewrite that will help new contributors get up to speed faster.