Getting Started With Cross Browser Testing
By Alex Langshall
You’re a tester. You’ve been handed a story that needs testing, and the developer working on it has asked you to do some cross browser testing on it. What does that mean? Why would someone want to test a story in multiple browsers? Which browsers should you test this in? What are you looking for while you are testing?
Testers should ask these questions of themselves and other stakeholders when cross browser testing is a requirement of a story or project. Getting a detailed analysis of an application in one browser is hard, let alone any other browser your software might support. Let’s take each of these questions one by one and look at solutions which could help you be successful at cross browser testing.
The Definition Of Cross Browser Testing
Cross browser testing is the activity of testing a feature in multiple, relevant browsers. It is an important piece in feature testing because, while there are standards for the web, each browser implements those standards differently. At a deep level, there are different behaviors in the browsers for different elements implemented on the page. How a feature behaves in Safari, for example, can be widely different than how it behaves in Chrome.
Testing should happen in multiple browsers any time there is a possibility of functional or cosmetic difference between different browsers. Any time that usability, display, or functional aspects of your site are being developed, testers should be doing testing in multiple browsers.
Cross browser testing assures that the user experience is consistent. Regardless of whether it’s existing features or newly written code, the application should appear to have the same functionality from browser to browser. For all of the “Internet Explorer is terrible” rhetoric, there are users that are locked into a choice of using the browser specified to them by work, school, or due to legacy software requirements. Those users deserve the best possible experience, regardless of your personal beliefs about the browser they are required to use.
The Browser Rubric
It’s not realistic to spend testing time looking at every single browser on the market. There are too many possible permutations to make that useful. Making smart choices about which browsers to test can save time and work. Using a risk based approach can narrow your testing focus down even further, taking some of the pain out of testing in multiple environments and the browsers associated with them.
If your product or website has explicitly supported browsers, that should be the list you would test from on a regular basis. If you only support IE11, Firefox, Safari, and Chrome - you could limit your testing to those four browsers. For testers who do not have an explicit policy for which browsers to use, gaining access to user statistics and choosing the top five browsers the application is accessed with is probably a good way to have a fair amount of cross browser coverage for an application.
If you have access to business analytics through Google Analytics, or something like Kissmetrics, you might be able to tell what the actual browser user statistics are for users using your product or site. While this can help you plan around the browsers that your users are actually using, it’s by no means a foolproof method of deciding which browsers to examine.
Some testers like to plan their cross browser testing using a chart that could look something like the example below. This allows the tester to make sure they have covered necessary variations and supported browsers for the OS and mobile OS versions supported and call out browsers that won’t be tested or supported for the application.
Selecting the browsers that seem to be the most troublesome for your application is always a good idea. It’s possible customer service has some insight about how many defects are logged against particular browsers or browser versions.
The majority of browser specific issues often encountered involve IE11 and Safari because of their proprietary nature. They often require web developers to use workarounds to get consistent behavior across all browsers. Extra attention should be paid to web elements in these two browsers.
Chrome and Firefox are both what are referred to as “evergreen” browsers. That means that these will automatically update as soon as a new release is available. They also have a rapid release schedule where they deploy new versions of the browser every month. Safari and IE11 have less frequent updates as they are dependent on operating system updates.
Changes can be accounted for by comparing beta versions or overnight releases of these two browsers. Chrome Canary and the Firefox Developer Version will give you access to a full version ahead of the current evergreen browsers. There is a Safari Technology Preview as well that will give you access to upcoming technologies in Safari.
A Note About Equipment
If there are limitations on equipment and access is limited to one computer, requesting a Mac as your testing computer is probably the best option. This allows for installing a VM with Windows and IE for testing. Unfortunately, it’s harder to legally gain VMs of a Mac OS if you are testing on a Windows machine.
Also consider whether you need to test in non-desktop browsers. On iOS devices, Safari is the standard. For Android, that’s Chrome, although there are a variety of alternative browsers on both platforms, and native browsers depending on the type of phone. For example, Samsung has it’s own web browser already installed on it’s devices. When investing in mobile testing, having the lowest supported screen size and the highest will give a reasonable matrix for testing. If the application could be used on other types of non-desktop browsers, obtaining devices like an Xbox, PlayStation, Apple TV or even a Wii U might be a good idea as they tend to have specific browsers for their console platforms.
Deciding What To Test Across Browsers
Depending on the kind of story that you are testing, if there are major UI changes or new features running through all the flows you are testing in each browser is a very good plan. If testing time is a factor, you may be best served by looking at things likely to break in different browsers.
If wireframes or mockups of the application are available, use them as a reference for your cross browser testing. By comparing elements in the mocks to the browser, a tester could quickly see anything that differs which could be a defect.
Talking with developers during testing about how they’ve implemented the feature and any browser-specific workarounds they are currently using could indicate or inform your cross browser testing and whether or not inconsistencies are really defects or how the feature was implemented due to the browser.
Warming Up With A Few Pointers
The list below is not an exhaustive list of what a tester should be looking for, but it’s a good list to start with.
UI elements: Start with UI elements.
Every button, box, text area, background, panel - these should all be rendered the same way in each browser.
Fonts should look the same.
Resize Windows - Resize the window to the minimal reasonable design and make sure that all elements remain usable at all sizes. If the design is reactive, then the elements should collapse in some way as the window shrinks. Look and see if you can find any points where the collapse has not happened, but the elements of the design are no longer present.
DPI - Make sure that all the elements render properly on a high-DPI screen versus a low-DPI screen.
Cursors - Any time there’s a change in the mouse cursor (hand, grasping hand, crosshairs), make sure that the correct changes are happening at the correct times.
Scrolling - Can you scroll correctly? Touch pad, touch button, mouse pointer, mouse wheel, up/down arrows, anything that lets a user scroll the application in the browser. Don’t forget to check horizontal as well as vertical scrolling where necessary.
Zooming Level - What happens when you change the zoom level in the browser. If responsive design is being used, elements should gracefully resize.
Tips And Tricks
The following are a few tips and tricks that should help when trying to reproduce bugs and write bug reports.
In Chrome and Firefox the URL “about:” will give you a summary of version and build information. Internet Explorer and Edge give version information in the “About” menu item at the bottom of the hamburger menu accessed to the right of the address bar. Safari will give version information from the “About Safari” menu item in the Apple menu.
Accessing Developer Tools:
Chrome, Firefox, IE11, and Edge use the F12 hotkey to open developer tools. If you have enabled the development tools in Safari, either Command-Option-C or Command-Option-I will do the same.
Some development tools will allow you to emulate mobile views or report that they are a different browser. In Chrome, the device toolbar (Control or Command-Shift-M) will let you change the reported device size. In IE11, there is a dropdown that will let you change the reported version of Internet Explorer.
With a Mouse: With the browser window focused, holding Control (or Command on a Mac) and using your scroll wheel can quickly change browser zoom to check for layout issues. This also works with a trackpad when using the supported gestural scroll. On Mac trackpads you can change zoom by holding Command and using the two finger scroll.
In-browser controls: Control (or Command) + and - will change zoom level by steps. Return to the default zoom with Control (or Command)-Zero.
Clearing Cache And Cookies in Browsers:
Chrome, Firefox, IE11, and Edge can clear cache and cookies with the Control-Shift-Delete hotkey (Command-Shift-Delete on a Mac). In Chrome and Firefox, this action is limited to the current profile, other profiles you may have will be unaffected. IE11 and Edge should be closed and restarted after this action is performed. Safari has a “Clear History…” in the “Safari” menu that performs the same action. It is possible to map a keyboard shortcut to “Clear History…” in Safari by using the Keyboard panel in System Preferences.
Isolated Instances of Chrome and Firefox:
Firefox and Chrome offer different profiles that can be used to provide complete and isolated instances of the browser. In Chrome, click the profile name in the upper right hand corner to get started. Firefox requires more setup.
Incognito mode or private browsing is available in Chrome, Firefox, Edge, Safari, and Internet Explorer. This provides a one-time scratchpad browser that loses it’s settings and cookies when closed. This is ideal for testing situations where persistent information is not important. Note that all incognito windows in a single browser share the same browser session. Cookies, for example, are shared between active incognito windows.
After some practice, you should be able to quickly scan a site or application for common issues and develop a good sense of what types of issues will be specific to the project.
James Bach describes a heuristic that he refers to as “galumphing”. Testing using this heuristic could uncover some interesting bugs. Consider that browsers do not often run in “ideal” circumstances. Tabs are left open for long periods of time, the browser is constantly placed into the background in favor of other applications, and other operating system actions may occur in the background. Some random play with the product in less than ideal conditions can give a tester insight into what kinds of issues customers might encounter.
Using Cross Browser Tools
Tools can be part of a comprehensive testing plan for cross browser testing. BrowseEmAll is a tool that can browse in multiple render engines at the same time in the same window. While it’s not a browser that customers would use, it’s an ideal tool designed for quickly checking for common elements, and possible alignment issues, or missing content.
Browserstack is a cloud based tool that runs different browsers inside a browser session. Their plugin allows access to sites inside your corporate VPN or firewall. It’s an excellent tool for looking at specific browser versions that you may not have available on your test machine. The browsers run in virtualization with Browserstack. If the product or web app relies on specific rendering engines (WebGL is a good example of this), Browserstack’s virtualization will not give you the same experience as a real browser.
There are several options for taking automated tests and running them across multiple browsers. Selenium Grid is a tool for hosting your own browser instances and running automated tests against them. If maintaining the necessary servers for Selenium Grid is cost prohibitive, Sauce Labs provides a cloud-based solution for comprehensive cross browser coverage.
Surviving CRM - cross-browser matrix image
About Alex Langshall
Alex Langshall has worked as a stage manager, grocery store checker, substitute teacher, college adjunct, and now tests software for Lucid Software. He lives in Salt Lake City with his partner, daughter, and two cats.