How to QA test a cross-platform, white-label mobile app
(Editor’s note: Rafael Costa contributed to this post)
Recently, ArcTouch product manager Ben Ogilvie wrote about how our EmployBridge app project is a case study in mobile platform excellence. This family of Android and iOS apps, built for six different EmployBridge-owned staffing brands across, recently earned the “Genius” award from the American Staffing Association as the industry’s top-rated mobile career app. We’re pretty proud of that.
It took a lot of mobile platform thinking and a white-label approach to app development to create a trusted, reliable and successful mobile app with 12 releases every two weeks. The platform in this white-label project was bigger than just creating a foundation of shared code. We scaled our agile process to manage this project, created common design assets and built a modular code base, and modified our devops methods for simultaneous releases. And, the focus of this post, we modified our QA test process so we could efficiently test these 12 apps — yet maintain the high-quality QA standards ArcTouch is known for.
What white label means in the world of mobile apps
There are many companies that offer white-label products in different industries that can be purchased by other companies, packaged and branded, and sold (or offered for free) to users. In the world of mobile, it’s fairly common for app development companies to build a mobile solution and then “re-skin” and resell it to various companies. For example, there are several event-based app builders that allow companies to add custom content and branding to a pre-built mobile app and use it for their customer and trade events.
In the case of EmployBridge, our white-label approach was a bit different. EmployBridge owns 16 leading staffing brands under its name. So, we created one single project that encompassed six branded apps, developed using Xamarin.Forms, and 12 releases (iOS and Android). The end result is 12 apps that leverage a unified API. They are similar in design and function, but every app is its own unique package of code. As a QA team, we had to define how to test those 12 different packages thoroughly for each release in our two-week sprints.
The challenge of QA testing a white-label app and 12 releases
As part of every project, we used automated testing tools to ensure quality on the thousands of device variations in the market. In this project, we used Microsoft’s App Center Test cloud device farm, which runs automated tests created with the Xamarin UI framework. That allowed us to run test scenarios using the SpecFlow language with NUnit unit testing framework (it converts our scenarios into automation steps). But none of this, of course, replaces the need to do real device testing. And that’s where things got interesting with this white-label app project.
When the project started, we were tasked with creating Android and iOS apps for three of EmployBridge’s brands, for a total of six releases. At the time, our project team consisted of five software engineers, one QA analyst, and one QA tester. The testing challenge was magnified when the number of brands grew to six.
The high number of releases made it even more important to have a functional mobile app “device coverage matrix.” The device coverage matrix allows us to get the broadest possible coverage from our physical app QA testing — without testing every possible combination of device, screen resolution (pixel density) and OS.
This was our original device matrix when we were testing the three white-label app variations:
We determined the Tier 1 devices by working with EmployBridge to identify the most frequently used devices by their target users — people looking to find jobs. Tier 2 devices represent a group of still somewhat widely used devices — but not as common as Tier 1.
The Tier 1 devices are also used by our engineers during development to test their own work before sending it to the QA test team. All our developers test their individual user stories on actual devices, not just simulators. Our QA team fully tests all the stories in the sprint on the full matrix of Tier 1 and Tier 2 devices. When the project scaled up to all the EmployBridge brands, we had to manually test six apps in 10 devices, for iOS and Android, and also test it in both English and Spanish. The only way to effectively do this — and stay within schedule and budget — was to use the common features of the white-label apps to our advantage.
Some important QA test questions to figure out
To successfully QA test this many releases, we had to answer some important questions:
- For every app, do we need to repeat the same test scenarios?
- How long will it take to execute all scenarios?
- Should we modify our test strategy for testing white label apps?
To answer these questions, we examined our typical process. This is what the process would look like for 12 releases and the number of test executions we’d have to perform:
- Devices: 10 (5 iOS x 5 Android)
- Executions for each scenario (6 brands x 10 devices): 60
- Executions for sprint regression (3 days, 10 devices): 2,140
- Executions for full regression (6 days, 10 devices): 9,820
Performing nearly 12,000 total executions in the course of some two-week sprints simply wouldn’t be possible given the time allocated to QA testing. So, we started looking at different ways we could test common components across the apps. What we determined was that the apps only truly differ in their look and feel (a.k.a. the user interface design) — but had 99% of the same functionality. So we could dramatically increase the efficiency of our testing by separating the functionality of the app from its design.
Each specific branded app only needed to be tested on the specific features for that app, which included mostly the design and a few bits of custom functionality. And, we could add in some happy-path end-to-end testing to verify that the visual changes didn’t accidentally break the functionality.
So, our QA test and engineering teams jointly defined the features to be tested, and when to test those features.
Here is how we run the tests now in the 12 releases:
- Devices: 10 (5 iOS x 5 Android)
- Execution time for each scenario: It varies based on the functionality and the impact on other features. In some cases, we decreased the execution number to 20.
- Executions for sprint regression (2 days, 10 devices): 880, a 59% decrease
- Execution time for full regression (3 days, 4 devices): 3,532, a 64% decrease
- QAs and engineers work together during sprint planning to define which features should be tested in which brands (using the below custom field we created in JIRA)
In all, we’ve saved about 60% of our total time spent running executions.
Lessons learned and advice to QA test your white-label app
We learned some valuable lessons during this project.
First, good communication between QA testers and engineers is essential! Our team has such a great relationship here in the ArcTouch Brasil office. We are true partners and we all want the same thing: apps that excel in every way.
Second, a thorough evaluation of new features — and how they are common or different from other related features — can help when executing tests on a series of white-label apps. Otherwise, QA professionals may be inefficiently performing redundant tests.
Third, this project reinforced the importance of mobile test automation, which could decrease the number of manual testing needed.
Finally, the success of your white label app, like any app, will depend upon how it performs in the hands of your target users. Throughout the development process, we’ve partnered with EmployBridge to do user testing — and we’re reading the user reviews in Apple’s App Store and Google Play. The feedback so far has been awesome — and we’re thrilled. We’re confident that our approach to white-label app testing is key to our client’s success.