Quickly understand the essentials of Android device testing

With the rapid rise of smart phones and tablet devices in the consumer electronics field in recent years, all manufacturers have been trying their best to compete. If we divide the market by operating system, leaving aside the long-standing Nokia Symbian operating system, it can be said that Apple's iOS and Google's Android are currently competing with each other, while RIM's Blackberry and Microsoft's Windows Phone 7 (and the recently launched Mango) have also tried to break through with different strategies.

According to the latest statistics released by technology industry market research organization Gartner in August this year, it can be clearly seen that the sales of smartphones equipped with the Android operating system have made a significant leap. In addition, for iOS, it is the product window period before Apple launches the new generation of iPhone, which is a major boost to Android sales.

However, the transition of high-tech industries is changing rapidly. The division of the software industry seems to have been resolved, but the hardware industry is surging. The most shocking news in recent days is that Google has acquired the mobile division of Motorola, a mobile phone giant, and HP changed its mind again two months after announcing the divestment of its PC business, saying that it will retain its personal computer business. The news of Google's acquisition of Moto Mobility is seen as a benefit for obtaining a large number of mobile phone patents to continue to deepen its presence in the mobile phone market; while HP's policy reversal on the PC business, the initial split plan is seen as a move of unwillingness to continue to invest huge costs, and the final choice to retain the PC business is speculated to be a decision made after evaluating the long-term market potential and the impact on the overall brand value. Regardless of the impact that the daily changes in the market will have on the rise and fall of hardware in the future, it is certain that, for now, the industry is generally optimistic that iOS and Android will continue to be the mainstream mobile device operating platforms.

Since Apple's iOS adopts a closed software design, the open source Android operating system has become a lesson that manufacturers who want to conquer the smartphone and tablet device market must learn. As a leading laboratory unit for testing and verification, Allion Test Labs, Inc. is naturally not absent. In the past, we have dedicated an article to deconstruct the basic technical architecture of Android and explore the five major technical aspects of Android device software and hardware integration (please see the full text on the Technical Direct website). This time, we will explore the key points that must be paid attention to and verified in Android from the perspective of testing experts.

Before that, let's take a look at some simple data to prove that Android is unstoppable. According to the official data officially released by Google I/O Conference in May this year, some statistics as of that time are as follows:

• There are more than 100 million Android devices used by consumers worldwide
• 36 OEM manufacturers
• 215 telecom operators (carriers)
• 450,000 Android developers
• 310 Android devices
• Sold in 112 countries
• More than 200,000 apps exist in the Android Market
• Number of Android devices activated daily: 100,000 (May 2010)  200,000 (August 2010)  300,000 (December 2010)  400,000 (May 2011)

*The above data is as of May 2011

Although the above data is only collected before the I/O conference in May this year, it is enough to show the rapid growth of Android. And because of the high degree of openness of Android, both manufacturers and individuals can become developers. As a result of their respective development, different risks of problems may arise in design or during the introduction. So what are the main verification points for manufacturers? Please see the following analysis:

Common Android Device Risks

Why do we need to discuss the risk of Android problems first? An astonishing data shows that the number of Android OS bugs reported by hardware manufacturers (IHVs), developers and users has reached 120,000 so far, including various problems and defects of varying sizes, some of which only affect the smoothness of operation, while others are critical bugs that may cause data leakage or system crashes. From this perspective, as an open system, Android must be integrated with hardware vendors and application programs, so there are many design aspects that must be carefully considered. Common Android problems that are currently known or discussed include:

1. Security

Recently, there have been several pieces of news that have caused the public to question the security of smartphones. For example, both iOS and Android systems have features that record the user's geographic location and can be used by someone with ulterior motives to track whereabouts (involving personal privacy); and security vulnerabilities before Android version 2.3.4: when using an unencrypted Wi-Fi network (Open Wi-Fi), personal information including contacts, calendars, personal photo albums, and other Google services may be leaked and stolen by hackers. What's more, since the Android Market does not have clear and strict content standards for the review of applications on the shelves, the phishing and virus problems that bother users have also been transplanted from the PC side to Android devices. These malicious programs (malware) containing viruses may not only affect the operating performance of the device, but also allow someone with ulterior motives to steal user data through remote control and conduct illegal theft.

It is worth noting that these various security issues are likely to become potential threats to personal privacy. In this era where mobile phones are always with you, users begin to store various private information, personal photos, important meeting data, etc. in mobile devices such as mobile phones or tablets , not to mention using these devices to trade stocks or log in to various website platforms. However, the biggest concern is that once these users' accounts and passwords are copied and stolen, the consequences can be disastrous. Consumers naturally do not want such concerns to continue to exist, so that they can only passively scan for viruses through the mobile antivirus software that has sprung up recently. What consumers expect is that the relevant manufacturers will take active actions to eliminate these security concerns. Therefore, how to find problems to strengthen the deficiencies that Android originally failed to detect and how to avoid the intrusion of malicious programs can be said to be the current primary task of Android devices and a major challenge that manufacturers have to face.

2. Integration of customized UI

Looking at the products of major Android phone manufacturers on the market, it is not difficult to find that several major phone manufacturers will package their own user interfaces (UI) under the native Android software framework when designing their products, including HTC's Sense UI, SAMSUNG's TouchWiz UI and Motorola's MotoBlur UI, which are all customized UIs with distinctive characteristics that are further developed from Android. The purpose of manufacturers developing these independent UIs is nothing more than to provide consumers with more convenient and diverse usage methods and to consolidate their own niches through product differentiation. For example, HTC's Sense UI allows users to set desktop shortcut paging, social network content aggregation, convenient time and weather widget presentation, etc.; and SAMSUNG's TouchWiz UI also provides users with desktop widget shortcut settings and drag-and-drop touch feedback.

Why might problems occur with the user interfaces designed by these manufacturers? Because the operating systems of these devices are all rooted in Android, any changes to the details must be compatible with the original Android framework. Therefore, how to cleverly combine the functions pursued by the manufacturers with the native Android architecture to maintain smooth use is a big question. How to avoid losing sight of one thing while focusing on another and prevent possible errors in use caused by incorrect changes are all key points that must be verified in detail.

III. Supporting measures for version upgrade

Unlike iPhone, which can update the operating system version freely from the client, the operating system of Android devices is pre-installed with a single Android version when the phone leaves the factory. As for whether to provide subsequent upgrades, it depends on whether the mobile phone manufacturer releases updates for each model. Some manufacturers will provide updates as soon as possible, while some manufacturers' development schedules take several months, and some manufacturers even decide not to provide upgrades directly. In such a case, it is very likely that Android has released version 2.3, but users can only use Android version 2.2 of the operating system. This is not only equivalent to users not being able to enjoy the bug fixes or function expansions of the updated version, but will also greatly affect the user's perception, which will definitely have a negative impact on the continued expansion of the Android device market.

In order to reduce the questions that users have when purchasing, such as "Will there be upgrades in the future?" and "Will it take a long time to wait for the upgraded version?", manufacturers must carefully consider how to design a complete software architecture in the early stages of a product development so that they can quickly respond to subsequent Android version upgrades and launch updated firmware upgrades in a timely manner. All of this requires comprehensive verification skills combined with software and hardware. After all, when the operating system (software) version is upgraded, the memory, processor, camera module, etc. (hardware) on the device will not be upgraded accordingly. Therefore, from the beginning of product development to the supporting response after the new operating system comes out, to ensure that the software update can indeed cooperate with the hardware, it will be a link that manufacturers must strengthen.

In addition, Google has proposed an 18-month guaranteed upgrade plan in response to the frequent problem of users being unable to upgrade. Developers who join the plan must provide consumers with any operating system updates released during this period. This plan is a great boon for consumers, but for a variety of Android mobile device developers, while developing new products, they still have to consider the upgrade of old products. Under the pressure of this tight development schedule and development resources, they must pay more attention to the quality of upgrade verification.

4. Application Upward and Downward Compatibility

For application developers, whenever the Android operating system is updated, they must re-examine whether their applications can run on the new operating system version. However, the actual situation is that when consumers upgrade their Android device version, the original App on the device may not be compatible with the new version, and they must wait for the developer to launch an updated version of the App; or the newly launched App only supports the new version of the Android operating system, and users of devices with older versions of the operating system cannot use the new App on their devices.

In addition to the operating system version, the compatibility of apps on different devices is also closely related to differences in hardware specifications such as OpenGL support, screen size, resolution, etc. The official Android website also specifically explains how developers can design apps that are compatible with different screen sizes. Even so, the differentiated designs of Android by various manufacturers will still lead to compatibility issues. Only actual user verification on various Android devices can ensure the compatibility quality of the app.

The longer it takes for developers to update their apps, the longer users have to wait and the more likely they are to switch to similar apps from other developers. The fewer operating system versions an app supports, the smaller its potential customer base. Therefore, in addition to whether the app itself is attractive to users and whether it is smooth to use, when designing an app, it is also important for application developers to consider support for existing/previous versions and how to respond to future operating system version upgrades in real time.

In addition to the common problems with Android devices mentioned above, since these devices have multiple functions such as web browsing, audio and video multimedia, and gaming entertainment, there are naturally many functional, compatibility, and reliability issues that will arise accordingly, such as the signal reception capabilities of 3G and Wi-Fi modules, power consumption, and external accessory compatibility. Due to the complex characteristics of Android devices, detailed testing and verification are required to ensure the quality level of the device. Based on years of professional experience, Allion has summarized the following verification framework for the product characteristics of Android devices for reference by the industry. Here, we propose several points that must be paid attention to when testing and verifying Android device products. A set of test and verification methods for Android device product characteristics is summarized and compiled. The main verification framework and test items are described below.

Android device verification architecture and key test items

From the perspective of verification architecture, it can be divided into three major parts: hardware, operating system (OS) and application (App). Hardware verification includes the functionality of various components in the device, the suitability of new components and the compatibility between different modules, etc.; operating system verification includes the integration of the manufacturer's customized UI and the native Android system, the ability to operate normally during version upgrades, and the elimination of the most important security threats, etc.; as for application verification, it includes the functionality and ease of use of the App itself, as well as the upward/downward compatibility between the App and the operating system and the compatibility of various device hardware and screen specifications, etc. Only when these three aspects are taken into account can it be considered a comprehensive verification of the Android device.

As for the actual test focus, it can be mainly approached from the following angles:

• Functionality

  In order to ensure that Android devices can be used normally by users, functional verification is naturally indispensable, from communication modules (Wi-Fi, 3G, GPS, FM, Bluetooth, etc.), touch screen (including various sensing components), camera module, transmission interface (USB, HDMI, etc.), audio and video playback function, recording function, expansion slot, system operation capability (multi-tasking, DRM function, security protection) and other special functions such as gyroscope, magnetometer, etc., all must be covered.

• Application

  Whether it is a smartphone or a tablet, a major selling point of any Android device is that in addition to the built-in basic apps, users can freely download more apps of different functional types. Usually, the default apps of the machine are nothing more than web browsers, video players, notepads, messaging, phone calls, contacts, and maps. The compatibility of these apps with the machine must be ensured to be well operated before leaving the factory to avoid problems such as system crashes and data loss. The third-party apps that users can use are varied, ranging from various games, e-books, productivity software, communication tools, real-time synchronization software, etc. It will be a very time-consuming approach to test these hundreds of thousands of apps one by one to ensure that they are fully compatible with the user's device. Therefore, filtering these apps and selecting the most representative and most in need of verification is the homework that must be done before verification.



• Interoperability

  The third verification focus is on the interoperability between different devices. Since the functions provided by Android devices are like a microcomputer, they may need to be connected to many different electronic products in many usage scenarios, such as surfing the Internet through wireless APs, transmitting data between other devices, connecting to external keyboards and mice, wireless printing, etc. For this reason, interoperability verification with a large number of mainstream external devices and consumer electronic products on the market to ensure normal functions and avoid the public's negative impression that "a certain brand of Android products often cannot be detected and used" is naturally a necessary measure to maintain a good user experience and maintain a good reputation for the product.

• Android Compatibility

  In order to ensure the high compatibility of the various Android devices and operating systems on the market, Google has also designed an Android Compatibility Program. Only through this program can it be ensured that the tested device is compatible with the Android operating system, and can use the Android trademark image and the Android Market. The so-called Android Compatibility Program includes the CDD file (Compatibility Definition Document) and the CTS suite (Compatibility Test Suite). Device manufacturers must first follow the specifications in the CDD file, which defines the functional items that must be included in each different Android system version, as well as the software and hardware capabilities, performance levels, and security regulations that must be supported, and then conduct CTS suite testing. All product problems and errors in the CTS suite test must be corrected and passed before they are considered compatible with the Android operating system.

  However, the current CDD files and CTS suites are only basic level specifications, providing a one-size-fits-all test policy that cannot cover all aspects that must be verified. More advanced functional reviews still rely on internal verification by each manufacturer. In view of this, Allion has also begun to cooperate with many major manufacturers at this stage. In addition to continuing to delve into the development of advanced automated testing tools and programs, it also provides exclusive test plans and product design suggestions for the characteristics of Android devices of different manufacturers to meet the compatibility verification needs that different manufacturers may have in various functional projects.

• Storage

  Android devices usually use embedded eMMC as storage media. However, based on NAND Flash technology, although it is suitable for mobile devices, it is more prone to data loss, performance degradation, and tolerance than traditional storage methods. Imagine if the photos taken by the user, the important matters temporarily recorded, or the meeting data created by the tablet at any time just disappeared out of thin air, what a huge loss it would be for the user, and it would be a great harm to the reliability of the product itself. Therefore, ensuring the integrity of the data maintenance of the storage media is also one of the indispensable prerequisites for verifying Android devices.

• Wireless Connectivity

  Smartphones and tablet computers make mobile Internet access possible, but they also bring more aspects to the related devices that need attention, such as whether the compatibility of the network chip will cause no signal or weak signal, whether the placement of the module in the mechanical design will cause mutual interference between different signal sources and affect the reception quality, the sensitivity of the receiver, and whether it can share signals normally when used as a Wi-Fi hotspot, etc. For users who may need to use the Internet anytime and anywhere, whether the portable device has smooth Internet connection capabilities is definitely the key to whether to buy it.

• Audio Quality

  In addition to mobile Internet access, being able to listen to music and browse videos at any time is also a major demand of Android devices. Therefore, whether it can provide good reading and listening enjoyment will greatly affect the user's perception. In the past, the public generally did not have too high requirements for the ability to play audio and video through mobile phones or other portable devices. That is because we define the role of mobile phones as making and receiving calls; but when smart phones appeared and tablet computers rose, the high prices and dazzling promotional slogans made consumers have higher expectations for these products. External speakers designed specifically for mobile phones or tablet devices began to appear on the market. In such an era, ensuring excellent sound quality has become a major incentive to attract consumers.

We have mentioned in previous articles that there is a re-sampling design for the sound source in the source code of the Android operating system. If the manufacturer does not pay attention and modify it, the designed device will re-sample the 48K Hz sound source to 44K Hz, causing harmonic distortion and affecting the playback sound quality of the device. According to our actual test results, it is found that this is a common problem for most Android device manufacturers. Although sound quality verification is not a technical expertise for device manufacturers that were not oriented towards sound quality in the past and requires a higher entry threshold, if problems can be discovered and corrected in a timely manner, it will definitely be an advantage that can distinguish them from other competitors.

• Performance

  In addition to the functions mentioned above, which must be able to operate normally, another verification item worth emphasizing is performance. For the same Android device with the same hardware specifications, if one device takes 20 seconds to open the application, while the other only takes 10 seconds, most consumers will naturally choose the latter; or if you suddenly see a picture you want to capture on the road, phone A takes 6 seconds to turn on the shooting mode, while phone B only takes 2 seconds, then naturally you will choose phone B so you won't miss the wonderful shot. The same principle can also be applied to the speed of browsing the web, the time required to start the phone, etc., which all need to be verified. At the same time, in order to understand the true performance of the product, you can also compare and analyze it with other brands on the market through benchmark testing, find out the strengths of your own products, and strengthen the weaknesses, so that you can know yourself and the enemy.

• Reliability

  The reliability of a product is related to the probability of errors in the product and the system's ability to recover from errors. For example, whether a warning window will pop up when the connection is abnormal, whether more than a dozen applications can be opened and executed in the background at the same time during multitasking, whether the program can avoid being unresponsive or crashing when an error occurs, and whether it can be restarted by restarting after a crash, etc. The more errors occur, the lower the user's favorability towards the product will be, and it may even affect the trust in the brand's quality. Therefore, simulating various possible user usage scenarios to find possible product problems and conducting comprehensive error testing will be a necessary way to ensure product reliability.



• Power Consumption & Battery Life

  When we talk about mobile devices, we have to mention the issue of power consumption. Unlike desktop computers or other fixed electronic products, mobile devices are designed without a continuous power supply, so the length of standby time becomes crucial. If you attend a full day of meetings, no one wants the portable device that you use to contact customers, send and receive emails, or record work items to suddenly run out of power. Therefore, a reasonable and good standby capacity has become a major selling point for mobile devices. The standby time and the time that can be used for various operations of mobile devices are not only related to the battery capacity, but also the power consumption of the hardware components themselves and the software design are possible sources of problems, such as the recent iPhone 4S standby time problem. In addition, the endurance after continuous charging, the number of times it can be recharged, and the stability of the battery's own power supply from a safety perspective are all important items when conducting battery testing.

• OS Upgrade

  The last verification point that cannot be ignored is the integration verification during the operating system upgrade. When the Android version is upgraded, the manufacturer must spend a lot of effort to modify the firmware version. How to ensure that the original customized UI and all functions are normal, optimize the device performance and reach the best state, and complete the upgrade of the new version at the same time is the focus of this verification project. A new operating system version usually symbolizes the addition of new functions and the correction of errors. Therefore, the skill lies in maintaining these two prerequisites as much as possible under the existing hardware specifications and customized functions. The upgrade of the operating system actually means that all functions of the device need to be fully verified. At the same time, it is necessary to verify whether the upgrade action will cause cases that have actually occurred before: loss of original data, device crashes, etc. In fact, it can be said that it is more difficult than simply introducing a new operating system to the product, and it also requires the assistance of a third party to avoid blind spots in verification.

• Comprehensive verification to pursue high-quality Android devices

  Looking at the Android devices on the market, whether they are smart phones or tablet computers, they are all relatively new products developed recently. From the current actual situation, there are still many problems to be overcome. In order to help manufacturers develop more smoothly and provide consumers with higher quality products, Allion has also used its many years of testing experience and verification capabilities in network communications, handheld devices, storage products, mobile computing, and software and hardware integration to tailor-make the most suitable testing solutions for Android devices.

Different from simple functional verification, Allion adds the necessary verification points for all Android devices mentioned above to avoid common problem risks. Allion also provides complete error analysis through the collection and organization of problem databases, and further assists in problem troubleshooting and quality improvement. This article only discusses the common problem risks of current Android devices from an open perspective, and shares our testing concepts as a reference indicator for related manufacturers. We hope that by continuously promoting the technical maturity and quality improvement of Android devices, consumers can enjoy better and higher-level products.

(This article is provided by Allion Test Labs, Inc., a testing services and consulting company)