Testing Mobile Apps

gorgeousvassalSoftware and s/w Development

Nov 7, 2013 (4 years and 1 day ago)

70 views

+

Testing Mobile Apps

Joanne and
Xidong

+

Content


Types of Application on Mobile


Browser Application and its testing strategy


Extra: current
iOS

and Android Automation Testing
Software


Client Application and its testing strategy


Common Testing Technology

+

What is the focus of this topic?


To find an proper testing automation that significantly
reduces
the overall time needed to test each software
release.

+

3 Categorized Applications


Client Application:


Applications are installed on a mobile device and run on that
device, may be written to look and feel like a native application
for specific phone
models.


Messaging Application:


The servers need to receive and respond to the specifications of
SMS messages.


Browser Application:


Browser applications are server
-
based applications that can be
accessed through a web browser via a URL from a mobile device.


+

Aspects need to consider for
testing (in general)


The human languages (e.g., French, Kanji);


The locales (e.g., UK English, Australian, and US English),
which affect things like
formatting
numbers and currency
symbols;


Who pays for updates to be downloaded (users may be
unwilling to pay to download
updates
OTA);


How the software is installed on the device (e.g., in terms of
security permissions); and


The number of applications and versions you need to
support in parallel.

+

Goals when testing


To ensure we deliver attractive, easy
-
to
-
use, working
applications for as many users as practical. Lots of happy,
frequent users help show our software is successful and useful.


To have justified confidence in the quality of our software.
Ideally, most of the bugs would be found and fixed before the
software is widely used.


Fast feedback to developers. Fast feedback helps them to fix the
code while it is still “warm,” while they are still intimately
familiar with it.


To quickly detect issues so they can be addressed. This is
particularly relevant when the problem is related to external
factors (e.g., an operator’s network configuration or a specific
handset model).

+

Markup Language Application on
Mobile


Application is written HTML and run by browser in general.


But it has revolution for many years…

+

Markup Language Application on
Mobile (cont.)


Flash Lite is a system that enables devices like phones to
make use of a subset of the capabilities offered by Adobe
Flash. Flash Lite is used on some smartphones and feature
phones to enable things like embedded video viewing on
sites such as YouTube
.


XHTML Mobile Profile (XHTML MP) is a
hypertextual

computer language standard designed specifically for
mobile phones and other resource
-
constrained devices
.


Comparison of XHTML Mobile profile and XHTML basic.


http://mobiforge.com/designing/story/comparison
-
xhtml
-
mobile
-
profile
-
and
-
xhtml
-
basic



+

Testing Markup Language
Application on Mobile


Strategy: Using
a program or script since Markup languages
are designed to be processed by programs (such as the web
browser) rather than being directly interpreted by a human
.


Problems:
Cookies are sometimes intercepted by a gateway,
or a proxy server, provided by the carrier. Automated tests
may need to communicate across multiple distinct
connections, in parallel, probably across several devices or
computers, in order to trigger cookie sharing issues
.

+

AJAX Mobile Application


Ajax (Asynchronous JavaScript and XML) is a group of
interrelated web development techniques used on the client
-
side to create asynchronous web applications. With Ajax,
web applications can send data to, and retrieve data from, a
server asynchronously (in the background) without
interfering with the display and behavior of the existing
page.

+

Testing Strategy for AJAX Mobile
Application


Be able
to test rich UI interactions of the application running
within a web browser. (e.g., for an email application new
emails may arrive from time to time from the server without
any action by the user.
)


Testing how well an application copes with intermittent
network connectivity can be
convoluted
and may require an
intermediate
device. (e.g
., to intercept and modify network
packets
.)


Test specific features that are differentiated from applications
running on desktops. (e.g. rotation events)

+

What is Client Application?


A
pplications
are installed onto mobile wireless
devices.


They are able to provide users with more functionality, better
integration, and better performance than browser
applications.


Client applications can be divided into two categories:
portable applications and native
applications
.


Portable applications run in (or on) a virtual machine, such as the
Java Virtual Machine
. (e.g. Android)



Native applications are written to run directly on particular
architecture or platform of the target
devices. (e.g.
iOS
)

+

iOS

Automation Testing


ios
-
driver


Test any IOS native, hybrid, or mobile web application using the
Selenium /
Webdriver

API. IOS automation is as easy as automation
for a browser, due to reuse of the well known API.


High level features :


Implements the JSON wire protocol


Runs on emulators and devices


Can be run as a regular node in a Selenium Grid


Can be used in an existing Selenium Grid to manage device /
simulator farm


First class support for localized native apps ( 1 test works for all the
localized versions )


No need to change the source


No need to jailbreak the device


No
ios
-
driver app to install on the phone

+

iOS

Automation
Testing (cont.)


Appium

is an open source test automation framework for
use with native and hybrid mobile apps. It drives
iOS

and
Android apps using the
WebDriver

JSON wire protocol.

+

Android Automation Testing


MonkeyRunner
:


W
ritten
in
Python;


Has
APIs for connecting to a device, installing/uninstalling apps,
running apps, taking screenshots, comparing
images.


Robotium
:


A framework provides
functional, system and acceptance tests
running across multiple
activities
;

S


S
upports
Activities, Dialogs, Toasts, Menus and Context
Menus,
and
it is integrated with Maven and Ant for running continuous
integration
tests.

+

Android Automation
Testing (cont.)


Robolectric

(taken a different path)
:



Uses
shadow objects and runs the tests on a regular
workstation/server JVM avoiding
dexing
, packaging, deploying
and running the application tested on an emulator or a real
device, reducing the time needed to perform
tests.


LessPainful

(a step further)
:


Offering
a service for automated testing of applications on
multiple real devices. The user uploads the application (*.
apk
)
and a test file written in Cucumber, a business
-
readable DSL,
selects the configuration of devices he wants the tests to run on,
and then the tests are automatically executed and a report is
generated. The list of devices supported include Garmin Asus,
several HTCs, LGs, Samsung Galaxy, Sony
Xperia
, and Motorola
Motodefy
.

+

Techniques for Client Application


Automated Unit Tests


iOS

application
-

unit
testing framework is available on
code.google.com

(
http://
code.google.com
/p/
google
-
toolbox
-
for
-
mac/
) that even supports basic UI
testing
(
http://
code.google.com
/p/
google
-
toolbox
-
for
-
mac/wiki/
iPhoneUnitTesting
).


Client
applications are harder to
automate


Ideally the test automation would include interaction with the
native system events
(
e.g., for keyboard and other inputs) and the
ability to query the GUI layer (e.g., to read the contents of a text
box). Extra features could be provided to enhance our ability to
test the application (e.g., where system calls can be intercepted
and modified to force certain conditions to be triggered
.)



+


Automated Unit
Tests (cont.)


Example of Unit Tests






Running Unit Tests in Emulator


Challenging


capturing output, automating execution of the
tests in emulator


Code Injection


Used
to obtain information, e.g., statement coverage, timing data,
etc., or it may include automated
tests


protected void
testBasicAuthForEmptyStrings
() {




String
authResult

=
BasicAuth.encode
("", "");




assertEquals
("
EmptyStrings

should result in 4 byte result", 4,

authResult.length
());


}

Techniques for Client Application

+


Code Injection (cont.)


Unique features:


Source
code is not required for some forms of code injection,
e.g., to add code coverage
instrumentation.


The
application’s source code does not need
modifying


Injecting code requires
:


The application
,


The code to inject (which may be tightly coupled to the
implementation details of the application)
,


A way to inject the code,
and


Instructions on where and how to inject the code.

Techniques for Client Application

+


Test Tools for Client Applications


On
-
Device
debugging


Not
really a way to automate the testing


Emulators


Blackberry Emulator: supports
automation


Command
-
line tools are used to enter commands, and to
create screenshots


Challenges:


Driving the user
-
interface
.


Interpreting screen
-
responses, including stitching together
images for pages that require
scrolling
, pattern matching
where the content varies from request to requests
.


Latency in the device interaction


Techniques for Client Application

+

Common Testing Techniques


GUI
-
level Automation


Challenges (in terms of making the tests
relatively
reliable):


The need for high contrast,
opaque,
block colors for backgrounds,
etc., to make images


easier to match for menus, etc
.


Deciding whether to use image pattern
-
matching or
OCR (Optical
Character
Recognition)
.


Structuring test resources, e.g., you may need a set of images per
device
.


Image Stitching


Used
to assemble a number of screenshots into a composite single
image that
represents
the original page.


U
seful
when the page is too large to be displayed on a
physical
phone

+


Image Stitching (cont.)


Step 1:
Image
retrieval:







Step
2: Image
alignment:


Common Testing Techniques

+


Image Stitching (cont.)


Step
3: Image subdivision and election
:









Step 4: Image reconstruction:

Common Testing Techniques

+


Image Stitching (cont.)


Challenges of image matching:


Captured images may vary significantly from device model to
model: Content varies across screen sizes and resolutions








Common Testing Techniques

+


Image Stitching (cont.)


Optical
Character
Recognition (OCR): to
extract text from
images


Encoding Data in Pixels:


Each character of ASCII: 8 bits =>HEX “0xNN” (N= 0~F)


E.g. “A” = 0x41, “a” = 0x61







Common Testing Techniques

+


“Contact Sheet”


multicolumn format of rows of small
images
: screenshots of results
generated by
automated “user”
input


Using Transparency
Masking:


Common Testing Techniques

+


“Contact Sheet” (cont.)


Using
Transparency
Masking:


Common Testing Techniques

+


“Contact Sheet” (cont.)


Assisting test automation with human decision
making:

Common Testing Techniques

+


Model Based Testing (MBT)


used to test other software domains such as web sites,
desktop
applications, etc
.


helps to automate longer
-
running automated
tests


used MBT techniques to test a number of mobile wireless
applications, including a
search
engine and client applications.

Common Testing Techniques

+

When to test manually


UI and rendering problems:


alignment
issues,
flickering, etc.


When the UI is changing frequently, test scripts tend to break
and need repairing, requiring
lots
of work


When
content is dynamic and hard to
predict


When the cost of automating is likely to significantly exceed
the costs of manual testing

+

How to improve Testability


Generally:


Provide a complete API to enable the test automation code to test
“below” (without) the
GUI.


In
programming languages such as Java consider making
methods and data
protected.


Browser based application:


Add ID tags to the main elements the automated tests will interact
with


Client Applications


Consider using high
-
contrast colors, visual markers, and even
pixel
-
encoding to make GUI
-
based automation more reliable


SMS Applications


Provide a scriptable library or interface (e.g., available from
Python) to make tests easier to create.


+

Reference


http://mobiforge.com/designing/story/comparison
-
xhtml
-
mobile
-
profile
-
and
-
xhtml
-
basic


http://www.mobileburn.com/definition.jsp?term=Flash+
Lite


http://en.wikipedia.org/wiki/
AJAX


http://ios
-
driver.github.io/ios
-
driver/
index.html


http://
appium.io


http://appium.io/getting
-
started.html#hacking
-
with
-
appium
-
ios


http://www.infoq.com/news/2011/09/Android
-
Testing