Building The Ultimate Device Matrixx - Meetup

Picking The Right Set of
Mobile Devices

By Brian Kitchener

Software Quality Architect

bkitchener@prototest.com

Overview

•
About me

•
Some Background

•
The Problem

•
Understanding Android

•
How Apps Work

•
Building a Device Matrix

•
Example Matrices

•
Conclusion






About Me

•
Software Quality Architect at
ProtoTest

•
We're
a mobile test lab

that combines usability testing with quality
assurance

to

cultivate

a great user
experience

•
Project Architect, Technical Lead, Trainer.

•
Started in QA in 2001

•
BA in Applied Computing from University of Denver

•
Testing background : Functional, Performance, UAT,
Security, API, Database.

•
Automation : Selenium,
WebDriver
,
WatiN
,
MonkeyTalk
, SOASTA,
Fitnesse
, QTP,
EggPlant
, Squish

•
Languages : C#, Java, Ruby,
Javascript

BACKGROUND
INFORMATION

Some Stats for 2012

•
Mobile Apps achieved $17 billion in sales

•
5.2 Mobile Subscribers

–
1.2 Billion PC’s

–
4.2 Billion people use a toothbrush

–
1 Billion Smartphones

•
722 Million Smartphones sold

•
1.4 Million
iOS

+ Android Apps

•
25 developers
= half of app revenue

iPhone
-

June 2007

About the iPhone

•
Steve Ballmer :
Microsoft CEO

–
“There’s no chance the iPhone is going to gain significant market
share. No chance
.”

•
Patrick Stewart:

–
“
Last Wednesday, I stupidly dropped my iPhone in the bath, and my
life has sort of spiraled almost out of control
.”

•
Jon Rubinstein
–

Palm CEO

–
Is there a toaster that also knows how to brew coffee? There is no such
combined device, because it would not make anything better than an
individual toaster or coffee machine. It works the same way with the
iPod, the digital camera or mobile phone: it is important to have
specialized devices
.

•
Mike
Lazaridis

–

Blackberry CEO

–
And
so what
[the iPhone] has actually
done is increased our sales
.

ANDROID IS THE
PROBLEM

And Then There Were Two…

•
Android unveiled November 2007

•
First device was sold in October
2008.

•
Over 11,000 models have been
released.

•
48 Billion app installs

•
Over 1 Billion Android devices
activated

•
8 OS Revisions

OS Fragmentation

Device Fragmentation


Let’s do some math!

•
16 device display categories

•
20 different common resolutions

•
8 OS versions

•
6 Hardware Manufacturers

•
4 Major cellular networks

•
16
x 20 x 8 x 6 x 4 =
76,800 permutations

•
Pairwise approach = over 30 permutations

•
Who can afford to increase testing by 30X?


Our Approach

•
Efficiency, not coverage

•
Flexible: support small or large number of devices

•
Understand how apps work to logically select criteria

•
Use Market research to pick most common
configurations

•
Pick minimum and maximum boundary values for each
criteria

•
Choose a value that matches an edge case or abnormal
configuration.

•
Pick values that stress or tax the system



UNDERSTANDING
ANDROID

Android

•
Built and Maintained by Google

•
Open Source

•
Built on Linux kernel

•
ARM

•
X86 Ports

•
Built to support almost any type of device

–
Phones, tablets, phablets, media players,
tv’s
,
watches, etc.

•
Device Manufacturers customize code.




Example: Kindle Fire

•
Forked Android 2.3

–
Not updateable

•
Customized UI

•
Separate App store

•
Not all android apps work

•
Custom web browser




The Operating System

•
Google Releases “stock”
versions

•
10 Major Releases since
2008

–
API Level, not Version

•
Device manufacturers like
to customize the OS

–
Drivers, libraries, UI

•
“Stock” OS available in
Nexus devices or an
Emulator



Simulators / Emulators

•
Simulator imitates the software layer

–
OS and Libraries

–
Apple provides a simulator in
xCode

IDE

•
Emulator duplicates the hardware and software

–
Processor and Memory

–
Cannot mimic GPU, GPS, accelerometer

•
Always run stock OS

•
Can be used to test some functionality

•
Should always test on a physical device too

The Processor

•
ARM RISC
-
based instruction set

•
Specification defined by ARM holdings

•
32 bit

•
Same as
iOS

•
X86 patches and ports

•
It’s only a spec, can be modified.





SoC

–

System On A Chip

•
Main Board

•
Processor

•
RAM Bus

•
GPU

•
May include :

–
Cellular

–
WiFi

–
NFC

–
GPS

–
Bluetooth



2 Samsung Galaxy S4s

Quallcomm

Snapdragon

•
Quallcomm

Krait 300

–
Quad core ARMv7
Cortex A15 Architecture

•
Adreno

320 GPU

•
Dual Channel 533Mhz
Bus

•
Integrated LTE

Samsung
Exynos

5
Octa

•
Samsung
Big.little

processor

–
Quad core Cortex A 15

–
Quad core Cortex A7

•
PowerVR

SGX 544 GPU

•
Dual Channel 800Mhz
Bus

•
No Integrated LTE

Common
SoC

Manufacturer

Device Name

Cores

Processor

GPU

Qualcomm

Snapdragon S4

2 or 4

ARM Cortex
-
A15

Adreno

Nvidia

Tegra 3

4+1

ARM Cortex
-
A9

Geforce

Samsung

Exynos 4

2 or 4

ARM Cortex
-
A9

Mali

Intel

Medfeld

1

Intel x86

PowerVR

Texas
Instruments

OMAP 4

2+2

ARM Cortex
A9

PowerVR

ST
-
Ericcson

NovaThor

2

ARM Cortex
-
A9

PowerVR

Resolution is not enough

•
Unlimited number of
screen sizes available

•
Screens range from 3” to 11”

•
Each screen has a resolution, same as a
monitor

–
If you increase the resolution everything shrinks!

•
Pixels per Inch = Density

•
Screen Size + Density = Display Bucket

•
Resolution is not enough!


The Display Buckets

Size :

•
Xlarge

: 8”
-

10” tablet.

•
Large : 5”
-

7” tablet.

•
Normal : 3.5”
-

5” phones.

•
Small : 3”
-

3.5” phones.

Density :

•
ldpi

= Low DPI (~120)

•
mdpi

= Medium DPI (~160)

•
hdpi

= High DPI (~240)

•
xhdpi

= Extra High DPI
(~
320)


Low density
(120),

ldpi

Medium
density
(160),

mdpi

High
density
(240),

hdpi

Extra high
density
(320),

xhdpi

Small
screen

QVGA
(240x320)

480x640

Normal
screen

WQVGA400
(240x400)

WQVGA432
(240x432)

HVGA
(320x480)

WVGA800
(480x800)


WVGA854
(480x854)


600x1024

640x960

Large
screen

WVGA800*
*
(480x800)


WVGA854*
* (480x854)

WVGA800*
(480x800)


WVGA854*
(480x854)


600x1024

Extra
Large
screen

1024x600

WXGA
(1280x800)
†

1024x768

1280x768

1536x1152

1920x1152


1920x1200

2048x1536

2560x1536


2560x1600

Display Buckets

•
Galaxy S3

–
1280 x 720

–
Xhdpi

density (331ppi)

–
Normal screen (4.7”)

•
Galaxy Tab 10.1

–
1280 x 800

–
ldpi

density (149ppi)

–
Xlarge

sceren

(10.1”)

•
Galaxy Note LTE

–
1280 x 800

–
hdpi

density (285ppi)

–
Large Screen (5.5”)




Market Analysis

Aspect Ratio

•
UI is manipulated from code

•
Density Pixels adjust for screen size

–
But can use regular pixels!

•
Need to take both X and Y into account!

–
Easy to overlap or hide things

•
Includes orientation

•
Some devices include an aspect ratio changer!
(LG
Optimus

Vu)

Cellular Carrier

•
Four Major US Networks

–
Verizon, Sprint, AT&T, T
-
Mobile

–
Some phone interoperability

–
2 protocols


•
GSM
–

T
-
Mobile AT&T

•
CDMA
–

Verizon and Sprint

–
Carriers assigned specific frequency bands

–
LTE will be new standard
-

But spectrum issues will
prevent cross
-
network phones

•
So if the phone supports the carrier’s protocol
and band it can theoretically connect.


HOW APPS WORK

How Apps work

•
Apps need to work on all screen sizes


–
May not be functional

–
May be wasted space

–
May not make sense

•
Apps define XML layouts similar to HTML


–
Node structure

–
Static Content
–

Images,
etc

–
Dynamic Content
–

Color, Text, etc.

Layouts and Fragments

•
XML Fragments are
reusable components

•
Layouts stitch together
fragments for a specific
sized device

•
App may need different
flow for tablet
vs

phone


BUILDING THE

DEVICE MATRIX

Our Criteria

•
Operating System

–
OS customizations, missing libraries, driver issues,

•
Screen Size

–
Rendering issues, usability, missing layouts

•
Pixel Density

–
Density Independence, missing layouts.

•
Aspect Ratio

–
X,Y calculations, overlapping panels, display issues

•
SoC

–
Hardware performance, Instruction set, battery, signal

•
Carrier

–
Network protocol, speed, responsiveness, packet loss



The Goal

•
Efficiency, not coverage!

•
Build a set of devices to be used for app and
website testing.

•
Know when to update them

•
Define a list of simple categories of devices

•
Pick devices that offer broad coverage

•
Adjust the number of devices based upon
needed coverage




Categorical Approach

•
Define scope

–

Android,
iOS
, phone, tablet, etc
.

•
Understand Testing requirements

•
Self
-
descriptive Names

•
Help to broaden coverage

•
Will adjust devices chosen to cover our criteria

•
Should be apparent when to update a device

•
Spread coverage :

–
Usage
-
> Edge Cases
-
> Strange
-
> Stress

Example Categories

•
Common

–
Matches most common
display configuration

•
Newest

–
Latest OS version, largest
screen, highest resolution

•
Oldest

–
Oldest, slowest, smallest
device.

•
Abnormal

–
Non
-
standard OS, aspect
ratio, orientation, size

•
Popular

–
Most popular device in
terms of sales

•
Budget

–
Low
-
priced new model.
Tend to have strange specs

•
Flagship

–
Nexus device running stock
Android OS

•
Catch
-
All

–
Cover any missing
criteria

Android Phone Matrix

March

2012

Device Name

OS

Display

Aspe
ct

SoC

Carrier

Newest

HTC Droid DNA

4.2

Normal
-
xhdpi

9:16

Snapdragon S4

Verizon

Oldest

HTC Tattoo

1.6

Small
-
ldpi

3:4

Snapdragon S1

AT&T

Common

Motorola Droid 3

2.3

Normal
-
hdpi

9:16

TI OMAP 4

Verizon

Popular

Samsung Galaxy
S3

4.1

Normal
-
xhdpi

9:16

Exynos 4

Sprint

Abnormal

LG Optimus VU

4

Large
-
hdpi

3:4

Nvidia Tegra 3

Tmobile

Flagship

LG Nexus 4

4.2

Normal
-
xhdpi

3:5

Snapdragon S4

TMobile

Budget

Dell Venue

2.2

Normal
-
mdpi

3:5

Snapdragon S3

AT&T

Catch
-
All

Sony Xperia P

2.3

Normal
-
hdpi

9:16

Sony NovaThor

AT&T

iOS

Matrix

March
2012

Device
Name

OS

Display

Aspect

SoC

Carrier

Newest

iPhone 5S

7

4” 1136 x 640 326ppi

9㨱:

䅰灬攠e4扩W⁁

T
-
䵯扩le

Oldest

iPhone 3g

6

3.5” 320 x 480 165ppi

2㨳

䅰灬攠䄳

A否T

Common

iPhone 5

6

4” 1136 x 640 326ppi

9㨱:

䅰灬攠䄵

V敲iz潮o

Popular

iPhone 4

6

3.5” 640x960 330ppi

2㨳

䅰灬攠䄴

印物SW

iPad
(Retina)

iPad 3

7

10” 1536x2048 264ppi

3㨴

䅰灬攠䄵X

V敲iz潮

iPod

iPod Touch
(4
th

gen)

5

3.5” 640x960 326ppi

2㨳

䅰灬攠䄴

坩䙩

Mini

iPad Mini

6

7” 1024 x 768 162ppi

3㨴

䅰灬攠䄵

A否T

Conclusion

•
Understanding how everything works allows
us to
logically select devices.

•
A large number of permutations can be
covered in few devices

•
If additional coverage is needed additional
devices can be added

•
White Paper :

–
http://www.prototest.com :

–
Building the Ultimate Device Matrix