Speed to Market in Mobile Development

wrendressMobile - Wireless

Nov 12, 2013 (4 years and 1 month ago)

91 views

Speed to Market in
Mobile Development
Finding the right solution with continuous
integration on real devices

Tina Su, Director of Development, Intuit
About me
Nearly 20 years of industry experience in R&D,
quality and engineering management, with a strong
passion for automation.


§

Engineering Director at
Intuit
, Developer Productivity
§

Sr. Engineering Director at
Yahoo
, Mobile Platforms
§

Sr. Engineering Manager at
Sun
, Java Standard and
Mobile Editions
§

Compiler Developer at
MIPS

and

Tandem

Tina Su, Intuit
Agenda
§

Mobile trends & developers’ challenges
§

Speed & quality through continuous integration
-

Intuit Virtual Device Lab
-

Intuit mobile test automation
§

Takeaways
Aggressive mobile momentum
“2013 Internet Trends,” KPCB

Will surpass desktop
by 2014
It should just work!
Proliferation of platforms,
OS versions & device types
Developer challenges
§

Immense configuration matrix
-

OS platforms (Android,
iOS
, Windows, phone vs. tablet, …)
-

Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)
-

Outputs (portrait/landscape, screen resolution, language)
§

Application delivery through the markets
-

App signing, provision (app & device), submission
§

Offline experience
-

Network, different network bandwidth/latency, no-network
-

HTML5 offline app
§

Media challenges
-

Some codecs don’t work on a device
-

Streaming, sound
§

Security & privacy
Intuit mobile app portfolio today
Apple

Google

Amazon

SMS

0

5

10

15

20

490

22

15

11

3

491 mobile apps for financial institutions

Ramping up Intuit’s innovation engine
Pain:
Takes
days per build
to test
combinations of device/OS configurations
physically & manually, one by one …
QA can’t keep up with the changes

Goal
:
Reduce development iteration cycle
from
days to minutes
on an ever-growing
number of devices

Strategy:
Speed & quality through mobile
continuous integration






iPhone 4


iPhone 4S


iPhone 5


iPad
2


iPad
3


iPad
Mini
Devices
OS versions


iOS
5.0


iOS
5.1


iOS
6.0


iOS
6.1.x


iOS
7.0
Builds
X
X
Daily
builds


Apple


Google


Amazon
Platforms
X
Continuous integration (CI)
What is CI?
§

Automate build, test & deployment process
§

Integrate & build early, often, daily, for every check-in
§

Best practice of Agile development
§

Jenkins is widely adopted, flexible & extensible

Benefits of CI in a mobile world
§

Instant feedback on quality, functionality & system impact
§

Less complex, no integration hell
§

Detect and address device configuration issues early
§

Involve stakeholders early
§

Ultimately achieve shippable release any time

Speed and quality
through mobile CI
Key enablers:



Mobile device lab


Test automation
Test processor
Real
devices
Jenkins
SCM:
source
repository
Developers/
IDE
Automatic
test run
Test results
Submit the app to
app store if passed
Check in
Check-in
triggers
automated CI
Install the test app to
real devices in parallel
Run tests on real
devices in parallel
Analyze
test results
Trigger testing in
mobile device lab
Process starts here
3
rd
-party vendor solutions didn’t work for us
§

Couldn’t handle E2E automation of testing pre-release apps
-

Dependency on internal resources & infrastructure

§

Vendor lock-in: customized CI & test automation solutions
-

No flexibility to use our existing tools & frameworks

§

Disadvantages of vendors’ shared devices
-

Security concern on testing on shared devices
-

No access to corporate network
-

Cost-prohibitive to reserve devices

§

Private cloud (on-site setup) with local devices not practical
-

With sensitive customer data, we need private cloud
-

Extensive use of devices for daily testing of hundreds of mobile apps is expensive

Why test on real devices?
Most mobile features can be tested on simulators, but
simulators have limitations:

§

Device capabilities (GPS, camera, accelerometer, gyros),
form factors & pixel density
§

Real-world experience over the real connection
(3G, 4G, crappy Wi-Fi, etc.)
§

True performance, security tests
§

Testing on non-default Web browsers (e.g., Opera Mini)
§

Not all simulator/emulators have good quality
(don’t reflect real rendering/behavior, bugs)
§

Testing SMS & app integration
Intuit Virtual Device Lab
Intuit Virtual Device Lab (VDL)
Gives access to
real mobile devices
using a browser
Intranet
VDL
User machine
User machine
User machine
3-click access
Always on
Integrated with
dev
infrastructure
Globally available
iPhone
iPad

Galaxy
Captivate
Optimus

Nexus
One
Desire
Incredible
Manufacturers/ 
devices:

Droid
Milestone
Moto
VDL architecture – Android
Windows PC
Tomcat
Java image
transferring
application
Browser &
Java applet
Internet/
intranet
Device
interaction app
Developer’s laptop
VDL
Image transfer
USB connection
VDL architecture –
iOS

Mac with
Xcode

installed
Tomcat
Java VNC
client
Port forwarding
Browser &
Java applet
Internet/
intranet
VNC server
USB connection
Developer’s laptop
VDL
Open source tools & technologies used in VDL
Key capabilities
Android
iOS

Installing mobile applications into
the real device
adb
install <APK file>
Fruitstrap

https://github.com/ghughes/
fruitstrap
Transferring device screen images,
controlling device
Screencast tool
http://code.google.com/p/
androidscreencast

Veency
VNC app
Fetching device local files
adb
pull <filename>
sftp
command
(
jailbroken
devices only)
VDL key features & functionality today
Control the device

through standard browser/
RESTful
APIs
Change orientation
to landscape or portrait
Copy local file to the device

from your PC

Get file from the device
onto your local PC
Execute an ADB command
on the device
See current running logs

in your browser
Download history logs
to your local PC

Continuous integration – how it works
Product builds
Trigger file launches test
VDL device
inventory
Resource
manager
Select a device
Test execution
engine
Results
Monitor
for new
builds
Mobile continuous integration
Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.

Submit, build, test … in minutes vs. hours
Select devices & tests
1
Tests run automatically across selected
devices with each build
2
See the results!
3
Next step: over-the-air replaces USB connection

Fast onboarding
Scalable
Any device
Internet/
intranet
App
installation
Test
execution
Results
retrieval
VDL


Mobile Test Automation
Mobile test automation tools
Instrumentation-based
Non-instrumentation
Android
iOS
MonkeyRunner
MOET
Sikuli
Android SDK
Robotium
TestDroid (BitBar)
Calabash
(LessPainful)
Soasta
iOS SDK
UIAutomation

UISpec
Frank
MonkeyTalk
DA/
PerfectoMobile
eggPlant
Cross-platform solution
iOS Driver
Appium


Assessed as of 2012
Mobile test automation trends
§

Abstraction with domain-specific language (DSL)
-

Common test script over different devices (e.g., Cucumber,
MonkeyScript
, MOET)
-

Support behavior-driven development (BDD)
§

Client/server-based approach (remotely driven tests via HTTP)
§

Instrumentation & non-instrumentation solutions complement
each other
§

Mobile cloud testing environment
-

From hardware (cradle) to software approach to drive tests on
real devices
§

Automated OTA app installation & tests
Intuit automation framework of choice/recommendation
§

Native apps:
-

Cucumber + Calabash for BDD on
iOS
& Android

§

Mobile Web:
-

Selenium/
WebDriver

What is Cucumber?
It lets stakeholders (customers, QE or developers) describe how
software should behave in plain text.

§

BDD: Written in business domain-specific language (DSL)
-

Good abstraction over different device types
§

One format for multiple purposes
-

UI feature specification
-

Automated tests
§

Cucumber itself is NOT a test automation tool
-

Add-on to any test automation framework
-

Examples: Calabash,
Sikuli
& Frank mobile test frameworks
What is Calabash?
§

Open source cross-platform mobile test framework
-

iOS + Android, simulator + real device, native + hybrid
§

Supports Cucumber for BDD
§

Application needs instrumentation that embeds a Calabash HTTP server
-

Tests run from a remote client (in a Jenkins slave)
-

Tests can be driven via Wi-Fi (no USB wire)
§

Ruby clients in API level (Java clients in experiment)
§

Works better in touch simulation & real device testing
§

Mobile cloud test execution available by
LessPainful

Intuit shared test library – rapid test development




Internal open source repository
Teams focus on creating
Use shared test lib
… and then contribute to
common test library
Tests
Contribute
Tests
Common test library
Product tests
Tests
Tests
Globalization test framework
(I18N/L10N)

Performance/load test framework

Security test framework

Native/hybrid test framework

High-level test domain-specific language
(Cucumber steps)
Mobile
back-end
Mobile
front-end
Code
build
Deploy
Test
Results
Connect
device
Notify
email
Get
devices
Consolidate
report
Upload
report
From: manual
50*5
= 250

15

30

5
10
20
15
30

10

= 6 hours
Code
build
Trigger
Execute tests
in parallel
Consolidated
report in archive
Deploy on
devices
0

5

10

0

= 15
mins

To: automated CI
TIME SAVED: 95%
Release quality increased
*
Stunning results:
reduced iteration cycle from hours to minutes

Tested on 5 devices
VDL cost saving over vendors’ private clouds (today)
Annual
cost ($)
# of devices
Intuit Virtual Device Lab
150
3
rd
-party vendor 1
local device in cradle
Assumptions:
50 native
apps & daily check-in

Note:
As vendor pricing
changes, so do the savings

Savings
25
3
rd
-party vendor 2
local device w/o cradle
(Software approach)
Savings
Private cloud setup behind
corp
firewall
Key takeaways
Building custom solution vs. 3
rd
-party vendor solution
§

Understand your technical/platform/global spread
§

Many vendor solutions provide a larger set of devices/OS versions
in the global market
§

Quite a few offer mobile test services – might be tightly coupled with their
device access solution & automation framework
§

Connecting to your corporate network is a challenge – some provide
private cloud (on-site setup within your network) but it’s cost-prohibitive


Invest in automation and CI
§

Automate tests with robust automation framework
§

Enable rapid test development through shared test library
§

Adopt CI in mobile development for speed & quality

More mobile innovations at Intuit
§

Mobile crash reporter
§

Mobile components & design patterns
§

Mobile analytics SDK & gateway

Visit booth #601 – Accelerate Development
§

Demos & conversations
§

Contest
§

Mobile device charging station
§

Foosball
§

We’re hiring !
Win a weekend with a
TESLA!

Thank you!!
Tina_Suintuit.com