Jolien Coenraets applications Scenario testing of mobile webview ...

estrapadetubacityMobile - Wireless

Dec 10, 2013 (3 years and 4 months ago)


Jolien Coenraets
Scenario testing of mobile webview-based native
Academiejaar 2011-2012
Faculteit Ingenieurswetenschappen en Architectuur
Voorzitter: prof. dr. ir. Daniël De Zutter
Vakgroep Informatietechnologie
Master in de ingenieurswetenschappen: computerwetenschappen
Masterproef ingediend tot het behalen van de academische graad van
Begeleider: Heïko Desruelle
Promotor: prof. dr. ir. Frank Gielen
This thesis could not have been written without the help of Heko Desruelle,who served
as my supervisor.I would like to thank him for his guidance and coaching during this
year.He helped me focus on the subject at hand and sparked my interest for the academic
world.Secondly,I would like to express my gratitude towards Gielen
for believing in the subject I proposed and allowing me to investigate the matter.I gained
a lot of insight in the subject that will prove useful in my future career.
I would like to thank my colleagues at G- ux,who helped by suggesting new angles for
me to explore.I would also like to thank the Bike To The Moon committee at IBBT,
without them there would not have been a Moonbiker application or a case study.
Further,I would like to thank my ance Jenne for thoroughly proofreading this thesis and
revising where necessary.I am grateful for the endless support I received from him and
my parents during my studies.
Finally,I would like to thank everyone who contributed directly or indirectly to this thesis.
Jolien Coenraets,May 2012
Copyright notice
\The author(s) gives (give) permission to make this master dissertation available for con-
sultation and to copy parts of this master dissertation for personal use.In the case of
any other use,the limitations of the copyright have to be respected,in particular with re-
gard to the obligation to state expressly the source when quoting results from this master
Jolien Coenraets,May 2012
Scenario testing of mobile webview-based
native applications
Master's thesis submitted to obtain the academic degree of
master of computer science engineering
Academic year 2011{2012
Faculty of Engineering and Architecture
Ghent University
Department of Information Technology ZUTTER
The fragmentation of the mobile market is pushing application developers to use web or
webview-based native applications (also known as hybrid applications) instead of normal
native applications.They allow a larger reuse of code since they use web technology
(HTML,CSS and JavaScript) to build an application.Unfortunately,testing web or
webview-based native applications is hardly supported by the existing test solutions.Nev-
ertheless it is very important since it in uences the user experience of the application.
This dissertation tries to create an automated scenario test solution for web and webview-
based native applications.A proof of concept is developed which executes scenario tests
on webview-based Android Cordova applications by using JUnit.This proof of concept is
evaluated in a case study that tests four scenarios on an application of this type.
Scenario testing,webview-based native applications,hybrid applications,Cordova,JUnit
Scenario testing of mobile webview-based native
Jolien Coenraets
Supervisor(s) Gielen,ir.He¨ıko Desruelle
Abstract—This article aims at creating an automated scenario test solu-
tion for web and webview-based native applications.These types of appli-
cations are used more and more because of the increasing fragmentation
of the mobile market,but there are almost no test solutions for them.A
proof of concept is developed which executes scenario tests on webview-
based Android Cordova applications by using JUnit.This proof of concept
is evaluated in a case study that tests four scenarios on an application of this
Keywords—Scenario testing,webview-based native application,hybrid
EVELOPERS of smartphone applications are increasingly
suffering from the mobile market’s fragmentation.Not
only the different operating systems such as Android and iOS
need to be supported,but also their different versions and this
on a growing amount of devices,all with their own hardware
specifications.This fragmentation is especially a problem on
Android.Developing native applications is resource-consuming
since each platform uses a different programming language so
no code can be reused.This causes a shift from native appli-
cations towards web or webview-based native applications (also
known as hybrid applications).Since these applications run in a
browser or webviewand are programmed by using web technol-
ogy (HTML,CSS and JavaScript),the major part of the code can
be reused on the different operating systems.Webview-based
native applications allow to package the web technology as a
native application and thus to distribute them through the app
At the moment,testing these types of applications is a time-
consuming activity since most of the testing needs to be done
manually.The large fragmentation requires the tests to take
place on a large number of devices and thus magnifies the prob-
lem.Automation of this tedious testing task would be an extra
advantage of using web or webview-based native applications.
Section 2 of this article discusses the development of a testing
solution by reviewing the existing tools,envisioning an archi-
tecture and elaborating on the proof of concept.In Section 3 the
proof of concept is used to test an application and a conclusion
is made in Section 4.
There already are some test solutions to test web applications
or JavaScript code,such as Selenium or QUnit.Different W3C
initiatives are dealing with this subject and MobiWebApp is cre-
ating its own test solution for web applications [3].However,in
all the existing solutions webview-based native applications are
overlooked.The combination of a native application and code
written in JavaScript leads to a special situation which makes
the existing solutions inapplicable.Also,these solutions are fo-
cused on unit testing while scenario testing is far more important
to smartphone applications since this tests the behavior of the
application when the user is interacting with it [4].Since smart-
phone applications behave differently in different situations (e.g.
with or without Internet connection) it is important that the ap-
plication is tested in all these situations.
The elaboration of the architecture was done by using the
attribute-driven-design technique [5] and with usability,exten-
sibility and scalability as most important quality attributes.Dur-
ing the first iteration,the Broker pattern was applied which lead
to the architecture in Figure 1 [6].The left side shows the com-
ponents that interact with the user (the website and the command
line) and the right side contains the components that execute the
tests.The device connector connects the different devices to the
test systemand executes the tests on them,the scheduler sched-
ules the tests and the processor processes the results.The inter-
mediate component provides the communication between both
During the elaboration of the different components,the device
connector showed up as being the most challenging one.The au-
tomated executing of tests on webview-based native applications
in particular seemed to be a challenge.The partly native appli-
cations could be tested by using a native testing framework,but
only if the JavaScript code of the application could be accessed
fromwithin the test.It was decided to create a proof of concept
for the device connector which could be integrated into a larger
test solution later on.
B.Proof of concept
The proof of concept tests Cordova applications (also known
as PhoneGap applications [7]) on Android devices by using JU-
nit [8].Cordova was chosen since this is a widely used open
source framework to create webview-based native applications.
The proof of concept could not address every operating system
within the time limit.Android was chosen for its open source
character but also for its integration with JUnit.Android has ex-
tended JUnit with quite some features that alleviate the testing
of Android applications,e.g.manipulation of the application’s
life cycle.JUnit was meant for unit testing but can be used for
scenario testing as well by looking at a scenario as a unit.
Cordova creates a bridge from JavaScript to Java and back
but for the proof of concept the other direction is needed since
the application needs to be manipulated fromwithin a JUnit test
which is written in Java.This is implemented by creating a new
class which can be reached from the activity of the application,
Fig.1.Overview of the architecture
which in turn can be reached from the JUnit test project.The
Cordova plugin mechanism is not used since this is designed to
bridge the gap from JavaScript to Java and not the other way
around.The proof of concept succeeds in manipulating the
HTML elements within the application such as firing events on
it (clicking buttons),or adapting or requesting the content,prop-
erties or attributes of the element.The proof of concept allows
sending commands to the device or to the Android emulator.
The native buttons,such as menu and back,can be controlled as
well on the device as on the emulator.Other functionality only
targets the emulator such as adapting the environment by chang-
ing the battery status or the voice connection.By making use of
the emulator,the user’s location or phone calls can be mimicked
The development of the proof of concept revealed several
problems.At the moment,it is not possible to handle pop-
up dialogs since the test project can not access the object that
is representing these,which means that the buttons can not be
clicked.No screenshots of the device can be taken,only screen-
shots of the application itself.This may misrepresent the appli-
cation since it does not show what is actually happening on the
device screen.Another screen might be on top,hiding the appli-
cation under test.Once the application is brought to the back-
ground,there is no way of bringing it back to the front.This
may cause problems if native buttons like menu are used since
these are executed on the application on the foreground.Files
that are needed or created during the tests can not be pushed to
or pulled from the device automatically.This needs to happen
manually before or after running the tests.The same applies to
the collection of artifacts such as screenshots or log messages.
Because of these problems the proof of concept can only be
applied to a limited amount of applications or scenarios.These
problems have to be solved before this solution can be integrated
into a larger test solution.
The case study tests the Moonbiker application of smartphone
application company G-flux [9].The Moonbiker application
tracks the route of a user while cycling,registers the cycled
distance and sends this result to the website to generate rank-
ings and statistics.This application is created with Cordova and
intensively uses the GPS technology.The application behaves
differently in different situations.If there is no Internet connec-
tion,for example,the application will save the data to send it
at a later time when there is a connection.This combination
of GPS technology,a webview-based native application and the
different actions in different situations makes it very hard to test
this application.An automated test solution would be a great
Fig.2.Example of an error that can be detected by checking an HTML field
improvement for G-flux.
Four scenarios were tested during the case study,two of them
make use of the GPS technology (while cycling) and each sce-
nario reacts in a different way in case there is an Internet con-
nection or not.One of these scenarios contains accepting an
incoming call while cycling.In the end only two out of the four
scenarios could be tested because of the limitations or problems
of the proof of concept which were mentioned before.Still,sev-
eral bugs could be found and solved.G-flux mentioned that they
could not have found these bugs without the proof of concept,
or it would have taken a lot of time spent at searching and de-
Figure 2 shows an example of a bug that can be detected by
checking an HTML field value in this application.It should not
be possible to cycle a negative amount of kilometers.
The proof of concept proves that native testing solutions can
be used to test webview-based native applications.Besides it
shows that the scenario tests can be executed in different situa-
tions which alleviates the testing effort and improves the quality
of the applications.Unfortunately the proof of concept is still
a work in progress since the problems discussed impose large
limitations on the types of applications or scenarios that can
be tested with it.Once these problems are solved this solution
could be integrated in an automated scenario test solution.
[1] AdamM.Christ,“Bridging the mobile app gap,” Sigma:Inside the digital
ecosystem,vol.11,pp.27–32,October 2011.
[2] Maximiliano Firtman,Programming the Mobile Web,O’Reilly,first edition
[3] W3C,“MobiWebApp Test Suites Report Year 1,” http:
testing_report-year1.pdf,September 2011.
[4] Cem Kraner,“An introduction to scenario testing,” http://www.,June 2003.
[5] Software Engineering Institute,“Attribute-Driven-Design method,”
[6] Len Bass,Paul Clements,and Rick Kazman,Software Architecture in Prac-
tice,Second Edition,Addison-Wesley Professional,2003.
[7] Cordova,“Apache project website,” http://incubator.apache.
[8] JUnit,“Project website,”
[9] G-flux,“Company website,”
Scenario-testing van mobiele webview-gebaseerde
native applicaties
Jolien Coenraets
Supervisor(s) Gielen,ir.He¨ıko Desruelle
Abstract—In dit artikel wordt geprobeerd een geautomatiseerde scena-
rio test oplossing voor web en webview-gebaseerde native applicaties te
cre¨eren.Door de groeiende fragmentatie van de mobiele markt worden
deze applicaties meer en meer gebruikt.Nochtans zijn ze erg moeilijk te tes-
ten.Er wordt een proof of concept oplossing ontwikkeld die scenario tests
uitvoert op webview-gebaseerde Android Cordova applicaties door gebruik
te maken van JUnit.Deze proof of concept oplossing wordt ge¨evalueerd in
een case study waarbij vier test scenario’s op een dergelijke applicatie wor-
den uitgevoerd.
Keywords—Scenario testing,webview-gebasseerde native applicaties,hy-
brid applicaties,Cordova,JUnit
MARTPHONE applicatie ontwikkelaars hebben steeds
meer te lijden onder de fragmentatie van de markt.Niet
alleen verschillende besturingssystemen,zoals Android en iOS,
moeten ondersteund worden,maar ook hun verschillende ver-
sies en dit op een erg groot aantal toestellen met elk hun eigen
hardwarespecificaties.Vooral bij Android is deze fragmenta-
tie een probleem.Native applicaties ontwikkelen vereist vol-
doende tijd en mankracht aangezien elk besturingssysteem een
andere programmeertaal gebruikt en code dus niet hergebruikt
kan worden.Daardoor schakelen steeds meer ontwikkelaars
over op web of webview-gebaseerde native applicaties (ook ge-
kend als hybrid applicaties).Aangezien deze applicaties in de
browser of webview draaien en met web technologie (HTML,
CSS en JavaScript) ontwikkeld worden kan de code voor een
groot deel hergebruikt worden in de verschillende besturings-
systemen.Webview-gebaseerde native applicaties laten toe om
deze web technologie toch als native te verpakken en zo ook in
de app stores te plaatsen.[1],[2]
Het testen van deze applicaties is momenteel een tijdrovende
bezigheid aangezien het meeste manueel moet gebeuren.De
grote fragmentatie zorgt er voor dat de tests uitgevoerd moeten
worden op een groot aantal devices,wat het probleemenkel gro-
ter maakt.Indien dit testen geautomatiseerd kan worden zou dit
een extra voordeel zijn om web en webview-gebaseerde native
applicaties te gebruiken.
Sectie 2 van dit artikel bespreekt de ontwikkeling van een
test oplossing door eerst de bestaande oplossingen te bespreken,
daarna een architectuur te belichten en uiteindelijk de proof of
concept oplossing te bespreken.In sectie 3 wordt ont-
wikkelde proof of concept oplossing een bestaande applicatie
getest en daarna volgt een conclusie.
Er bestaan reeds verschillende test oplossingen omJavaScript
code of web applicaties te testen,zoals QUnit of Selenium.
Er bestaan verschillende W3C initiatieven rond dit onderwerp
en MobiWebApp is bezig een test platform voor web appli-
caties te ontwikkelen [3].Bij alle bestaande test oplossingen
worden webview-gebaseerde native applicaties echter over het
hoofd gezien.De combinatie van een native applicatie met code
die in JavaScript geschreven is resulteert in een speciale situa-
tie die overal uit de boot valt.Daarnaast ondersteunen de be-
staande oplossingen vooral unit-testing waar smartphone appli-
caties scenario-testing vereisen aangezien dit de scenario’s test
die gebruikers zullen uitvoeren met de applicatie [4].Aangezien
smartphone applicaties dikwijls op een andere manier reageren
in andere omstandigheden (bv.wel of geen internet) wordt de
applicatie best ook getest in al deze verschillende situaties.
De uitwerking van de architectuur gebeurde volgens de
attribute-driven-design techniek [5],waarbij als belangrijkste
quality attributes gebruiksvriendelijkheid,uitbreidbaarheid en
schaalbaarheid werden gekozen.Tijdens de eerste iteratie werd
het Broker patroon toegepast en werd de architectuur uit Figuur
1 bekomen [6].De linkerzijde bevat de interfaces naar de ge-
bruiker toe (website en command line) en aan de rechterkant
bevinden zich de componenten die de tests uitvoeren.De device
connector bevat de verbinding met de toestellen en voert de tests
uit,de scheduler plant de tests op de verschillende toestellen en
de processor verwerkt de resultaten.De intermediate component
zorgt voor de communicatie tussen alle andere componenten.
Tijdens de uitwerking van de verschillende componenten
kwam de device connector naar voor als zijnde de grootste uit-
daging.Vooral het automatisch uitvoeren van tests op webview-
gebaseerde native applicaties leek een uitdaging.Via native tes-
ting oplossingen zouden testen geautomatiseerd kunnen worden,
maar dan moeten die aan de JavaScript code van de applicatie
kunnen.Er werd beslist omeen proof of concept voor de device
connector te maken dat later eventueel kan ge¨ıntegreerd worden
in een grotere test oplossing.
B.Proof of concept
De proof of concept oplossing werd gemaakt om Cordova
(vroeger gekend als PhoneGap [7]) applicaties te testen op An-
droid door gebruik te maken van JUnit [8].Er werd gekozen
voor Cordova applicaties aangezien dit een veel gebruikt frame-
work is omwebview-gebaseerde native applicaties te ontwikke-
len en omdat de source code vrij beschikbaar is.Niet alle opera-
ting systemen konden uitgewerkt worden binnen de tijdslimiet
en er werd gekozen omte beginnen met ondersteuning voor An-
droid applicaties.Dit omdat Android open source is maar ook
omdat Android JUnit uitgebreid heeft om het testen van de ap-
plicaties nog gemakkelijker te maken.JUnit is bestemd voor
Fig.1.Overzicht van de architectuur
unit-testing maar kan gebruikt worden voor scenario-testing
door elk scenario als een unit te zien.Door gebruik te maken van
JUnit kan er gebruik gemaakt worden van de reeds voorziene
functies omAndroid applicaties te testen,zoals het be¨ınvloeden
van de levenscyclus van een applicatie.
Cordova legt reeds een brug van JavaScript naar Java en te-
rug,maar voor deze proof of concept oplossing was de andere
richting nodig aangezien de applicaties vanuit JUnit be¨ınvloed
moeten kunnen worden en dus vanuit de Java code.Dit werd
ge¨ımplementeerd door een nieuwe klasse te maken die kan be-
reikt worden via de activity van de applicatie,die op zijn beurt
bereikbaar is vanuit een JUnit test project.Het Cordova plu-
gin mechanisme werd niet gebruikt aangezien dit ontworpen is
om vanuit JavaScript naar Java te gaan en niet voor de andere
richting.De proof of concept oplossing slaagt er in om HTML
elementen in de applicatie te manipuleren,zoals het uitvoeren
van events (klikken op een knop),of het aanpassen of opvragen
van inhoud,eigenschappen of attributen.De oplossing maakt
het ook mogelijk om commando’s naar een smartphone of naar
de Android emulator te sturen.Zo kan de menu knop zowel op
een toestel als op de emulator bestuurd worden.Het aanpassen
van de omgeving is enkel mogelijk via de emulator die naast het
aanpassen van bv.batterijstatus en internet- of telefoonverbin-
ding ook locaties en telefoonoproepen kan simuleren.
De ontwikkeling van deze proof of concept oplossing bracht
ook enkele problemen naar boven.Zo is het op dit moment niet
mogelijk om pop-up dialoogvensters te manipuleren aangezien
het test project niet aan dit object kan en er dus ook geen ac-
ties op kan uitvoeren.Verder kunnen geen screenshots van het
volledige toestel genomen worden,maar enkel van de applicatie
zelf.Dit kan een verkeerd beeld geven aangezien dit niet weer-
geeft wat er echt op het scherm van het toestel staat.Eens de
applicatie naar de achtergrond verdwenen is kan ze niet naar de
voorgrond gebracht worden,wat problemen geeft indien knop-
pen zoals menu en back gebruikt worden.Daarnaast kunnen
bestanden niet in de test zelf van en naar het toestel verplaatst
worden maar moet dit vooraf of achteraf en manueel gebeuren.
Hetzelfde geldt voor het verzamelen van alle artefacten zoals
screenshots en log berichten.
De bestaande problemen zorgen ervoor dat de proof of con-
cept oplossing slechts voor een beperkt aantal applicaties of
scenario’s kan gebruikt worden.Vooraleer deze oplossing
ge¨ıntegreerd kan worden in een groter geheel zullen deze pro-
blemen opgelost moeten worden.
Voor de case study werd de Moonbiker applicatie van
smartphone applicatie bedrijf G-flux getest [9].Deze applica-
tie neemt de route van een fietsende gebruiker op,registreert
het aantal gefietste kilometers en stuurt dit door naar de web-
site om zo allerlei rangschikkingen bij te houden.Deze appli-
catie is gemaakt met Cordova en maakt intensief gebruik van
GPS technologie.Daarnaast gedraagt de applicatie zich anders
indien er geen internet verbinding is.In dat geval gaat de appli-
catie alles opslaan om op een later tijdstip de gegevens door te
sturen.Deze combinatie van gebruik van GPS technologie,een
webview-gebaseerde native applicatie en de verschillende acties
in vari¨erende omgevingen maakt het erg moeilijk omdeze appli-
catie te testen.Een geautomatiseerde test oplossing zou G-flux
zeker vooruit kunnen helpen.
In de case study werden vier scenario’s getest waarvan twee
gebruik maakten van GPS locatie (tijdens het fietsen) en elk sce-
nario anders reageert in het geval er wel of geen internet ver-
binding is.Tijdens ´e´en van de scenario’s werd een telefoonop-
roep ontvangen tijdens het fietsen.Uiteindelijk konden slechts
twee van de vier scenario’s getest worden wegens bovenver-
melde problemen met de proof of concept oplossing zelf,maar
werden er toch verschillende bugs gevonden en opgelost.G-flux
liet weten dat zij deze bugs zonder de proof of concept oplossing
niet of pas veel later gevonden zouden hebben,waarschijnlijk na
urenlang zoeken en debuggen.
De proof of concept oplossing toont aan dat het gebruik
van een native testing oplossing voor het testen van webview-
gebaseerde native applicaties mogelijk is.Daarnaast wordt ook
aangetoond dat scenario tests in vari¨erende situaties uitgevoerd
kunnen worden,wat het testen van deze applicaties sterk zal ver-
gemakkelijken en de kwaliteit verbeteren.De proof of concept
oplossing is echter nog geen kant-en-klaar product.Er zijn nog
enkele essenti
ele problemen die moeten worden opgelost voor-
aleer deze oplossing op grote schaal gebruikt kan worden.Eens
deze problemen van de baan zijn kan er gekeken worden om de
proof of concept oplossing te integreren in een geautomatiseerde
test oplossing.
[1] AdamM.Christ,“Bridging the mobile app gap,” Sigma:Inside the digital
ecosystem,vol.11,pp.27–32,October 2011.
[2] Maximiliano Firtman,Programming the Mobile Web,O’Reilly,first edition
[3] W3C,“MobiWebApp Test Suites Report Year 1,” http:
testing_report-year1.pdf,September 2011.
[4] Cem Kraner,“An introduction to scenario testing,” http://www.,June 2003.
[5] Software Engineering Institute,“Attribute-Driven-Design method,”
[6] Len Bass,Paul Clements,and Rick Kazman,Software Architecture in Prac-
tice,Second Edition,Addison-Wesley Professional,2003.
[7] Cordova,“Apache project website,” http://incubator.apache.
[8] JUnit,“Project website,”
[9] G-flux,“Company website,”
Table of contents ii
Glossary iii
1 Introduction 1
1.1 Context......................................1
1.2 Goal........................................1
1.3 Overview.....................................2
2 State of the art 3
2.1 The mobile ecosystem..............................3
2.2 Technologies....................................13
2.3 Testing and debugging mobile applications...................22
2.4 Working groups and projects..........................30
2.5 Conclusion....................................36
3 Design of the test framework 38
3.1 Vision.......................................38
3.2 NABC.......................................41
3.3 Scenarios.....................................42
3.4 Architecture....................................47
3.5 Conclusion....................................50
4 Technical research 51
4.1 Cordova......................................51
4.2 JUnit for Android................................59
4.3 Android emulator.................................61
4.4 Weinre......................................62
5 Proof of concept 63
5.1 Specications...................................63
5.2 Implementation..................................64
5.3 Usage.......................................76
5.4 Encountered problems..............................77
5.5 Conclusion....................................81
6 Case study 82
6.1 Situation of the case study............................82
6.2 Test scenarios...................................85
6.3 Implementation of the test scenarios......................87
6.4 Evaluation.....................................92
6.5 Conclusion....................................94
7 Conclusions 95
7.1 Future work...................................95
A Scenarios 97
A.1 Terminology....................................97
A.2 Quality attribute scenarios............................97
A.3 Functional scenarios...............................101
B Screenshots Moonbiker application 104
Bibliography 108
List of Figures 110
List of Tables 111
Glossary iii
Android Debug Bridge |command line tool to communicate with an Android device
or emulator.72,77
Application Programming Interface |a specication that describes the functionality
of a software component and how the functionality should be used.16,17,22
app store
application store | an electronic market that contains all the applications for a
certain OS.10,13
continuous integration
automated build process that aims to improve the quality of the software.Often
includes an automated build environment,automated test runs and deployment and
a code repository.25,37,40,58,96
Document Object Model | a platform and language-neutral interface that allows
dynamic access and updates of the content,structure and style of an HTML docu-
GPS Exchange Format |an xml format used to save geographical data like routes,
tracks and waypoints.74,84
Glossary iv
hybrid application
another name for a webview-based native application.10
Keyhole Markup Language |an xml format used to save geographical data,mostly
used by Google.84
native application
an application that has been developed for use on a particular mobile platform.10
Operating System | in this dissertation refers to a mobile operating system such
as Android,iOS or Windows Phone 7.7
scenario testing
a software testing activity that focuses on testing specic use cases from the user's
point of view.23,29
test runner
responsible for running a series of tests and gathering results for all of them.32
World Wide Web Consortium | an international community that develops open
standards to ensure the long-term growth of the Web.12
Wireless Application Protocol |mobile wireless network protocol that was used for
mobile internet on the rst mobile phones.3
weakly typed
used in programming languages when variables are generic types rather than specic
types such as String,boolean,etc.17
web application
an application that runs inside a browser.For this dissertation web-applications will
be limited to mobile web-applications (accessed on a mobile device).10
Glossary v
an interface denition language that can be used to describe interfaces that are
intended to be implemented in web browsers.31
web technology
in this dissertation refers to the combination of HTML,CSS and JavaScript.10,12
web widget
a small application that can be installed and executed in a web browser by an end
a native class that renders web pages and can be embedded inside a native application
to show contents created with web technology.10,19
webview-based native application
an application that is written in web technology but is deployed and executed as a
native application.Often called hybrid app.10
a software development methodology which is intended to improve software quality
and responsiveness to changing customer requirements.The full name is Extreme
Chapter 1
1.1 Context
The fragmentation problem on mobile devices is becoming larger every day.This makes
testing mobile applications very resource-consuming since they have to be tested on as
much targeted devices as possible.It is especially hard to test web or webview-based
native applications,since these are created with web technology and cannot make use of
the native test solutions.A lot of mobile applications act dierently in varying situations
such as with or without Internet connection.Simulating these dierent situations is not
included in any automated test solution at this moment,but will denitely help developers
to improve the quality of the applications.
1.2 Goal
The goal of this dissertation is to create a scenario testing solution for web and webview-
based native applications that is able to performthe tests in dierent environments.Later
on this will be rened to creating a proof of concept for the most challenging part of this
solution:automate the execution of scenario tests for webview-based native applications
by using the native testing framework.
1.3 Overview 2
1.3 Overview
Chapter 2:State of the art
This chapter includes an overview of the mobile ecosystem,the dierent types of mobile
applications,how they are tested nowadays and the work in progress done by the W3C.
Chapter 3:Design of the test framework
The design of the test framework includes a vision,NABC,functional and non-functional
requirements and an overview of the architecture.
Chapter 4:Technical research
To start developing the proof of concept,some more profound research had to be done,
which is described in this chapter.
Chapter 5:Proof of concept
This chapter describes the proof of concept that has been developed for this dissertation.
It elaborates on the implementation and the problems that were encountered during the
Chapter 6:Case study
The proof of concept is used to test G- ux's Moonbiker application.Based on this case
study,an evaluation of the proof of concept could be made.
Chapter 7:Conclusions and future work
This dissertation is concluded with the evaluation of this proof of concept and the future
work to be done to improve it.
Chapter 2
State of the art
This chapter will describe the latest changes in the mobile industry,explain some old
and new technologies that are used to create smartphone applications and discuss several
tools that can be used to test them.Problems that are encountered when testing these
applications will be discussed,together with possible solutions.
2.1 The mobile ecosystem
Before the details of mobile applications are explained,an overview of the environments
in which they will be used and tested needs to be given.The article`The what,why
and how of mobile applications'by Daniel Y.Na [1] gives a clear overview of this mobile
ecosystem,just as Chapter 1 (The Mobile Jungle) of the book`Programming the Mobile
Web'by Maximilano Firtman [2].
2.1.1 The history of mobile devices
The history of mobile devices starts in 1987 with the introduction of GSM,the rst mobile
phone standard that has found its place in the market.25 years later the mobile ecosystem
has changed a lot,with the introduction of the smartphone and recently the tablet.
Mobile phones had little functionality in the beginning:they were meant for calling,that
was it.Gradually features were added,such as vibration mode,sms,bluetooth and WAP.
2.1 The mobile ecosystem 4
Together with the increased number of features,the number of users increased.Where it
was rst only for businessmen,in the end even 6 year old children had one.
Almost at the same time as mobile phones were released,the Personal Digital Assistant
(better known as PDA) was launched.These PDAs were like pocket computers.They had
touch screens and memory cards and could be connected to a computer to synchronize
the data.PDAs had useful applications like text editors and spreadsheet programs.
Around 1996,Nokia thought that it would be better if these two devices would be merged
into one and the smartphone was born.Smartphones have changed a lot since then and
they now have GPS-functionality,mobile Internet,e-mail-functionality,lots of applications
and lots of memory capacity.Just like mobile phones and PDAs,these devices were rst
only for business people,but since a couple of years they are used by a broad range of
people,from children to elderly.
Another type of mobile device is the tablet.Tablets are quite old,the rst one was released
in the fties and during the nineties Apple released the Apple Newton which never became
a success.Tablets only recently boomed (2010),with the introduction of multi-touch,the
thinner screens and the launch of the iPad by Apple.They are characterized by their
bigger screen sizes and their focus on documents and browsing.
In this dissertation mobile devices will be limited to smartphones and tablets,since a
network connection and mobile browser are needed to use web applications.These func-
tionalities are often not available on other mobile devices.
2.1.2 Mobile brands and operating systems
There is a big variety of players on the mobile market.It is not clear yet who will come
out on top.The mobile market is changing as we speak.Only one thing is certain:the
smartphone market is booming.Gartner studies show that smartphone sales grew with
47% during the fourth quarter of 2011 and are expected to grow with 39% during 2012
2.1 The mobile ecosystem 5
Figure 2.1:Worldwide smartphone sales to end users by operating system (in millions of units
sold),by Gartner [4,5,6,3]
Smartphone sales
Smartphone sales more than doubled within two years,as visualized by Figure 2.1.Where
smartphones used to be for business men only,teens and adolescents are using them too
now,which leads to a big growth in sales.The good sales records are due to the popularity
of Apple iOS and Android.Android introduced budget smartphones while Apple has a
loyal community behind their iPhone.There is a peak in smartphone sales during the
fourth quarter of each year due to the popularity of smartphones as a gift during the
Operating systems
There was a shift in the leading operating system over the last two years.Figure 2.2
depicts a substantial growth of Android,who released their rst smartphone only in
2008.Gartner expected in the beginning of 2010 that Android would be the no.1
operating system by 2014 with a market share of 29.6 % [7].They already reached this
share before the end of 2010.
2.1 The mobile ecosystem 6
Figure 2.2:Worldwide smartphone sales to end users by operating system (in % market share),
by Gartner [4,5,6,3]
Android's success is due to its open source character.Vendors are willing to use it because
they can adapt it to their own wishes but still keep the similar Android interface towards
the users.The open-source approach also brings disadvantages for developers:it is hard to
know if the application will work the same on all Android devices,just because each vendor
can make changes in the operating system.There also is a large variety of screen sizes and
hardware specications.By using the Android Compatibility Test Suite,automatic tests
created with JUnit,which will be described in more detail in Sections 2.3.3 and 4.2,can
be run on dierent devices to see whether the application is compatible with all of them.
These devices need to be physically attached to the system,which leads to a lot of devices
that have to be purchased and thereby to an increasing development cost.These dierent
reasons resulted in a small drop in their market share at the end of 2011,as shown on
Figure 2.2.
iOS is the operating system Apple uses for its iPhone,iPad and iPod.It is as popular as
these devices since they are the only devices that use this operating system.Figure 2.2
shows that iOS's market share is slowly increasing,due to the fact that they only have
one smartphone they sell.During the fourth quarter of 2011,iOS took a part of Android's
share with the release of the iPhone 4S.
2.1 The mobile ecosystem 7
The fact that iOS is only used on one type of smartphone certainly has advantages.A
developer knows which screen resolutions and sizes are available and on which hardware
specications the application should work.Apple also oers a great backward compati-
bility so an older device can still benet from the latest novelties in the operating system.
On the other hand developers depend entirely on the whims of Apple.Apple checks all
applications that pass through their application store (which is the only way to install an
application),and this process often takes one week or even more.
The other operating systems are losing market share to Android and iOS.Symbian is
the worst aected by the arrival of Android.It is the operating system that Nokia used
for its phones before they started using Meego,an open-source mobile operating system
they still use for some devices but which has already been replaced by Tizen.Recently,at
the end of 2011,they started to release smartphones that use Windows Phone.Symbian's
market share will denitely get smaller by the years,and possibly Windows Phone will
take the leftover parts.Symbian was one of the rst operating systems used on mobile
phones,but did not evolve when the other platforms did.This gave the operating system
a bad image.
BlackBerry OS is the operating systemused for BlackBerry devices,which are produced
by RIM (Research In Motion).BlackBerry is not as popular any more as it was before.
It used to be the no.1 for business people,but RIM perceives lots of competition from
Android devices and iPhones.Opinions on their future are divided.Some people think
they will gain market share,others think they won't.
Microsoft's share has declined over the years but with their new operating system Win-
dows Phone 7,this could change.The rst phones are being sold at the moment of writing
so it is too early to make predictions.The future will tell how this operation system will
Fragmentation of the Android market
The Android platform is characterized by its fragmentation.There are several types of
fragmentation:by platform version,by device model and by OS version.The dierence
between platform version and OS version is that Android releases the platform versions,
but everyone can adapt these versions and hence create a separate OS version.For example
2.1 The mobile ecosystem 8
Figure 2.3:Android platform distribution on May 1,2012 by Android
Figure 2.4:Android platform historical distribution on May 1,2012 by Android
Figure 2.5:Android device fragmentation as experienced by TweetDeck.Each color represents
a dierent device.
2.1 The mobile ecosystem 9
Samsung,HTC,Motorola and Sony all have Android 4.0 phones (same platform) with their
own adapted version of Android 4.0,leading to four dierent OS versions for one platform
Figure 2.3
shows the platformfragmentation as it was on May 1,2012.Android 2.3.3 has
almost 2/3
of the share but the other versions are taking their part too.As a developer
it is very hard to support all these versions,let alone that the application could be tested
on all of them.A big part of the problem are the device manufacturers that don't update
the OS of their sold devices.Chances are small that a device with Android version 2.2
will receive an update to 4.0.A new version can only be obtained by buying a new phone,
which most people don't when their phone is only one or two years old.This leads to a
lot of old versions being in circulation.This in contrast to Apple which makes it possible
for devices that are several years old to be updated to the most recent OS version.Figure
depicts a historical chart of the Android platform fragmentation.
A second source is the device fragmentation.There are lots of dierent Android devices
on the market,each having their own characteristics such as memory,processor,gps
antenna,screen,presence of a hardware keyboard and many more.This leads to a dierent
performance on each device and in uences the user experience.While some users think the
app is super fast,others may nd it slow.For location enabled applications,the quality
of the application depends on the quality of the GPS signal and hence also on the quality
of the GPS chip.Dierent screen sizes force developers to use responsive user interfaces.
The device fragmentation as experienced by the TweetDeck beta app in 2010 is shown on
Figure 2.5
The last form of fragmentation is the OS version.Since it is hard to create a bug free
operating system,each OS version solves some platform bugs but also introduces new
ones.Besides the versions of the brands,people can create their own Android OS version
starting from the Android release or download one from specialized websites such as XDA
.A developer can not assume that a certain type of device will have a certain
Graph taken from
Graph taken from
Graph taken from
2.1 The mobile ecosystem 10
OS version since the user may have installed another one himself.
2.1.3 Smartphone and tablet applications
One of the characteristics that distinguishes smartphones (or tablets) from mobile phones
in general is the ability to install extra applications on the smartphone or tablet.These
applications can be downloaded from the Internet via the app stores of the dierent plat-
There are generally three dierent types of applications for smartphones and tablets:
native applications,web applications and webview-based native applications.
Native applications are applications that have been developed for use on a particular
platform or device.For example when developing for iOS this is done in Objective C,
while developing for Android is done in Java.Table 2.1 shows the language that each
platform uses.
Web applications are websites that represent mobile applications in which some or all
parts are downloaded from the web each time they run.These applications are written
with web technology,such as HTML,CSS and JavaScript.
Webview-based native applications are applications that are written with web tech-
nology but are deployed and executed as native applications.These native applications
contain a renderer (called a webview) with a web page that contains the functionality.
Most of the times the application does not need to download parts from the Internet,
although this is still possible.These applications are often called hybrid applications,
but the term webview-based native applications is preferred since this better covers the
Comparison between native,web and webview-based native applications
These tree types of applications all have their advantages and disadvantages.Before
starting with the development of a mobile application it is important to choose the right
application type.Choosing the wrong one can give unneeded extra work or problems.
There is no application type that is always the right one.
2.1 The mobile ecosystem 11
Programming lan-
Platforms it can be used on
Java,portions can be
in C or C++
Only Android (because of Dalvik
Only Symbian
Only iOS
Only BlackBerry (because of RIM
Windows Phone
Only Windows Phone (not even on
Windows Mobile,the predecessor)
Table 2.1:Programming languages on dierent platforms [8]
Figure 2.6:Development of native (left),webview-based native (middle,called hybrid) and web
applications (right).Source:[8]
2.1 The mobile ecosystem 12
Table 2.2 indicates that native applications have the best performance and can use the
most functionality.Their big disadvantage is the development eort if the application
needs to support multiple platforms,which is also shown on Figure 2.6.Each platform
has its own programming language,which is shown in Table 2.1.It is hard for a developer
to master all these languages and APIs,so developing a native application for several
platforms requires a bigger programming eort.This problem can be solved with web or
webview-based native applications since they use web technology to create the applications
(Table 2.3 and 2.4).Only the web technology languages (HTML,CSS and JavaScript) have
to be learned and all platforms can be supported.In webview-based native applications a
small part of the code base has to be platform specic and thus in the native language.
Native applications can make use of the native GUI elements,which is not possible for web
applications and only possible in a limited way for webview-based native applications.A
native user interface is very recognizable for the user and leads to a better user experience.
In a web or webview-based native application the user interface is created in HTML.There
are frameworks to create a similar interface as the native ones,such as Sencha Touch,but
the dierence is often obvious.It is not only about the user interface itself,but also about
the interaction with the operating system.Each platform has its own way of interacting
with the user.Android for example comes with a back button embedded in the OS or in
the hardware so it does not need a back button in the GUI.iOS does not have this back
button since it expects it to be programmed in the user interface of the application itself.
A similar dierence is the menu button which is made available by Android,but that
should be implemented inside an iOS application.Developers can make their own choice
between ignoring the native UI habits and create one user interface for all platforms,or
creating a dierent interface for each of them.
Most browser functionality is standardized by W3C,which means that it needs to go
through their standardization process
.First,W3C receives a submission which they
turn into a note that is published for discussion.If the discussion on the note shows
that this is a valid request,a working group is created which,after a while,publishes a
working draft that contains the work in progress.Sometimes a candidate recommendation
is published too.Once the working draft reaches its nal stage,it is turned into a proposed
recommendation which is still work in progress but will not change very much.The nal
2.2 Technologies 13
step is the W3C recommendation that should be implemented by all browsers since this is
a standard now.This process often takes a lot of time (several years) and therefore,some
browsers already implement a working draft while others don't,leading to fragmentation
problems.Webview-based native platforms like Cordova (see Section 2.2.2) take care of
these problems by implementing working drafts on each platform.Native applications
can use the functionality that is supported in a new platform version right away,without
waiting for any standardization between the dierent platforms.
Web applications purely use the browser's API,so they are limited in functionalities.This
can be solved with webview-based native applications where the core functionalities,such
as the menu or back button on Android,can be accessed too.This requires native code
so these extensions have to be written for each platform.
Since web applications and webview-based native applications run in a browser they have
an extra layer that needs to process all the commands,resulting in a lower performance.
In some browsers JavaScript is not optimized,which is often noticed by the user and leads
to a bad user experience.This in contrary to native code,which is usually optimized,
specically for heavy operations or monitoring operations.
Native and webview-based native applications can be downloaded froman app store,which
makes the application available for everyone who wants to use it.The procedure to upload
an application is sometimes cumbersome and the application is not readily available on
the market | it can take up to one week or more for iOS.A web application can not
be downloaded from a store,a user needs the right link to nd the application.Since
it is just a website,it depends on the developer itself when it becomes available.A
web application is a lot easier for applications that are only used for several days,like
promotional applications or conference apps.
A more in depth analysis of this comparison can be found in [8] or [9].
2.2 Technologies
2.2.1 Languages
The term web technology indicates the combination of HTML (structure and content),
CSS (presentation) and JavaScript (behavior).It is a very powerful combination that is
2.2 Technologies 14
Native applications
 Access to all core functionality
 Optimized performance
 Directly use new functionality
 Native user interface
 Use other applications
 Downloadable from the app
 Every platform has its own
Table 2.2:Advantages and disadvantages of developing native applications [8]
Web applications
 Same language for all plat-
 No download needed
 No core functionality (only
browser functions)
 No optimized performance
 Wait for standardization be-
fore using new features
 Web interface
Table 2.3:Advantages and disadvantages of developing web applications [8]
2.2 Technologies 15
Webview-based native applications
 Same language for all plat-
 Access to all core functionality
 Use other applications
 Downloadable from the app
 Small parts still platform de-
 No optimized performance
 Wait for standardization be-
fore using new features
 Web interface
Table 2.4:Advantages and disadvantages of developing webview-based native applications [8]
gaining popularity.The best example is the Metro interface of Windows 8,which relies
on the power of this combination.
HTML (HyperText Markup Language) is the main markup language for web pages.HTML
elements are the basic building blocks of web pages and web applications.HTML has an
XML structure and consists of tags (like <html>).These tags represent the building
blocks and elements,which can have several attributes and/or child elements.The HTML
standard is very important to the Web,since every web page uses this language to show
its content.
The rst version of HTML was proposed by Tim-Berners Lee in 1991.Now,the HTML
standard is managed by the W3C (World Wide Web Consortium).They work together
with industries to create and implement new standards.After HTML4 was released (in
1999) the W3C decided to stop improving this standard and replace it with a new one
(XHTML).The industry did not follow this vision and in the early 2000's they started to
work on a new version of HTML themselves.They created a new working group,indepen-
dently from W3C,called WHATWG (Web Hypertext Application Technology Working
Group).The goal of this new standard,called HTML5,was to make it easier for devel-
opers to create web applications in web pages and thus not use HTML only for static
content.Several years later (in 2007),W3C and WHATWG joined forces to create a new
2.2 Technologies 16
Figure 2.7:Structure of a Document Object Model Tree
HTML5 standard.5 years later,the standard is still in a working draft version and plans
are to release it in 2014.[10]
HTML5 introduces several new tags that add new functionalities or that are semantic re-
placements for common uses of the general <div> element.Important new APIs include
the canvas element that lets users draw 2D images within a browser,oine web applica-
tions,drag-and-drop functionality,web storage that allows developers to save key-value
pairs,geolocation,le support and much more.
Only the last two years HTML5 is becoming mainstream,despite of the standard not being
released yet.Steve Jobs announced in 2010 that Apple banned Flash because they believe
HTML5 is the future and will replace Flash in the end
.Even Microsoft is jumping on
the HTML5 bandwagon with their new Metro style for Windows 8 being developed with
HTML5.HTML5 is seen as a possible solution for the mobile market fragmentation,since
every smartphone has a browser.
The DOM|Document Object Model |is a platformand language-neutral interface that
allows dynamic access and updates of the content,structure and style of an HTML docu-
ment.It is standardized by the W3C and very often used when dealing with HTML pages.
In web and webview-based native applications,the DOMis accessed with JavaScript meth-
ods and is altered to change the user interface,update text elds,dynamically load records,
etc.All HTML elements and tags,which are called nodes,can be accessed and changed,
2.2 Technologies 17
just as their child elements,attributes,properties and contents.An example of a DOM
node tree is shown in Figure 2.7.
JavaScript (JS) is a prototype-based scripting language that is a dynamic,weakly typed
and general purpose programming language.It is a multi-paradigm language,supporting
object-oriented,imperative,and functional programming styles.
JavaScript is used a lot as client-side scripting language in web pages to enhance the user
interface or make the website more dynamic.A lot of browser APIs are available,such
as APIs to use the location of the user,play dierent types of media like video or music,
XmlHTTPRequest,touch events and much more.There are also APIs for use in mobile
devices which allow access to mobile specic functionality such as contacts list.The DOM
of the web pages can be changed,which implies that their contents can be changed at
runtime.Since the language is weakly typed,it is very exible.Code can be injected and
evaluated at runtime with the eval function,which requires a piece of code as String and
then executes it.
A large disadvantage of such a exible language,is the error handling.JavaScript does not
throw a lot of errors,most of the times it just stops execution.On top of that,there is no
step by step debugger for JavaScript.The only thing a developer can do to evaluate the
code at runtime is to add console.log(...) in the code to display a value.This way
he can see where exactly the code stops executing.The combination of these problems
makes it a dicult language to use for developing complicated functions.
Next to client-side scripting,JavaScript can also be used for server-side scripting.Where
client-side JavaScript is executed in the user's browser,server-side JavaScript is executed
on the server itself.Dierent libraries exist to use server-side JavaScript.The most known
one is node.js
.These libraries are the counterpart of PHP for server-side scripting.
2.2 Technologies 18
CSS (Cascading Style Sheets) is a style sheet language used for describing the presentation
(look and formating) of HTML web pages.It separates styling from content and structure
which improves exibility in layout.The CSS code can be saved in a separate le.By
applying a dierent style sheet to the website,a dierent look can be created.
CSS is easy to use.The class attribute indicates which style needs to be used on an
HTML element.In the CSS le the properties of each style are listed.CSS is heavily
used to give web and webview-based native applications a native look and feel.For each
platform there is a separate CSS le that loads the correct look and feel.
2.2.2 Techniques to create web or webview-based native applications
Web or webview-based native applications can be created in various ways,with or without
the use of extra tools or libraries.
Adjust a website to create a web application
The easiest way to create a web application is to create a website that behaves dierently
on a mobile device.E.g.,one that has a dierent layout,takes advantage of the user's
location or uses smaller pictures.Little adaptation is needed to create such a webapp,just
use a dierent CSS le that is loaded when a user is using a mobile device,and maybe
add some JavaScript.There is no need in changing the content of the website,content
that is not important on a mobile device can simply be omitted.These kinds of web pages
are often called`responsive',because they respond to dierent screen sizes.Since this
is an enhanced website,a browser and an active Internet connection are required.This
technique will only work when the website is content-driven.In case it contains a lot of
JavaScript that needs a dierent execution on dierent types of screens,it will be much
harder to adapt the website.
HTML also oers the possibility to make the website available oine,by adding the les
needed when oine to a manifest that downloads these les to the device when the user
surfs to the website.When he is oine it uses the cached les to show the website or web
2.2 Technologies 19
Create a web widget
The termweb widget can imply many things,but in this context it refers to a packaged web
application of any degree of complexity as described by the W3C
.This can be a simple
form,a shared calendar,a game or even a full-blown productivity application.These web
widgets are created with web technology (HTML,CSS,JavaScript).All the resources are
packaged in a zip le,to be run on a desktop,mobile device or on a server;with all the
same functionality that can be expected from a webapp,including server access.When
added to the manifest,web widgets have access to local system resources as well,just like
local applications.Web widgets are not dened completely yet,since they are part of the
non-nished W3C Web applications working group.
Embed a webview in a native application
In case a website needs to be transformed to a native application,this can be accomplished
by using a webview and thus creating a webview-based native application.The application
shows one view,the webview,that displays the homepage of the website.If it needs to be
available oine,all the necessary HTML-,CSS- and JavaScript-les need to be included in
the application.Since a native application is created,it can be made available through the
app stores.Access to native functions can be obtained by using a bridge from JavaScript
to native code and back.
Creating a JavaScript to native bridge for all kinds of native functionality is a lot of work,
especially when testing it on all sorts of devices.Some platforms already have this bridge,
but it works dierently on all of them so not all the code can be reused that easily.That
is why Cordova has been created.
is an HTML5 app platformthat allows developers to author native applications
2.2 Technologies 20
with web technology and get access to APIs and app stores.It has been downloaded more
than 500 000 times,there are thousands of Cordova apps in the stores and it is used by
companies like Adobe,IBM,Fox and many others.By using Cordova,these bridges to
the native code don't need to be written anymore and developers can focus on developing
the application itself.
Cordova started in 2008 as the open source framework of open standards called PhoneGap
and was developed by Nitobi,a Canadian company developing web applications.When
Nitobi was acquired by Adobe in 2011,PhoneGap changed names to Cordova and its
code was donated to the Apache Foundation which means that it will stay open source
[11].Cordova can be used for free in free,commercial or open source programs,as it is
released under the Apache License
.Adobe keeps its own copy of Cordova and still calls
it PhoneGap
.A lot of people still use the name PhoneGap instead of Cordova,but it
represents the same thing.
The framework oers support for native functions on iOS,Android,Windows Phone 7,
BlackBerry,webOS,Symbian and Bada but the best supported platforms are iOS,Android
and Windows Phone 7.It includes support for dierent kinds of sensors (accelerometer,
compass,geolocation and network) and native functionalities (camera,contacts,le,me-
dia,notications and storage).On top of that it has a plug-in based architecture that
allows developers to write their own plug-ins,which are written in partially native and
partially JavaScript code,or to download them from the PhoneGap plugin github
What Cordova does not oer is a full native user interface,like native views,buttons or
lists.Visually attractive webview-based native applications can be created by making use
of HTML5 and CSS3,or use a GUI framework like Sencha touch
or JQuery
that already contains styles that make HTML elements look like native ones.
Next to the open source framework,Adobe also oers a build service,support and training.
The PhoneGap Build service
allows developers to push the JavaScript,HTML and CSS
code to the build server,where the service automatically creates applications that can be
2.2 Technologies 21
launched in the app stores of the dierent platforms.It oers a write once,run everywhere
service,as long as no proprietary plugins are used.These services are paid services.
The internal structure of Cordova will be described in depth in Section 4.1.
is one of the direct competitors of Cordova.They claim that it is much easier
to create applications with than with Cordova.Developers don't need to set up
a native development environment like Eclipse or XCode but can just start developing.It
also includes a debug tool,something Cordova does not oer. only supports Android,iOS and Heroku,which is a web application platform.
Windows Phone is in beta and will be available soon.Native features can be accessed
similarly to Cordova but there is no plugin based architecture that allows developers to
extend the functionality. is not open source but a free license is available.Paid licenses oer support,
training and access to beta functionality such as for Windows Phone.
Titanium Mobile
Another framework that is often mentioned in this context is Titanium Mobile
tanium Mobile is a variant of the Titanium framework,which is used to make cross-
platform native development easy.It compiles web based code written in HTML,CSS
and JavaScript to native applications.Technically speaking,it does not create web or
webview-based native applications because they consist purely of native code without we-
bview.They provide lots of APIs to access native functionality.The user interface written
in JavaScript is translated to a native user interface.Titanium is not open source,but
the smallest license is free.
2.3 Testing and debugging mobile applications 22
2.3 Testing and debugging mobile applications
Things are changing in the mobile web applications area.It seems that web or webview-
based native applications will be booming in the coming years.But this will only be
the case if there is a developer-friendly way to test these applications on all the dier-
ent devices.Although a lot of developers still underestimate the importance of testing,
techniques as SCRUM,XP and Agile programming are growing,especially in the mobile
development sector.These techniques are all based on test-driven development (TDD),
which means that tests are run each time a commit is made.The changes in the commit
are only accepted if all the tests succeeded.This requires an automated build environment
with a testing framework attached to it.And there lies the problem for JavaScript and
webview-based native applications:a lot of testing has to be done manually,which con-
sumes time that a developer would rather use to actually develop the applications.These
mobile web applications,and especially webview-based native applications,will only reach
full potential if they can be tested automatically.[12]
2.3.1 Dierent types of testing
There are dierent types of testing,each one used for a specic purpose or scope.Func-
tional testing for example,is about testing the functionality of the application.This
means testing whether the user is able to do what he or she is supposed to be able to do
with the application.It is important to know that the application fullls all the promised
requirements.The counterpart of functional testing is called non-functional testing
ands means that non-functional requirements such as security,performance or usability
are tested.This is important too since it helps to prevent security or performance issues
and avoids user complaints.
White-box and black-box testing are two opposite forms of testing that dier in the vis-
ibility of the code under test.When doing white-box testing,the tester has access to
the internal code,data structures and algorithms of the application.White-box testing
includes code coverage,to check if all the code is tested and no parts are forgotten,and
private and public API testing,to see if the interfaces work as expected under all cir-
cumstances.Black-box testing treats the code as a black box and runs the application
with random parameters to test dierent partitions of input (equivalence partitioning) or
2.3 Testing and debugging mobile applications 23
to test the boundaries (boundary-value analysis).Grey-box testing is the intermediate
solution that has some notice of the underlying algorithms and structures while executing
black-box tests.
Dierent levels of testing exist too.Testing at the most detailed level is called unit testing.
Unit testing is a testing method that is used to test specic parts of the code at function
level,which are called units.The units are tested with dierent types of input,to see
if they produce the correct output.If the underlying functions are correct,chances are
higher that the total program will be correct.The next testing level is called integration
testing.Its goal is to test whether the integration of several software components works
well,such as the integration between the client- and the server-side.This is done by
exposing the defects in the interfaces and interactions between the dierent components.
Testing on the highest level is called system testing.This tests the total system or
application,after the dierent components have been integrated.
Regression testing is used to check if old bugs have come back into the application.
Each time a bug is solved,a test should be written to test if it came back.These tests
should be run at least before each release to prevent old bugs from popping up again.
When doing scenario testing,complete use cases are tested,where a use case is described
as a list of steps to achieve a goal.The focus lies on a good user experience when users are
performing dierent tasks in the application.It is important to test the use cases under
dierent circumstances and check whether the application always reacts in the correct way.
This is the type of testing that will be used in the proof of concept described in Chapter
2.3.2 In uencing the environment
Mobile web-apps and webview-based native apps are evolving from mobile versions of
websites to full-featured apps.Developers are chasing the limits of HTML5,JavaScript
and CSS,but also the limits of the smartphones themselves.These developers often would
like to know the impact of their application on the status of the smartphone.
For some applications it is crucial that tests are performed in dierent environments and
under dierent conditions,but with the large amount of types of devices and the large
amount of possible in uences,it is very hard to perform all these tests and to replicate
2.3 Testing and debugging mobile applications 24
bugs that only happen in certain circumstances.If these in uences could be tested on a
simulator or emulator this would be a step forward,but it would even be better if those
tests could be performed on real devices.It would be a huge help for developers and testers
if they could perform automated tests in certain environments and on dierent types of
Dierent situations a developer might want to test are:
 How does the application aect the battery under various conditions (for example
with screen on or o) or on dierent devices?
 How much memory does the application use in a certain scenario?Does it depend
on the type of device?
 Are the notications called from JavaScript executed on all types of devices?Did
the phone vibrate at the right time and did the music start playing at that moment?
 Does the application act correctly if there is no Internet?What if the connection is
lost while downloading a le?
 Is the application showing the correct information if the user is in a certain place
(based on a GPS location)?Does it show the correct speed while moving?
 What if there is no carrier,or if the carrier type is changed?
 How does the brightness of the screen aect the battery when the application is
running?What happens when the screen is turned o?
 Does the application react appropriately when the device's Bluetooth connection is
lost or found?
Unfortunately there are not many testing platforms or tools that help monitoring or adapt-
ing these variables on real devices.Native testing frameworks do oer the possibility to
mimic certain information providers such as GPS,carrier and Internet but these have not
been integrated in any JavaScript testing framework so far.
2.3 Testing and debugging mobile applications 25
2.3.3 Available testing and debugging solutions
Testing and debugging of native applications
Native application testing is well supported on all platforms.All the native platforms
oer testing solutions to enable developers to test their applications thoroughly.These
testing solutions are able to control the user interface of the applications.On emulators
or simulators,they are also able to manipulate the environment such as network and data
connection or gps coordinates.
Applications developed for the Android platform use JUnit
to be tested.The process
of testing native Android applications is described in Section 4.2.JUnit is an open source
unit testing framework for Java that is very popular and often used as a reference in terms
of testing.A test project that relates to a Java project is created.It is structured in
test cases that contain several tests.In the tests (which are written in Java too) dierent
methods are executed,and assertions (for example assertTrue) are used to check if the
results are as expected.When these tests are run,a report is generated that tells which
asserts did not give the expected result and thus failed.Tests can run in any order,which
is an advantage since the code is better tested.
JUnit is very popular,due to its ease of use in automated build and test systems.In`Test-
driven development concepts,taxonomy and future direction'[14] Janzen and Saiedian are
claiming that JUnit is responsible for the popularity of test-driven development and XP
techniques.Because JUnit is so often used as reference for other languages,the family of
testing frameworks that are JUnit-like are called xUnit.
When developing iOS applications,dierent tools can be used to automate the testing
process.Apple recommends their own testing tool called UIAutomation
,but little
documentation can be found about it.In UIAutomation,the tests are written in JavaScript
and they control the user interface of the application.The tool does not aimat test-driven
development,nor at using it from within a continuous integration solution or other tool.
It focuses on regression and user interface testing.Since few documentation can be found
about it and the source is not publicly available,it can be hard to gure out how to use it.
2.3 Testing and debugging mobile applications 26
This is why many developers use other tools such as Frank
or FoneMonkey
renamed to MonkeyTalk).They succeed at fullling the developer's testing needs,but are
often payed solutions.
Debugging an application is no problem on both platforms.All development tools have
a debug mode,which enables breakpoints in the code and stops the execution at these
points.Developers can go step by step through the code and meanwhile inspect the values
of the variables at that moment.The step by step process is very useful to help nd and
solve bugs.
JavaScript testing
The behavior of web and webview-based native applications is programmed with JavaScript,
which makes it important to nd a way to test this JavaScript code.Dierent frameworks
already exist,mainly for unit testing of JavaScript code,but only the most prominent
ones will be described below.The greater part of these platforms are meant to be used
for server-side or client-side JavaScript that is executed on a server or in a`normal'web
browser.Web applications can be tested by using one of these solutions,but it is much
harder to use them for testing webview-based native applications since the JavaScript is
executed in a webview,not on a server or in a browser.Some tools publish their test
results on a web page which makes automation from outside the application impossible.
All tools require that the application is altered to enable the test run, adding a
library and the test code.Testing this altered application can not ensure that the applica-
tion without these les still works ne,since the test frameworks often override JavaScript
functions to be able to run the tests or process the results.Nonetheless using some of
these solutions in a specic way could help testing webview-based native applications.All
platforms described below are open source.
is a JUnit port to JavaScript.It provides similar functionality,but is written
entirely in JavaScript so it can be used directly in the browser.Tests are included in an
HTML page which is shown in a browser.The test results are shown on this HTML page.
It was meant to test function results of web pages in desktop browsers.The JSUnit server
2.3 Testing and debugging mobile applications 27
allows automatic testing of the web page on dierent browsers at the same time.JSUnit
requires an HTML page that contains the test and thus cannot be used for webview-based
native applications since the web page that contains the application itself can not be shown
too.JSUnit is no longer maintained and is replaced by Jasmine.
is the successor of JSUnit and is created by Pivotal Labs because they were
missing a solid behavior-driven JavaScript test framework where a browser environment
was not needed to test the code.This means that Jasmine can be used to test a simple
JavaScript project,a web page,a server environment,etc.It still oers the same function-
ality as JSUnit,but the syntax has changed and the functionalities were better mapped
onto JavaScript development.This framework provides test runners and suites and a lot
of matchers and spies,which can be used to test callback events.Jasmine oers a very
basic JavaScript support,e.g.the DOMcan not be accessed,but is extension based which
allows the addition of extra functionality.There are extensions for DOMchecking,HTML
canvas checking,and many others.Jasmine can be integrated in a continuous integration
environment and is actively maintained.It is build based on dierent older JavaScript
testing frameworks that are not maintained anymore such as JSUnit,JSpec
and JSSpec
uses a dierent grammar than Jasmine.The grammar is more similar to JU-
nit with its modules and tests.QUnit is originally designed to test JQuery,which is a
lightweight JavaScript framework used to manipulate the DOM and add animations and
tools for rapid web development.It can test all forms of JavaScript code (server- and
client-side) but it does need a browser to run the tests in and to display the visual re-
port.It has a start and stop method to make it possible to test asynchronous methods
(stop when going into the method and restart when the callback is received).It can be
linked with continuous integration tools,although this is mainly supported for non-mobile
allows validation of JavaScript code in Java.Native Java tests use a bridge
to the JavaScript code and can execute the code by using eval.JsTester is focused on
2.3 Testing and debugging mobile applications 28
testing results of methods and existence of instances within JavaScript itself.It can be
used together with JUnit.It is not actively maintained anymore.
provides distributed continuous integration testing for JavaScript.Its
goal is to alleviate JavaScript testing on dierent kinds of browsers,such as desktop and
mobile browsers.The main focus is not to create another JavaScript testing library but
to create an integration with other libraries such as QUnit,JSUnit,JSSpec,Selenium,
etc.It contains test runners for most desktop and mobile browsers.TestSwarm uses a
server-oriented architecture and saves all tests,congurations and browsers in a database.
On the clients,a test runner is running in the browser which pings to the server for
new tests that need to run in this type of browser.After a run,one test in one type
of browser,the results are sent to the server where a schematic overview of which tests
failed or succeeded on which types of browsers can be found.Next to the test runners,
TestSwarm proactively corrects some bad results that are coming in from clients that are
due to unreliable browser behavior.For example the test runners act appropriately when
the client loses its connection with the server and tests are automatically run again if there
was a problem that was related to any kind of time-out.TestSwarm is not easy to set up
and can not be used for webview-based native applications since there are no test runners
for webviews.
is the most famous tool for integration testing of web applications.It is best
known for the automation of testing web applications and websites in desktop browsers,
but it recently made the switch to mobile browsers by introducing Selenium 2.0.Selenium
oers scenario testing:developers can record or describe a test case that an actual user
would execute and test them against the new code in a lot of browsers.It is a testing tool
that runs in the browsers itself.
Selenium launched their 2.0 version in July 2011,and the most important change was
that it had been merged with WebDriver to allow automated testing of web applications
on mobile browsers.WebDriver was developed earlier (2009) by Google to test their web
applications because they thought that Selenium could not fulll their needs.Now that
2.3 Testing and debugging mobile applications 29
these two are merged,testing web applications and websites on mobile browsers became
Selenium can be used for webview-based native applications,but needs to run in an
adapted webview which can introduce bugs that will not be found when using the de-
fault webview or the other way around.Webview-based native applications often already
run in an adapted webview,which can not be replaced so easily with the webview that is
required for Selenium since the Selenium webview lacks functions that are needed to run
the webview-based native application successfully.To use Selenium 2.0 on mobile devices
the specic WebDrivers for Android,iOS or the HTMLDriver need to be used.Other
platforms are not supported yet.
Putting these disadvantages aside,Selenium has lots of good things too.It is a very good
example of what a scenario testing framework should be able to do.It can run tests or test
suites within the website or web application and it provides a lot of commands that can be
used for testing.It has three types of commands:actions (manipulate state,for example
clicks),accessors (examine states of the application and store them) and assertions (verify
states of the applications).The tests can be described manually or recorded by using the
record button and adapting the recorded tests.Selenium can be used in a continuous
build environment.
Debugging web and webview-based native applications
Debugging web and webview-based native applications is much harder than debugging na-
tive applications,since JavaScript does not have a debugger.If a developer wants to debug
or inspect the code at runtime,the only option he has is to print regular console.log()
messages that include the value of variables or that mark where in the code the bug is
situated.Unfortunately,a lot of console loggers literally print [object] when an object
is printed that consists of multiple elds.In that case the log message has to be adapted
and the code needs to be rerun to print all the necessary elds of that object).
Integrated debugging tools such as Webkit Web inspector
,Opera DragonFly
Mozilla Firebug
allow inspection of an HTML page and its contents at runtime and in
2.4 Working groups and projects 30
the browser itself.In webkit browsers it is even possible to start this inspection by right
clicking on the page and select`Inspect element'.These tools show the DOM structure
and the CSS styles that are linked to an HTML tag and allow to run JavaScript code on
this website.
is a debugging tool just as those described above,except that it is designed
to work remotely,and in particular,to allow a developer to debug web pages in a mobile
webkit-based browser or webview.The debugging view is not shown within the browser
or webview that contains the application,but on a separate web page that can be accessed
by a non-mobile browser,resulting in more space to show the debugging view.This is not
yet possible with the previously mentioned debugging tools since they need to run in the
browser itself.The developer of Weinre is Patrick Mueller,who is working on the Web
Inspector debugger at IBMEmerging Technologies.He started working on Weinre in 2010
as an experiment,but it soon became a solid project.Weinre was always closely related
to PhoneGap since it was almost the only way to debug PhoneGap applications.Since
PhoneGap's switch to Cordova,Weinre is a part of the Cordova project.Adobe runs its
own Weinre server
that developers of PhoneGap applications can use freely.Section 4.4