Augmented reality using Android

estrapadetubacityMobile - Wireless

Dec 10, 2013 (3 years and 7 months ago)

213 views

Augmented reality using Android
TDT4290 Customer driven project
Authors:
Yoann Marion
Dan Erik Holmstrm
Jan Eriksson
Mads Opheim
Kjetil Babington
Ken Brge Viktil
Jesus Valero
Supervisors:
Meng Zhu
Muhammad Asif
November 2010
2
Preface
This report is about the project realized in fall 2010,by a team of seven students
from the computer sciences department of NTNU,Trondheim.It has been done
for the course named\Customer driven project",whose two main goals are rst to
learn to work in a team,or at least improve the way to do it,and second to follow
a customer instructions.We can add a sub-goal concerning the team working,
which is the use of the SCRUM development method,which was new for all the
team members,at the beginning of the project.Our customer was Cyberlab Org.
AS,a company which we present later.Their project was to develop a smartphone
library which they could use to develop several applications and games,around
the theme of\Image recognition and Augmented reality".The entire work of
the project is documented in this report,including a description of the work,the
results,the preliminary studies on forehand as well as our planning.
For this project,the team was able to count on two supervisors:Meng Zhu
and Muhammad Asif,and on the two customers we met:Tor Ivar Eikaas and
Frank Jakobsen.We want to thank them,on one hand for helping us improving
our work and methodology,and on the other hand for having been patient with
us and our work.
Trondheim,November 24,2010
Yoann Marion Jan Eriksson
Ken Brge Viktil Dan Erik Holmstrm
Mads Opheim Jesus Valero
Kjetil Babington
i
Contents
1 Introduction 1
1.1 Project name.............................1
1.2 Stakeholders..............................1
1.3 Resources...............................1
1.4 Report structure...........................2
1.5 Customer description.........................2
1.6 Initial project description.......................3
1.7 Scope.................................3
2 Project plan 5
2.1 Overall project plan..........................6
2.1.1 General terms.........................6
2.2 Concrete project workplan......................7
2.3 Project organization.........................9
2.3.1 Roles denition........................9
2.3.2 Organizational diagram....................10
2.3.3 Weekly schedule........................11
2.4 Tools for documents and report...................12
2.4.1 Writing and sharing the documentation...........12
2.4.2 Controlling the documents'evolution............13
2.4.3 Templates and standards...................13
2.5 Tools for implementation.......................14
2.6 Tools for project management....................15
2.6.1 Status reports.........................15
2.6.2 Time table...........................15
2.6.3 Burndown chart........................16
2.6.4 A TODO list.........................16
2.7 Risk management...........................17
ii
CONTENTS
3 Prestudy 21
3.1 Augmented reality..........................22
3.1.1 The beginning of augmented reality.............22
3.1.2 Dierent augmented reality degrees.............23
3.1.3 Augmented reality using smart-phones...........25
3.1.4 Limitations..........................27
3.1.5 Conclusion...........................27
3.2 Object recognition..........................28
3.2.1 SIFT..............................28
3.2.2 SURF.............................29
3.2.3 Why we chose SURF.....................30
3.3 QR codes...............................31
3.4 Target platform/The choice of platform...............32
3.4.1 Choice of software platforms.................32
3.4.2 Conclusion...........................35
3.5 State of the art............................35
3.5.1 Android Market........................35
3.5.2 Licenses............................36
3.5.3 Google Goggles........................37
3.5.4 Google Sky Map.......................37
3.5.5 Google Maps.........................37
3.5.6 Wikitude...........................37
3.5.7 ZXing.............................38
3.5.8 Layar.............................38
3.5.9 ARToolKit...........................39
3.6 Android versions...........................39
3.7 Android NDK and JNI Bindings...................40
3.7.1 JNI...............................40
3.7.2 Android NDK.........................40
3.8 Server-side technologies........................41
3.9 Management and methodology....................42
3.9.1 Scrum.............................42
3.9.2 Waterfall............................44
3.9.3 Scrum VS Waterfall for this project.............45
3.9.4 Test-driven development...................46
3.9.5 Xtreme Programming.....................47
3.9.6 File sharing and synchronization...............47
3.9.7 Programming code synchronisation.............49
3.10 Synthesis of this prestudy......................49
iii
CONTENTS
4 Quality Assurance 51
4.1 General considerations........................52
4.2 Quality of the documentation....................52
4.3 Testing and testing policy......................52
4.3.1 Measurement of\quality"..................53
4.3.2 Testing at dierent levels...................53
4.3.3 Conclusion...........................60
4.4 Test plan...............................60
4.4.1 Use of standards.......................61
4.4.2 Test items...........................61
4.4.3 Approach...........................61
4.4.4 Item pass/fail criteria.....................62
4.4.5 Test plan identier......................63
4.4.6 Test case template......................63
4.4.7 Test case overview......................64
4.4.8 Environmental requirements:.................65
4.4.9 Responsibilities........................65
5 Software Requirements 66
5.1 Overall description..........................67
5.1.1 Product functions.......................67
5.1.2 User characteristic......................67
5.1.3 Constraints..........................68
5.1.4 Assumptions and constraints.................68
5.2 Specic requirements.........................68
5.2.1 Priority............................68
5.2.2 Functional requirements...................69
5.2.3 Non-functional requirements.................70
5.2.4 Software requirements....................71
5.2.5 Textual use cases.......................72
6 Product backlog 80
6.1 General information..........................81
6.2 Initial product backlog........................81
6.2.1 Comments to initial backlog.................84
6.3 Final product backlog........................84
6.3.1 Comments to nal product backlog.............89
7 Software Architecture 90
7.1 Overview................................91
7.2 Deployment view...........................93
iv
CONTENTS
7.3 Logic view...............................94
7.3.1 Presentation layer.......................95
7.3.2 Software logic layer......................96
7.3.3 Caching and data source layer................96
7.3.4 Data storage and retrieval layer...............99
7.4 Physical view.............................99
7.4.1 Inferring the location.....................100
7.4.2 Recognizing objects......................100
7.5 Patterns in design...........................103
7.6 Rationale...............................103
8 Sprint 1 105
8.1 Sprint goal...............................106
8.2 Sprint breakdown...........................106
8.2.1 Sprint schedule........................106
8.2.2 Sprint backlog.........................106
8.2.3 Burndown chart........................108
8.2.4 Sprint 1 prototype......................108
8.3 Sprint 1 summary...........................111
8.3.1 Problems encountered:....................111
8.3.2 Sprint retrospective:.....................112
9 Sprint 2 113
9.1 Sprint goal...............................114
9.2 Sprint breakdown...........................114
9.2.1 Sprint schedule........................114
9.2.2 Sprint backlog.........................114
9.2.3 Burndown chart........................116
9.2.4 Sprint 2 testing........................116
9.2.5 Sprint 2 prototype......................117
9.3 Sprint 2 summary...........................119
9.3.1 Problems encountered:....................120
9.3.2 Sprint retrospective......................121
10 Sprint 3 123
10.1 Sprint goal...............................124
10.2 Sprint breakdown...........................124
10.2.1 Sprint schedule........................124
10.2.2 Sprint backlog.........................125
10.2.3 Burndown chart........................127
10.2.4 Sprint 3 testing........................127
v
CONTENTS
10.2.5 Sprint 3 prototype......................128
10.3 Sprint 3 summary...........................135
10.3.1 Problems encountered:....................136
10.3.2 Sprint retrospective......................137
11 Evaluation 139
11.1 Working as a team..........................140
11.1.1 The group...........................140
11.1.2 The SCRUM method.....................141
11.2 Risk evaluation............................142
11.2.1 Implementation problems..................142
11.2.2 Communication problems with the customer........144
11.2.3 Interfering work from other subjects............145
11.2.4 Conclusion...........................146
11.3 Experiences with Android programming and augmented reality de-
velopment...............................146
11.3.1 Android programming....................146
11.3.2 Android documentation...................147
11.3.3 Augmented reality on Android................148
11.4 Software architecture.........................148
11.4.1 The grand plan........................149
11.4.2 Architectural update due to change in requirements....151
11.4.3 Unexpected consequences..................152
11.4.4 Time estimations.......................152
11.5 Software requirements........................153
11.5.1 Constraints..........................153
11.5.2 Functional requirements...................153
11.5.3 Product scope.........................154
11.5.4 Non-functional requirements.................156
11.6 The product..............................157
11.7 Workload...............................158
11.8 Further improvements........................160
11.8.1 Image recognition.......................160
11.8.2 Location and sensors.....................162
11.8.3 Client-server communication.................163
11.9 Conclusion...............................164
Appendices
vi
CONTENTS
A Project Management A-1
A.1 Customer meeting summary.....................A-2
A.2 Supervisor meeting summary....................A-3
A.3 Meeting agenda............................A-4
A.4 Status report.............................A-5
A.5 Time list................................A-6
B Test cases B-1
B.1 Test cases...............................B-1
B.1.1 Test case template......................B-1
B.1.2 Unit tests...........................B-2
B.1.3 Integration tests........................B-9
B.1.4 System tests..........................B-12
C The documentation quality C-1
C.1 Organization.............................C-1
C.1.1 Legend.............................C-1
C.1.2 Detailed le organization...................C-2
C.2 File headers..............................C-3
C.2.1 General information:.....................C-3
C.3 Reviews information:.........................C-3
C.3.1 Table of contents.......................C-3
C.4 Final versions.............................C-3
D The client architecture D-1
D.1 Overall system............................D-1
D.2 Client program............................D-3
D.3 Camera Management.........................D-4
D.4 Image recognition...........................D-5
E The server architecture E-1
E.1 Program environment.........................E-1
E.2 Alternatives..............................E-2
E.3 Structure...............................E-3
E.4 The structure of a client object...................E-7
E.5 Thread cancellation..........................E-7
F The remote database F-1
G Sending-format G-1
vii
CONTENTS
H Matching H-1
H.1 Matching steps............................H-1
H.1.1 Evaluation of a match....................H-1
H.1.2 KD-tree............................H-2
H.2 Implementation............................H-10
H.2.1 MatchingPoints package...................H-10
H.2.2 FindMatchingFeatures package...............H-10
I Building the pictures database I-1
I.1 General process............................I-1
I.2 On the client side...........................I-2
I.3 On the server side...........................I-2
I.4 Selection of the best results.....................I-3
I.5 Possible way of improvement.....................I-4
viii
List of Figures
2.1 Project gantt chart..........................7
2.2 Project organization.........................10
2.3 Example of an intermediate Time Chart..............16
3.1 One example of the rst applications of augmented reality[1]...22
3.2 Example of duciary markers[2]...................23
3.3 Tissot use augmented reality to let the customer to try the watch[3] 24
3.4 Wikitude World Browser,on one of the most famous smartphones,
iPhone 3GS,using GPS and solid state compass[4].........25
3.5 [5]...................................26
3.6 Android version distribution,Nov,2010 [6].............41
4.1 LogCat view as seen from Eclipse..................56
4.2 JUnit testing in Eclipse........................58
4.3 TraceView...............................59
5.1 Application use case.........................72
5.2 GPS positioning use case.......................77
5.3 Feature extraction use case......................77
5.4 Glyph recognition use case......................77
5.5 Text overlay use case.........................77
5.6 QR-code reader use case.......................78
5.7 Integrated compass use case.....................78
5.8 Indoor location use case.......................78
5.9 Object tracking use case.......................79
7.1 MVC pattern diagram........................91
7.2 Deployment diagram.........................93
7.3 Layer overview............................95
7.4 Querying for QR code........................97
7.5 Querying for QR code........................98
7.6 Package level view..........................102
ix
LIST OF FIGURES
8.1 Sprint 1 burndown chart.......................108
8.2 Sprint 1 burndown table.......................109
8.3 The message the user gets if he has no QR code reader......109
8.4 The local database..........................110
8.5 The database with menu items....................110
8.6 The prototype of the application...................111
9.1 Sprint 2 burndown chart.......................117
9.2 Sprint 2 burndown table.......................118
9.3 Initial glyph to be recognized....................118
9.4 Glyph with features painted on...................119
9.5 Testing client for the server.....................120
9.6 Server output for a single client...................120
10.1 Sprint 3 burndown chart.......................127
10.2 Sprint 3 burndown table.......................128
10.3 The main menu after sprint 3....................132
10.4 The local database activity,including the menu..........133
10.5 The QR code activity.........................133
10.6 The SURF activity in use......................134
10.7 The results of the GPS activity...................134
11.1 The package diagram for the client side - before..........149
11.2 The package diagram for the client side - after...........150
11.3 Work percentages...........................159
11.4 Weekly work hours..........................160
11.5 One example of a false positive found by RANSAC........161
A.1 Time list example...........................A-6
D.1 Final deployment diagram......................D-2
D.2 Package diagram for the resulting client program.........D-3
D.3 Sequence diagram of camera management.............D-4
D.4 Outline of the actions taken to process an image on the client..D-5
E.1 Server outline.............................E-4
E.2 Server setup..............................E-5
E.3 Creating a client object........................E-6
E.4 Client object outline.........................E-8
E.5 Client object setup..........................E-9
I.1 Example of testing table.......................I-4
x
List of Tables
4.1 Test plan identiers..........................63
4.2 Test case template..........................63
4.3 Test nished template........................64
6.1 Main backlog initial entries......................84
6.2 Main backlog initial entries......................89
8.1 Main backlog entries.........................107
8.2 Additional backlog entries......................107
9.1 Main backlog entries.........................115
9.2 Additional backlog entries......................115
9.3 Sprint 2 testing............................116
10.1 Main backlog entries.........................126
10.2 Additional backlog entries......................126
10.3 Sprint 3 test 1.............................128
10.4 Sprint 3 test 2.............................129
10.5 Sprint 3 test 3.............................129
10.6 Sprint 3 test 4.............................129
10.7 Sprint 3 test 5.............................130
10.8 Sprint 3 test 6.............................130
10.9 Sprint 3 test 7.............................130
10.10Sprint 3 test 8.............................130
10.11Sprint 3 test 9.............................131
10.12Sprint 3 test 10............................131
10.13Sprint 3 test 11............................131
B.1 Test identiers............................B-1
B.2 Test case template..........................B-2
B.3 Surf algorithm test..........................B-3
B.4 Image matching tree test.......................B-6
xi
LIST OF TABLES
B.5 Image matching neighbour test...................B-7
B.6 Finding matching features in the imagebank............B-9
B.7 Server connection test........................B-10
B.8 Server key point request test.....................B-11
B.9 Server close connection test.....................B-12
B.10 QR-code installer test........................B-13
B.11 QR-code reader test.........................B-14
B.12 Indoor location test..........................B-15
B.13 GPS identify building test......................B-16
B.14 Object recognition test........................B-17
B.15 Glyph recognition test........................B-19
xii
List of abbreviations
API Application Programming Interface
AR Augmented Reality
CPU Central Processing Unit
CVS Concurrent Version System
ECC Error Correcting
GPS Global Positioning System
IPC Inter-Process Communication
ISO International Organization for Standardization
JNI Java Native Interface
LAMP Linux,Apache,MySQL,PHP
MVC Model View Controller
MVP Model View Presenter
NDK Native Development Kit
OS Operating System
SDK Software Development Kit
SIFT Scale-Invariant Feature Transform
SRS Software Requirements Specication
SURF Speeded-Up Robust Features
SVN Subversion
xiii
LIST OF TABLES
TDD Test-Driven Development
UI User Interface
UML Unied Modeling Language
xiv
Chapter 1
Introduction
This section will give a basic understanding of the project this group has been
working on,the characteristics of the customer and the general structure of the
report.
1.1 Project name
This project got the name\Augmented reality using Android",as this is the
one-sentence summary of what the project actually is about
1.2 Stakeholders
The stakeholders of this project is the group,the customer and our supervisors.
1.3 Resources
The main resource for this project is the working time and the work power of
the group members.Each group member will each week of this project spend
approximately 20 hours working with it.As the project starts on the 31th of
August and nishes on 25th of November,this means 12 working weeks and thus
approximately 260 working hours per student.
Concerning physical resources,each group member will use his personal lap-
top.As this project focuses on the Android mobile phone-system,the four group
members who own an Android phone will use it for testing.
For the code sharing and version control,we will use a Mercurial HGrepository
stored on one group member's personal server.
1
1.INTRODUCTION
We will excessivly use the group rooms on the fth oor of the P15-building
on Glshaugen.
1.4 Report structure
This report is divided into eleven chapters.The rst seven chapters describes our
project as a whole,from planning and prestudy to Software requirements and
the product backlog.Chapter eight to ten are the sprint reports we have written
after each sprint.The last chapter contains our evaluation of this project,things
that we would have changed if we did start over again and things that went well.
To be more specic,the chapter you are reading now is the introduction to
this report as well as our project,and we will continue by describing our planning
and planning process,followed by the result of our preliminary studies.
Next,we will discuss the quality assurance of the project,followed by our
section on software requirements.This is followed by the product backlog and
then the architecture of our application.Then,we provide the reports from each
of the programming iterations (sprints),and conclude by evaluating the project.
The last section of this report consists of the appendices.
1.5 Customer description
Cyberlab Org.AS is a small company developing games and simulators,for ed-
ucation and training.They have developed their own tool to develop simulators:
PIDstop.Several projects of Cyberlab are focused on the energy challenges (The
Energy Game,The Climate Doctor).They have worked for NTNU several times,
with projects such as a heat pump simulator and a simulation-based game for
automation and control.Also,some projects have been developed in cooper-
ation with NTNU,for instance for The Energy Game.Currently,Cyberlab is
interested in object recognition and augmented reality for mobile applications,
notably for iPhone and Android-based mobile phones.They want a library and
a prototype,subsequently allowing them to develop innovative games concepts
on these supports.Our project will focus on the Android-based mobile phones,
partly because the customer seems to be more interested in that solution.And
clearly,the team is more familiar with Java,the programming language used on
Android.Further,considering the nal product Cyberlab.org would like to sell,
Android market will be an easy and good solution.
2
1.6 Initial project description
1.6 Initial project description
Given here is the exact,word-by-word,description of the project,pro-
vided by the customer in front of the project:
D4 Cyberlab.org { Tor Ivar Eikaas
Title:Object recognition and Augmented Reality for Android and/or iPhone
mobiles
Customer:Cyberlab.Org AS
Address:Innovasjonssenter Glshaugen,Richard Birkelands vei 2B,7491 Trond-
heim
Cyberlab is a small company developing games for education and training.We
are interested in investigating new and innovative game concepts for mobile appli-
cations,and notably for iPhone and Android-based mobile phones.This project
assignment is focussing on object recognition and augmented reality for mobile
applications,and should work towards developing a sw library for object recog-
nition and augmented reality on android mobile phones together with a proof of
concept application.
- Present state-of-the-art for object recognition and augmented reality for
mobile phones (Android and iPone [sic]),including available commercial
and open source solutions.
- Development of a library for object recognition and augmented reality on
Android phones,building on available solutions where possible.
- Develop a prototype pervasive game utilising the developed object recogni-
tion and AR library in addition to other available location based services
(e.g.GPS) etc where applicable.
1.7 Scope
The rst part of the scope for this project is to research on existing solutions in
the eld of augmented reality and map out the things that are possible to make
in the course of this project.The customer is interested in the available solutions
and if these are possible to use in their needs of augmented reality.The team will
use this part to make a suggestion to the customer in modules for a prototype.
The second part of the scope for this project is to make an application for
an phone using the android OS,with the subject of augmented reality.The
application should be a prototype using a library.This will be done by imple-
menting several modules that serves as a basis to augment the reality,by using
3
1.INTRODUCTION
the phone's camera.This is the main part of the project,and it will be the most
time consuming.
4
Chapter 2
Project plan
The main goal of this section is to understand how the team has worked on the
project,how the members were organized,how the time were managed and what
tools the team has used to achieve the aims of the project.
Purpose
The purpose of the project plan is to establish a plan for the team to utilize
throughout this project.It is here to provide guidelines for how the project
should be executed.
Scope
The project plan contains the specic plan we have decided to use in this project.
That includes a general project plan,the way we organize our team,tools we
have decided to use and an assessment of the possible risks and their impact on
this project.This document may be changed as we go along in our project,as
we may have the need to redene our way of work as we experience the impact
of our early decisions.
5
2.PROJECT PLAN
2.1 Overall project plan
The above goals mix the team interest and the measurement of project eects
concerning the customer,Cyberlab.org AS.The team members want to improve
their ability to work as a team,to become familiar with the Scrum method-
ology,with Android development,and more generally with the technologies of
augmented reality.The customer goals are the following ones:
 Improve integration of image recognition techniques in future Cyberlab.org
projects.
 Improve the Cyberlab.org developers ability to understand and use dierent
positioning and recognition techniques
 Improve the customer's vision of Augmented reality,in terms of techniques,
applications and philosophy.
2.1.1 General terms
Workload and development methods
The duration of the project is twelve weeks,the rst pieces of information were
given on August the 27th,and the nal presentation is planned for November the
25th.The rst weeks are dedicated to the planning,the discovery of the project
and the customers wishes.As the customers requirements are a bit unclear,and
we have to investigate several issues and solutions,this results in that we have to
spend three weeks on the prestudy.To prepare the presentation,and complete
the nal report,we need at least one week at the end;the presentation being on
a Thursday,we plan to work on it from the beginning of the previous week.
This gives us eight weeks to develop the prototype.We plan to use the Scrum
method for this project,and we plan to do three sprints during these eight weeks.
The rst one will deal with simple facts,introduction into Android development,
and won't dig into image recognition algorithms.Thus,we allocate two weeks for
this sprint,and three weeks per remaining sprint.Three weeks is usually a good
length for a sprint,long enough to let the team deal with potential problems,
short enough to remain agile.
Team descripton
The team is composed of seven people who all have some skills into computer
sciences.Five of these people are Norwegian,one is French and one is Spanish.
Thus,it is obvious that all our documents,meetings,standards,code naming
rules and documentation have to be in English.
6
2.2 Concrete project workplan
Planned eort
Each group member has the following planned eort each week,in average:
Task
Weekly eort (h.)
Lectures
4
Meetings
5
Development
12
Documentation
3
Total
24
Lectures are not every week,and are not always four hours.It seems that the
priority is then given to the documentation when lectures are shorter.
Schedule of result
As we mentioned above,this project will be divided into three sprints:
Begins
Ends
Sprint 1
20/09/2010
04/10/2010
Sprint 2
04/10/2010
25/10/2010
Sprint 3
25/10/2010
15/11/2010
Total
24
2.2 Concrete project workplan
Figure 2.1:Project gantt chart
Here we describe the milestones of this project:
7
2.PROJECT PLAN
Milestone
Date
Deliverable
Sprint 1
04/10/2010
 Skeleton of the library
 Use of GPS
 Use of compass
 Use of QR code reading
 First graphical interface and function-
alities for the prototype
Pre-delivery
11/10/2010
 Abstract
 Pre-study
 Report table of contents
Sprint 2
25/10/2010
 Deep use of the phones'devices
 Introduction of image recognition
 Improvement of the protoype interface
Sprint 3
15/11/2010
 Deep use of image recognition
 Improvement of the combination of de-
vices'use and image recognition
 Improvement of the protoype
8
2.3 Project organization
Presentation
25/11/2010
 Presentation
 Demonstration
 Final report
2.3 Project organization
This part of the Project plan describes how the team want to work.It denes a
frame which should help each team member to know what he has to do and what
he must let someone else do.
Abbreviations
PIC:Person In Charge
DET:Dierent Each Time
Other abbreviations can be given throughout the document.
2.3.1 Roles denition
Role
Name
Who is he?
Project manager
(PM)
Yoann
He has to plan the project,to follow its
progression,organize and animate meetings,
solve con icts or dene a way to solve them,
make sure that other roles are ecient.He is
also in charge of the quality assurance of the
project.
Test manager
(TM)
Jan
He should make sure that each team member
has the same test habits.Some processes has
to be dened:integration tests,tests before
commit etc.He is not necessarily the tester.
System architect
(SA)
Kjetil,Dan
They are in charge of the consistency of the
whole application,and they are able to ana-
lyze all the layers of the product.
9
2.PROJECT PLAN
Customer con-
tact (CC)
Mads
He is in charge of the communication with the
customer.He should assist all of the meetings
if possible.If there is a con ict with the cus-
tomer,he will work with the project manager
to nd a solution.
Implementation
managers (IM)
Ken,Dan
They are responsible for the integration of
the separate software modules.The modules
should fulll requirements in reliability and
performance.
2.3.2 Organizational diagram
The following diagram shows how the roles are organized.
Figure 2.2:Project organization
The project manager organizes and plans the work of the team.He doesn't
say what to do but when.This way,he is not a leader,but he is coordinating
10
2.3 Project organization
the project.What the members should do is decided by the whole team,during
customer and internal meetings.During these decision meetings,the customer
contact makes sure that everything matches the customer's needs,the system
architect should prevent or anticipate large product changes,the test manager
and implementation manager decide if the work is consistent during the period
of work.
2.3.3 Weekly schedule
Meeting
Task
PIC
Deadline
Customer meeting
Monday 10:15
Book a room
CC
Tuesdays 14:00
Write agenda draft
PM
Fridays 10:00
Send agenda
CC
Friday 12:00
Write summary
DET1
Tuesdays 10:00
Send summary
CC
Tuesdays 12:00
Supervisor meeting
Mondays 15:00
Write agenda draft
PM
Fridays 10:00
Send agenda
PM
Fridays 12:00
Write summary
DET2
Tuesdays 10:00
Send summary
DET2
Fridays 12:00
Internal meeting
Tuesday 12:15
Write agenda
PM
Tuesday 18:00
Write summary
DET3
Thursday 10:00
Internal meeting
Friday 10:15
Write agenda
PM
Thursday 18:00
Write summary
DET4
Monday 10:00
Summary writing and validation
To avoid bad document validation we will at the beginning of each internal meet-
ing have a point which consists of going back of the previous summaries.If a
summary is already sent to the customer or to the supervisors,has to be changed
because of that point,a second version will be written and sent by the person in
charge of writing this summary.The person in charge of writing the summary,
change each meeting.We follow the order of the nouns given in the\contact
information"document.
11
2.PROJECT PLAN
Weekend work
It has been decided that the team members are not supposed to work during the
weekend.It doesn't mean that we can't ask a coworker to realize a task during
the weekend,it means that we can't assume that weekend days are labour days.
If some work is given between Friday and Monday,its length should be a few
hours maximum,not longer than what we can do in an evening.Obviously,this
remark depends on everyone's habits.The personal work of each member does
not have to be organized by the team,this is just a consideration.
2.4 Tools for documents and report
In this section we describe the tools we want to use to produce documents.
2.4.1 Writing and sharing the documentation
For ease of storage and access,Google docs will be used to write and export our
documents.It is a good collaboration tool,which will allow us to deal eciently
with documents updates.For other documents,such as dierent technical doc-
umentation found on the internet we want to share with the team,we will use
Dropbox.It is easy to use,because it is synchronized automatically,and available
space is big enough for our needs.Both Google docs and Dropbox are described
in the prestudy part,where we will explain the reasons behind these choices.
The report is a big document,composed of several dierent parts that we
need to separate clearly at the beginning of its production.We will use Google
docs to write its dierent parts,such as this one.This way we avoid writing
con icts,we are able to see more quickly the consistency of each part,and as we
create an overall report draft document,we are also able to see the consistency
of the whole report,and later to integrate each part easily.At the beginning of
October,we need to merge these parts into a rst version of the report.It has
been decided that we will use L
A
T
E
X to produce nal versions of the report.If
each part is dened correctly,and referenced in the report draft,the merging step
will not be a problem,and it should not take a lot of time.Once a consistent
report has been produced (basically,from the beginning of October),the merged
document will be the reference,and we will not use Google Docs anymore to add
some content to existing parts.But,for new parts,Google docs will still be used,
and merging will be done little by little,as this is the most natural way.
12
2.4 Tools for documents and report
2.4.2 Controlling the documents'evolution
Google docs has its own system to follow modications added by the dierent
writers.It is very complete,not readable and it is a tool separate from the doc-
ument.That is why we will use it only if we want to cancel some changes.To
follow documents'evolution,we will use a header describing steps of modica-
tion,authors,comments,dates and status of the document.Please refer to the
documentation quality document,in the appendix (Appendix C).
Concerning the report sub-parts,once we will have merged theminto a LaTeX
le,we will use the same versioning system as the one for the code:Hg.Please
see the upcoming (Chapter 2.5) Tools for implementation part for more details.
As L
A
T
E
X les are sources les,if we choose Hg for sources,it is also well adapted
to L
A
T
E
X.We face the same problem as from Google docs,the evolutions will
not be part of the document,and will not be very fancy,but the parts merged in
L
A
T
E
X for the preliminary report should not be changed too much.
2.4.3 Templates and standards
Here is a list of the templates used in this project:
 Meeting agendas
 Advisory meeting agendas
 Meeting summaries
 Document header
To use a template,the team member has to open it with Google Docs,then
click File and Create a copy,rename the new document generating,and move it
into the good folder because Google Docs creates it in the personal folder.
No matter if a document has been produced with a specic template or not,
we need some generic information to be sure that our documentation is consis-
tent.Below we describe this generic information that has to be present on each
separate document,except for the nal report,because the information is mainly
for internal use.
Files header
Each le might has headers comparable to this le's header.This header is
composed of following pieces of information:
General informations:
 Accurate title of the document
13
2.PROJECT PLAN
 Team number and reference
 Author
 Participants (if several authors)
 Creation date:dd/mm/yyyy
 Version
 Status:(ongoing/nished/reviewed/closed)
Reviews information:
 Review date:dd/mm/yyyy
 Reviewers
 Comments
 Version
Table of contents
A table of contents is a useful feature for long or complex documents.
In appendix,a documentation quality document shows how this information
has to be presented.This document also describes how the produced documents
has to be organized in the Google Docs folders.
2.5 Tools for implementation
To write the application's code and tests,we will use Eclipse and its ADT plugin,
which allow us to create Android applications easier and faster[7].As it is men-
tioned in the software requirements part,we have used the Android SDK as well.
This is the de facto standard program stack for Android developing,also strongly
recommended by the Google Android team,providing a working environment
saving us lots of time and eort compared to the alternative solutions.
For the code versioning,we will use Hg for the following reasons,also described
in the preliminary studies part of this report (section 3.9.7):
 It supports branching
 We already have a server
 It is decentralised
14
2.6 Tools for project management
We made guidelines for ourselves,stating that we have to commit our code when
we have done some changes and tested that they were working.
The customer gave us their development guidelines,which are attached to this
report in appendix.We have to follow these guideline very strictly.
2.6 Tools for project management
We rst used Microsoft Project for the global planning.It helped us to dene
our milestones and sprints lenghts.
During the whole project,we used several types of documents to follow the
team's progression:
2.6.1 Status reports
It helps us to see what we did the previous week,what we plan to do the next
week.Status reports are a good way to detect problems,and to prevent them.
It is a document that we can present to the supervisors to show them how much
work we do each week.
2.6.2 Time table
This document quanties the work we do.This document is made with two parts,
one is internal and allows us to compare the work of each team member between
them or between the dierent kinds of tasks.The second one is more synthetic
and has to be shown to the supervisors,to show them how much time it takes
us to realize the work we plan to do.A chart can be produced concerning the
internal part of this document,see an example of such a chart,below.The work
hours are reported on the vertical axis.
15
2.PROJECT PLAN
Figure 2.3:Example of an intermediate Time Chart
This kind of diagram is very talkative,and we are able to produce it auto-
matically during each week,to follow the team's progression.
2.6.3 Burndown chart
This document is specic to each sprint,and concerns only the application itself,
including the specication and the code.The goal here is to compare our time
estimations with the actual work needed.There are two kinds of applications for
this goal:rst we are able to adjust our future estimations,to make it more and
more realistic;second we are able to detect critical tasks which will take much
more time than planned.
2.6.4 A TODO list
This document is more than just a list of the remaining tasks to do.We initially
wanted to use a specic tool such as Trac,however,we have reached the conclusion
that it will be easier to use a simple spreadsheet.This list allows us to report
any bugs or tasks that remains to do,to place one or more members on it,and
16
2.7 Risk management
to follow the evolution of a task.Each task is described with an Id,a Tracker,
a Status,a Priority,a Description,a Deadline,a person Assigned to,a Created
on-date,and an Updated on-date.
2.7 Risk management
Every risk description will follow this blueprint:
Risk:Blueprint
Risk number
The number of the risk
Activity
In which activity the risk is considerable
Risk factor
What is the risk?
Consequences
What are the consequences of this risk?
Probabiltiy
How probable is this risk to happen?
Strategy and actions
How do we handle this risk?
Deadline
How long will this risk last?
Responsible
Who is responsible for handling this risk?
Second responsible
Who is the second responsible for this?
And here follows the dierent risks we have identied
Risk:Sickness
Risk number
1
Activity
All
Risk factor
Sickness,failure to attend meetings
Consequences
High:Loss of overall workpower,reduction in productiv-
ity
Probabiltiy
Medium
Strategy and actions
Avoid:Don't go out partying if you have a sore throat
etc.
Deadline
Continues
Responsible
Project manager
Second responsible
-
Risk:Implementation problems
Risk number
2
Activity
Sprint output
17
2.PROJECT PLAN
Risk factor
Implementation problems
Consequences
High:Less time to nish project if resources are invested
in a dead end solution
Probabiltiy
Medium
Strategy and actions
Avoid:Do a good background research on the module
we want to implement to ensure that it is within our
skill level to nish.
Deadline
Sprint 1
Responsible
Implementation managers
Second responsible
-
Risk:External communication problems
Risk number
3
Activity
Sprint output
Risk factor
Communication problems with the customer
Consequences
High:Less time to nish project if resources are invested
in a dead end solution
Probabiltiy
Medium
Strategy and actions
Avoid:Group must prepare before customer meetings
and query about their wishes and get response on sug-
gested solutions so there is no doubt what they want
Deadline
Sprint 1
Responsible
Customer contact
Second responsible
-
Risk:Internal communication problems
Risk number
4
Activity
All
Risk factor
Internal communications problems
Consequences
Medium:Group members may get unfocused,some
members may do the same work twice.
Probabiltiy
Medium
Strategy and actions
Avoid:Make the regular group meetings structured by
appointing a project manager and having an agenda.If
everyone is assigned a task there will be little confusion
about the days work
Deadline
Continues
Responsible
Everyone,(mainly Project manager)
18
2.7 Risk management
Second responsible
Secretary
Risk:Lack of motivation
Risk number
5
Activity
All
Risk factor
Lack of motivation
Consequences
Medium:Less productivity,members are more concerned
about how they are supposed to nish this
Probabiltiy
High
Strategy and actions
Reduce:Use scrumeectively,this way members will see
the progress that has been made after each sprint and get
a feeling of accomplishment
Deadline
Continues
Responsible
Everyone,(mainly Project manager)
Second responsible
-
Risk:Interfering work
Risk number
6
Activity
All
Risk factor
Interfering work from other subjects
Consequences
Medium:Less work on the project
Probabiltiy
Medium
Strategy and actions
Reduce:Plan ahead,make sure we delegate work so ev-
eryone knows what is supposed to be done and can plan
other subjects accordingly
Deadline
Continues
Responsible
Project manager
Second responsible
-
Risk:Technical problems
Risk number
7
Activity
Sprint output
Risk factor
Technical problems
Consequences
Medium:Loss of equipment will result in a lessened abil-
ity to carry out sprint work if we can't access similar
backup devices
Probabiltiy
High
19
2.PROJECT PLAN
Strategy and actions
Reduce:Try to have similar backup devices ready so that
we can still do some of the work.
Deadline
Continues
Responsible
Implementation managers
Second responsible
-
Risk:Lack of equipment
Risk number
8
Activity
Testing
Risk factor
Lack of equipment
Consequences
Medium:Loss of ability to test the modules without
members who has android phones
Probabiltiy
Medium
Strategy and actions
Accept:We will just need to accept this fact,as we can't
request everyone to buy an android phone.With good
planning for testing this won't be a problem
Deadline
All sprints
Responsible
Test manager
Second responsible
-
20
Chapter 3
Prestudy
This section will introduce us to the many dierent aspects of our prestudy.
For a project of this size,one very often has to get familiar with new concepts,
techniques and methodologies,of both technical and managerial art.In our case,
we were to develop an application for the Android operating system,using the
Scrum development framework,programming for augmented reality and so on.
The majority of the group had never done any programming for the Android
from before,no one knew Scrum very well and no one had any experience with
developing applications for augmented reality.From this,it was pretty clear
that we had to do a lot of studying on both the technical parts as well as the
managerial parts of the project.We will try our best to introduce these concepts
and describe the result of our extensive studies.
Purpose
The purpose of the prestudy is to introduce us to the many aspects of augmented
reality,and especially concerning augmented reality on smart phones.In addition
it serves as an introduction to the dierent methodologies and various technologies
that are relevant to this project.
Scope
The prestudy is a part that can be possibly huge,and it was necessary to limit
our focus area to the parts we thought the customer wanted us to have a closer
look at.That is why we have brought an emphasis on the augmented reality
on smart phones,and on a few image recognition algorithms that had a suitable
license that we could use.We also had a look at dierent methodologies so that
we could make an informed choice between them for this project.
21
3.PRESTUDY
3.1 Augmented reality
Augmented reality is the set of techniques that allow virtual elements to be xed
into reality.These virtual elements can use the ve human senses,the most
common application using images,and sometimes sound.Augmented reality has
to combine these three features:
 combining real and virtual reality
 interactive in real time
 registered in 3D
3.1.1 The beginning of augmented reality
The rst applications of augmented reality we can nd,are in the army.In
the cockpit of the plane,or in the helmet of the pilot,he can see several pieces
of information,overlaid on his current view.Then,some medical applications
appeared,helping doctors to understand some phenomena,or helping with high
risk and hard surgery operations.
Another application that is quite old and famous,is the overlaying of line and
information onto the sport grounds,such as in American soccer or ice hockey.
Figure 3.1:One example of the rst applications of augmented reality[1]
22
3.1 Augmented reality
We can see that before the smart-phones era,augmented reality was either
reserved to experts (doctors) or not controlled by the nal user (watching a sport
match).
3.1.2 Dierent augmented reality degrees
We can distinguish between three very dierent ways of doing augmented reality:
1.Using posistioning information
The application mixes information relative to where the user is or at the
place the user is looking at.It overlays the information it can retrieve
on the screen,concerning his position.See below for a screenshot of the
Wikitude World Browser,which allows some web search from this kind
of information.Using this technique,it is hard to make the reality and
the virtual elements match,the application has no idea of where exactly
the pointed object or building is on the user's screen.We can also argue
that it is hard to nd the users location accurately,for example with the
GPS system,the accuracy is about 15 m.Of course,depending on the
application,this could be sucient.However,the time constraint is well
respected with these solutions,so we can speak of real time applications.
2.Using markers
(a) Fiduciary markers
A duciary marker is a simple shape,strongly contrasted.Usually,
one can use a simple black and white gure.Squares are often used,
but not excusively.
Figure 3.2:Example of duciary markers[2]
23
3.PRESTUDY
These markers are easily and quickly recognized by a computer and a
camera.Here,the applications can localize precisely tracked objects,
and then integrate the information on it,in real time.The only prob-
lem here is that you can not put such a marker on each building of a
city,for instance.This marking process can be long and inappropriate,
for example in some museums on would like to help visitors without
vitiating the exhibit.
(b) Other markers
One way to get round this scaling problem is to use the reality as a
marker,which is possible in some situations.For instance,look at this
application,used by Tissot to sell watches:
Figure 3.3:Tissot use augmented reality to let the customer to try the
watch[3]
In this application,the user just put a wristband which is easily rec-
ognized by the computer,compared to ducial markers.Then he can
try all the brand's watches,quickly and easily.Another application
has been realized for toys'advertising,the marker being the toy box,
which customers show in front of a camera,to see on a screen what
does the toy look like in reality.This kind of markers is a good solution
to combine time eciency and ease of use,but we can not use them
with objects such as buildings,or exhibits.
3.Using image recognition
The limit between markers and image recognition is a bit fuzzy.For exam-
24
3.1 Augmented reality
ple,a lot of facial detection applications use the eyes as markers.It is very
easy to recognize eyes on somebodies face because it is two zones darker.
Image recognition use a camera and a computer to extract the main fea-
tures from a picture,and compare it to some other data that we acquired
earlier.Of course,this way one can know precisely where the object is,
without having marked it before.The problem here is the time required for
a computer to recognize an object.Further,this recognition can be harder
depending on the camera angle,or the light conditions.
3.1.3 Augmented reality using smart-phones
With the development and democratization of smart-phones,mobile based appli-
cation of augmented reality became more and more present.The latest smart-
phones include GPS,camera and compass,so it is easy to get basic global posi-
tioning.
Figure 3.4:Wikitude World Browser,on one of the most famous smart-
phones,iPhone 3GS,using GPS and solid state compass[4]
Smart-phones are complete devices,on which one can use all the techniques
seen above.That is one of the reasons why augmented reality is expanding very
quickly.But there are some limitations to the use of a smart-phone,you have
25
3.PRESTUDY
to keep your phone/camera in front of you while walking,which is not very nat-
ural,and it does not allow you to use your hands.You are focused on your
screen and have to choose between looking at the reality directly or through the
screen.Other solutions exist and they might be considered,such as the Sixth-
Sense project,which use a mini-projector to project additional information either
on you (for instance a hand) or on the walls or objects in front of you.The cam-
era is situated around your neck,so you don't have to keep it elevated all the time.
Figure 3.5:[5]
The smart-phone solution must not hide all the other ones,because it is not
as natural as it seems to use its camera.
What is and what is not augmented reality
All the ways of doing augmented reality have a common point:The mix of reality
and virtual.The most known way to do that is on a picture,but it is not the
only one,and there is still some new features to develop in applications.At the
same time,using the techniques presented above does not mean doing augmented
reality.It is a fashion name nowadays,and lots of project are mentioned to be
augmented reality although they are not.For example,some cities provide QR
Codes,which redirect to a website if you point at it with a smartphone.In these
projects,QR Codes are placed on famous buildings and places,and the website
you are redirected to explains you the building history,and some other extra
26
3.1 Augmented reality
information.This kind of project is not augmented reality,because it does not
mix this extra information with the reality,it just links it.Again,in augmented
reality applications,reality and virtual have to be mixed.The more integrated
the extra information will be in reality representation,the closer we get to perfect
augmented reality.
3.1.4 Limitations
As we already mentioned,one problem is that it is not that natural to walk with
your phone camera pointing everywhere.Some tools exists to avoid this problem,
such as augmented reality glasses (e.g.3D Visors),but these solutions are not
used by a representative part of people.Augmented reality should not physically
constrain the user,just to add some virtual information.It must simplify his
life and not complicate it.Another problem that is rising in people mind,is the
freedom to look.When you visit an exhibit,you might want to discover it by
yourself,maybe you will be interested in some special information,not in the main
subject.Looking through your camera,following the instruction,you become
more passive,you have less choice concerning the information you want to look
at.This problem has to be solved developing original application,integrating
the user's wishes,and giving him the choice to be active.Concerning these
applications helping you walking through a city and showing you the best stores,
the danger is that information about a big store might be hiding information
about little ones.If we consider this kind of system based on advertising,we
understand this problem better,how can we be sure that the application answers
our wishes only,without considering the stores'ones?
3.1.5 Conclusion
Along this state of art,we sawthat we have a long way to go before we have under-
stood everything that augmented reality can oer.If we examine the techniques
rst,there are some improvements to make concerning the image recognition re-
sponse time and the precision of the localization tools.Concerning the support,
the smartphone is not yet popular enough to see the impact of some application
on the population's habits,we will have to wait a little for the smartphone de-
mocratization.Further,it is not as natural as it seems to use it,but there are
some other support to provide augmented reality.Concerning the applications
themselves,they have to involve the nal user,to make him active,taking his
mind and habits into account.
Reference for the whole section:[8]
27
3.PRESTUDY
3.2 Object recognition
Object recognition is a sub-eld of the more general computer vision eld.The
task of object recognition is,given an image of an object,recognise this object in
another image,possibly rotated or scaled.The general case of object recognition
is still a hard task,requiring lots of computation power and time.In our project
we have focused our attention on two main object/image recognizing algorithms
/descriptors,SIFT and SURF.
3.2.1 SIFT
Scale-invariant feature transform (SIFT) is an algorithm to detect and describe
local features in images.This algorithm is used in computer vision and for ap-
plications that include object recognition,image stitching,3D modeling,gesture
recognition,robotic mapping and navigation,match moving,and video tracking.
It was published by David Lowe in 1999.
The SIFT transforms the image into some local feature vectors.These vec-
tors have a dierent features that convert in a distinctive and invariant to any
scaling,rotation or translation of the image.This algorithm can also be used to
nd the correspondences between dierent views of the same scene.These local
features are stored in descriptors that try to describe important areas of certain
image variables.This algorithm has a big computational cost but a successful
implementation can be used in a real-time application because the most high cost
operations apply only to certain function as those have passed an initial screening
test.SIFT is divided into four distinct phases:Scale-space extrema detection,
Key-point localization,Orientation assignment,and Key-point descriptor.
Scale-space extrema detection
This phase is responsible of nding the rst set of points of interest (key-points)
in the image.These key-points have to be ltered because some of these will not
meet certain requirements.
Key-point localization
This is the second SIFT phase.It focuses on storing all the information available
for each key-point.For each key-point,this phase saves the position in the scale
pyramid where it was found and its position on that scale.With this information
it will exclude key-points according to two criteria,because these points would
be very sensitive to them.The rst criterium is that low contrast points are
discarded.The second criterium is if it is located along the edges,since responses
from edge tends to be very high,the point will not be robust.
28
3.2 Object recognition
Orientation assignment
This phase will focus on calculating the key-point orientations.Once one has
them,one can build the descriptors in a way which is rotation invariant.Then
one store the following information in every image region:location,octave,scale
and main orientations.
Keypoint descriptor
The parameters we obtained above form a 2D coordinate system.This system
describes the local information about each image region and for this reason pro-
vides invariance to the same parameters.The next step is to obtain a descriptor
for each interest area that will also provide robustness to the system.One must
perform a series of modications to achieve greater robustness against possible
changes in lighting.The goal is to be invariant along three types of light variation:
brightness,contrast and nonlinear variation.
3.2.2 SURF
SURF:Speeded Up Robust Features is a scale- and rotation invariant interest
point detector and descriptor.It was rst presented in a paper by Herbert Bay,
Tinne Tuytelaars and Luc Van Gool in 2006.SURF substantially outperforms
other similar methods of image feature detectors/descriptors e.g.SIFT,with
regards to repeatability,robustness,distinctiveness,and speed.[9]
The basic features of the interest point detection part of SURF is the integral
image,the hessian matrix,the scale space construction and the interest point
detection.The detected points are then given a reproducible orientation,and a
64/128 dimentional vector with descriptors.
Keypoint detection
This is the part of SURF where the SURF points are detected,using a basic
Hessian-matrix approximation.[9] This is used in combination with a data struc-
ture called integral image,or sum area tabel.This is just an intermediate image
representation,for fast calculation of pixel intensities in certain rectangles.Given
an input image I and a point (x,y),the value of the integral image at point (x,
y) is then calculated by the sum of the values between the point and the origin.
I
P
(x;y) =
ix
X
i=0
jy
X
j=0
I(x;y) (3.1)
29
3.PRESTUDY
1
Getting the pixel intensity sum in any sized rectangle is now just an easy
constant time calculation,by getting the four points of a rectangle,A,B,C,D.
Then the sum of pixel intensities in the rectangle is:
A+D(C +B) (3.2)
SURF uses integral images to perform fast image convolution and this is one
of the reasons SURF is faster than SIFT [9] [10].
Only points with a stronger hessian response than a provided threshold will
be considered for further description.
Keypoint description
The SURF point description is the other part of SURF.The rst step of this part
is to give the points a reproducible orientation.First haar wavelet responses[11]
around the detected point is produced.Then a sliding window of size

3
is rotated
around each point.Haar wavelet responses are then summed up,this creates a
vector which is the dominat orientation for that point.
Then the actual descriptor of the point can be found.The 16 squares are
constructed around the SURF point,then the haar wavelet response in horizontal,
and vertical direction is computed,and stored in the description vector.The
absolute sum of the values of the responses are also added to the description
vector.This results in a vector of lenght 64.
The usefulness and robustness of this detector and descriptor was demon-
strated by Bay et al in their paper Speeded-Up Robust Features(SURF).[9] In
this paper they show how SURF preformes with respect to other image recogntion
schemes.
3.2.3 Why we chose SURF
The decision to use SURF was done for a couple of reasons.The rst one is
the limited processing power we have available on mobile phone.This is a good
reason to choose not to use SIFT.As was shown by Bay et al in [9].The SURF
detector and descriptor runs signicatly faster than the SIFT counterpart and is
more robust.This constitutes the greatest reason to choose SURF over SIFT.
The second reason is that the customer wants a general image recogntion module,
and that is why we have choosen to look at SURF and SIFT primarily.There are
other good image recogntion algorithms which can be used when the case is not
1
where I(x,y) refers to the pixel intensity of the pixel at x,y
30
3.3 QR codes
as general.Haar cascade codes[12],countour detection,and corner detection[13],
are a couple of useful schemes that comes to mind.
3.3 QR codes
QR Code is a barcode consisting of a matrix of black and white cells.These cells
are usually called\modules",and the module conguration as well as the number
of modules is determined by the version of the QR Code.The barcode can carry
various amount of data,ranging from 152 to 23 648 data bits with version 1
and 40 respectively.A higher version number is not necessarily better,but a
code consisting of more modules.This equates with a spatially more extensive
barcode if kept at the same scale.All QR Codes are square,and the specic
conguration of rows/columns is called a version.
A standards-compliant QR Code also provides ECC capability,with a choice
between four levels.This is accomplished by adding Reed-Solomon code to the
existing data.The dierent levels denes the number of codewords dedicated to
ECC.
The information encoded in the code could be both text or data.For our
purpose,encoding of a short text or a numeric string should be sucient.Storing
enough information in the bar code itself,may be possible in some situations,
but encoding a key string is more advantageous with regards to the exibility of
updating information.A solution using bar codes of any kind,should only do so
in order to grab the\actual"data from some kind of database.That way the
service provider has a single point of update,and the propagation is immediate
unless data is cached on the client side.Even if this is the case,it is trivial to limit
the database queries to\Do I need to update the information in my cache with
timestamp xxxx",and defer fromreceiving updates unless the cached information
is stale.
QR Code is standardized by ISO and JIS,among other standards bodies.
However the ISO-specication[14] is not available for free,although the standard
itself is.
A possible challenge is to nd a good compromise between the size of the bar
code and the required maximum distance between the camera and the bar code.
This depends on the amount of noise in the picture as well as the CCD-resolution.
Regardless of that,the camera has to be positioned relatively close to the bar
code { depending on the size of the code.
The use of QRCodes is widespread and there is a wealth of processing software
for most platform,including Android.There is certainly not a lack of o-the-shelf
software,and through the use of specic features in the Android system
2
makes
2
Intents
31
3.PRESTUDY
it easy to plug existing bar code reader software into any application if needed.
Data matrix is a competing standard for matrix barcodes.Unlike QR Codes,
the bar code does not have to be a square.However for our application it won't
make any dierence.They both serve the same purpose,and the same technical
restrictions apply to both of them.
Because there's a wealth of QR code-reader software available for Android,it
makes perfectly sense to call on QR Code reader software.
3.4 Target platform/The choice of platform
It is a given that the client side part of the software stack,is going to run on
some kind of mobile platform.Exactly what kind of platform that would be,was
an open issue at the beginning of the project.
These days,mobile phones are omnipresent,and advanced features like high-
resolution displays,high-performance graphics processors,fast CPU's and dedi-
cated signal processors for audio and video decoding have becoming increasingly
common.The feature gap between the most feature-rich { and the least expensive
models { has coined the term\smart phone".Our focus will be on such phones,
but there are a variety of options to choose between.
3.4.1 Choice of software platforms
The choice of what software platform what will be the target of this (client-
side) product is naturally characterized by the team members own knowledge
and impression of contemporary systems.In reality,the following platforms were
considered:
 Symbian
 Android
 iOS
3
The team never considered other alternatives,although the systems are by no
way exclusive to the mobile environment.
What follows,is a brief description of the platforms.Their relative merits
and disadvantages are in no way evaluated at depth,but the discussion serves as
a description of the alternatives from the team-members'standpoint.
3
This was usually called iPhone OS before the launch of iPad
32
3.4 Target platform/The choice of platform
Symbian
The Symbian operating system (and framework) has it roots in the EPOC-
operating system[15].Initially developed in the latter part of the 1980s by Psion
corporation,it grew in popularity throughout the 90's,and it was adopted in
product made by major corporations like Nokia,Sony-Ericsson and Motorola.
The\standard"nature,and advantage of having one OS on a multitude of prod-
ucts from dierent manufacturers is likely to be one of the reasons why Symbian
became increasingly popular.
Symbian is still a signicant player in the market,but has met erce com-
petition from Android and iOS.Even Motorola { a manufacturer that has used
Symbian on their products { have released at least three phones with Android.
It was hard for the team to assess the advantages of developing for Symbian,
because no members have any experience developing for it.However the use of
C++ and the QT framework makes it a strong contender feature-wise.
iOS
Since the introduction of the original iPhone in 2007,it has had an increas-
ing in uence in the smart phone market.Developers of iOS applications enjoys
relative homogeneity in the hardware.While this is generally true,there is a
dierence in hardware between the rst generation iPhone and the fourth gen-
eration.The screen resolutions spans from 320x480 to 960x640 on iPhone,and
1024x768 on iPad.Apple delivers a closely integrated product when combined
with the iTunes application.iTunes is the main,and only way for end-users to
purchase and download applications onto their phones and iPods when at their
PC.It is also possible to do the same through dedicated software on the mobile
hardware.
With respect to how familiar the teamis with the iOS SDK,no members have
any experience with developing for the iPhone,iPod Touch or iPad.Moreover
Objective C is the primary language if one are to develop iOS-applications.While
a programming language itself is nothing but a tool (thus no reason to get to
passionate about it),the truth of the matter is that Objective C is not widely
used outside the\Apple environment".It is possible to compile Objective C code
with mature and decent compilers like Clang and GCC { the latter being the
primary choice at this moment
4
.However,the libraries that give the applications
the iOS/Mac-"feel"are both proprietary and only available for OS X and iOS.
The iOS SDK is only available for OS X,and Apple's own IDE { XCode {
4
Calling Clang a\mature"product is stretching the truth,but we are fairly certain that it
will substitute GCC in OS X in some not too-distant future.(Of reasons that would go way
beyond this text)
33
3.PRESTUDY
provides a variety of tools necessary for iOS development.The availability is a
problem.As it is today,OS X market shares are insignicant in the desktop
market.In addition Apple does not approve the installation of OS X on non-
Apple products.In fact,it is disallowed by the EULA[16].In any case,the
upfront costs are high unless the developer already has access to Apple PCs.
As for the cost,it is possible to download the SDK for free.But it is not possi-
ble to distribute applications without enrolling in Apple's development program,
with the price of 99USD per year[17].But then there's no extra charge in order
to sell the applications through iTunes App Store.Unfortunately any application
distributed through the App Store,has to go through a review by Apple.It is
generally held to be true,that the rules that decides what gets approved,is not
transparent.But the guidelines are in fact available[18]
Android
The last major contender in the mobile market is backed up by Google.While
by no means very comforting in itself,a large company like Google has the in u-
ence to funnel a huge amount of resources into the maintenance and continued
improvement of the Android OS.
Android applications are written in the Java language,and is compiled to
bytecode as is familiar to any Java developer.However a Sun Java compliant
JVM is not able to run the resulting class-les.Instead Android uses a dierent
JVMaltogether,called Dalvik.This is not a major hindrance in itself,but serves
as a reminder that a given program is not able to run on an ordinary PC outside
the Android emulator.
All the members of the teamare quite familiar with Java and the standard Sun
Java class library.This is an advantage that is not easy to neglect when choosing
target platform.In addition,the Android platform is open.Not only open as in
\open source"(but with limitations on\approved hardware"for access to Android
market);the SDK is available for all major platforms,including Windows,OS X
and GNU/Linux.With an appropriate plugin,the integration with the Eclipse
Java IDE is unsurpassed { when compared to the alternatives.
Due to Android's openness,it is available on a variety of hardware.This
means that the functionality of Android devices diers signicantly.This is not a
problemin this project,because the target is Android hardware with very specic
hardware capabilities.But the dierences in screen sizes,resolution and densities
has to be taken into account when developing any Android application.
Unlike iOS,it is possible to distribute Android applications outside the ocial
channels.The user can install Android applications,much like they can install
arbitrary software on a PC,by running an installation package or executable.
34
3.5 State of the art
3.4.2 Conclusion
Choosing Android as the target platform is a natural choice:Three members of
the group have an Android phone,and one more member is going to get one in
the nearest future.One member have a fair amount of experience with program-
ming on Android and two more members have some introductory experience.
But no one on the group have ever worked with an iPhone.Also the fact that
iPhone uses its own programming language while Android uses Java,that all the
group members have experience with,counts heavily in the decision.In a time-
constrained project like this,familiarity with required tools and language is a nice
addition to Android's other benets,like availability of SDK and easy distribu-
tion of software.All these arguments,as well as guidelines from the customer,
are the reasons why we favor the chosen platform.
3.5 State of the art
As a part of the initial description given to us by the customer,we were given the
task to\a) Present state-of-the-art for object recognition and augmented reality
for mobile phones (Android and iPone),including available commercial and open
source solutions"[19]
We have spent quite some time on this,discovered some nice applications and
algorithms that already exists.The scope of this task was a narrowed down in
the very beginning of the project,as we agreed with the customer (see 3.4.2) to
only focus on making an Android program.
3.5.1 Android Market
On the Google Android system,the main place to download new programs is
through Android Market[20].This is a market place intended for every program
made for Android,free as well as priced ones.Until recently,there were only
nine countries in which the Android users could buy an application through the
Market[21],while the free apps are available to a lot of countries,as listed in the
previous link.Of course,this may (and probably will) change,even though only
a few months ago it did not seem as if Google would focus on making users in,for
example Norway,able to buy apps from the Market[22].This rapidly changed
in September-early October,and in the rst half of October,Google introduced
paid apps in Norway and 17 other countries.[23]
There exist some alternatives to the Android Market,for example Adroia[24].
Adroia is,according to themselves,\en norskutviklet markedsplass for Android-
applikasjoner med betalingslsning.",or in English:a Norwegian-developed mar-
ket place including a payment solution for Android applications.It is made to
35
3.PRESTUDY
make it possible for Norwegian developers to sell their applications.[25] SlideMe.org
is another alternative market place for Android apps,which also contains a pay-
ment solution.[26] In addition,there are some available hacks which allows the
customer to simulate being in a country opened for buying apps on Android
Market,but this is a bit edgy and denately out of our scope.
Bearing this in mind,we have considered mainly applications that are available
for free,of which most are some kind of open source.The programs we have had a
special look at are QR-code readers (\barcode scanners"),Layar,Google Goggles,
Google Sky Map,Google Maps,Wikitude,ARToolKit and Surf.
3.5.2 Licenses
An important aspect to know about and bear in mind when talking about specic
program and frameworks available.Here,we give a quick introduction to the
most common licensing paradigms for software.First,an important distinction
between proprietary and open software.In proprietary software,the source code
of the program is secret and not shared outside the company or equivalent.In
opposition to this,in open source software,the source code is available for whoever
interested to read and study it.[27]
There are several approaches to open source software,and also a signicant
discussion about terms,names and so on[28],which we will not dig into here.
Instead,we will describe the most common open source licenses in use.By this
we mean the GNU General public license[29],the Apache Software License[30]
and the BSD license[31].All of these are free software,meaning that they give
the user four essential freedoms:[32]
1.Freedom to run the program regardless of the purpose.
2.Freedom to study how the program works and adapt it to own needs.
3.Freedom to distribute copies.
4.Freedom to release one's own improvements to public domain.
Another central concept here is copyleft.If a license implies copyleft,every
distributed program that uses this licensed program must apply the same license
itself,or an equivalent license.GNU GPL is seen as the most typical copyleft
license,and also the most commonly used one.[33] On the other hand,neither
BSD nor Apache license uses copyleft.[34] From this,everyone can use parts of
code licensed under one of these licenses in his commercial program not open
source itself,and thus,we take a special look at the available software licensed
under this type of license.
36
3.5 State of the art
3.5.3 Google Goggles
Google Goggles lets the user take a picture of something,and then it analyses
the image and uses the result of this analysis to search the web (of course using
the Google search engine) for it.They state themselves that\Google Goggles
works better with certain types of queries.Try taking pictures of books & DVDs,
landmarks,logos,contact info,artwork,businesses,products,barcodes,or text.
Currently,it's not so good when taking pictures of animals,plants,cars,furniture,
or apparel."[35] Unfortunately,Google Goggles is proprietary and not applicable
for non-Google developers.
3.5.4 Google Sky Map
Google Sky Map uses GPS,the clock and the compass of the phone to show which
stars and planets are in the sight of the phone.[36] Of course,this updates as the
user moves the phone around and looks in dierent angles et cetera.Also,the
program supports several dierent layers,allowing the user to specify what of the
information he is interested in.Google Sky Map,as Goggles,is proprietary.
3.5.5 Google Maps
Google Maps works in general much the same way as Google Sky Map,but
focusing on the earth and not the skies.In addition,Google Maps can use
mobile phone tower triangulation to locate the user,in case GPS is not activated.
This means that it uses the footprint from each mobile phone tower to nd an
approximate location for where you are.[37] Also,Google Maps is synchronised
with a database of shops and enterprises nearby,as well as other services from
Google and several dierent map layers.
3.5.6 Wikitude
Wikitude works in much the same way as Google Maps does.It uses GPS
and compass to locate the user,and then searches through a user dened set
of databases (so called\wikitude world"[38]) for hits in those near the user.
Among these\worlds"are well-known services as Wikipedia,Flickr,Foursquare,
Youtube,Twitter and many quite unknown ones such as\Irish pubs nearby"or
\1800TravelBooking.com".The results from the search may be shown on a map
(for this,it actually uses Google Maps),in a simple list or by using the phone's
camera.When using the camera,the discovered items from the search which is
located in front of the user is shown on the screen.This way,what hits are dis-
played to the user depends on the direction the user is holding the phone.Either
37
3.PRESTUDY
way,the user is shown what is found and in which wikitude world it originated.
She can also easily get information like how far away she is from the place and
often a picture as well.
3.5.7 ZXing
Later in this chapter,(3.3) we discuss QR codes,including what a QR code is and
how it works.The QR codes are well documented and standardized,and there
are many Android applications that read QR codes and return the information
encoded in them.One example of such is Barcode Scanner,which as of 6th
of October is the sixth most popular application on the entire Android Market
(number one is the earlier mentioned Google Maps).Barcode Scanner is licensed
under an Apache license[39],which states that\the Apache License allows the
user of the software the freedomto use the software for any purpose,to distribute
it,to modify it,and to distribute modied versions of the software,under the
terms of the license.
The Apache License,like BSD licenses,does not require modied versions
of the software to be distributed using the same license (in contrast to copyleft
licenses).[40] Taking this into account,we may take use of the source code of this
project for our purposes,and adapt it to our application.
3.5.8 Layar
Layar is an\augmented reality browser",available for both iPhone and Android.
One can say that Layar is the available application that is closest to what our
nal application will be,if we bear in mind our product backlog (Chapter 6)
and project description (Chapter 1.6),and notice that the constructors of Layar
describe their application this way:\The idea is simple:Layar works by using
a combination of the mobile phone's camera,compass and GPS data to identify
the user's location and eld of view,retrieve data based on those geographical
coordinates,and overlay that data over the camera view."[41].One can see that
this is quite similar to the requested features of our project,but on the other
hand,the application area diers signicantly from ours.Layar is meant for
outside use,nding local cafes,restaurants,shops et cetera,while our library will
serve as the platform for pervasive games.Nevertheless,it could be quite useful
to adapt at least parts of the Layar code for our application,though we are not
able to do so:the terms and conditions of Layar clearly states that\You agree
not to modify,copy,or create derivative works based on the Services.You will
refrain from selling,trading,reselling,leasing,renting,loaning and distributing
the Services for any purposes."[42]
38
3.6 Android versions
3.5.9 ARToolKit
ARToolKit is a software library that provides the ability for the user to see a 3D
overlay on a physical marker[43].This is usually a paper with some black and
white squares on it,and ARToolKit can then handle both orientation and moving
of the physical object.We do not see a clear link between the scope of this project
and the use of ARToolKit,but we recommend the customer to have a closer look
at it while developing new products based on our library.The ARToolKit can be
used at a later moment for providing some nice extra features,and here,we notice
that it has been ported to the Android platform,in a project called AndAR.[44]
This project has a GPL license,meaning that the use of it will require either
releasing the actual application under the same open source license,or making a
specic deal with the developer.The ARToolKit itself is also dened under this
license,but on their web page,they explicitly state\ARToolKit is made available
freely for non-commercial use under the GNU General Public License.Commer-
cial licenses to a professional implementation of ARToolKit are available for users
for whom the GPL is not suitable,or who require a higher level of support,or
who require customization or other specialist modictions.Commercial licenses
are administered by ARToolworks,Inc.,Seattle,WA,USA.Click here for more
information on ARToolKit licenses and your usage options."[42]
3.6 Android versions
The Android mobile operating systemcomes in many dierent versions.Since the
release of version 1.0 back in September 2008,there has been four major version
releases.These are the four main version that are still in use today.They are:
 1.5 (Cupcake)
 1.6 (Donut)
 2.0/2.1 (Eclair)
 2.2 (Froyo)
As of november 2010,Android version 2.2,codename Froyo,is the most recent
released Android version.It was released on 20 May 2010,during the yearly
google IO conference.Already by the end of the summer 2010 it had grown to
become the second lagest version in use.
Platform
Distribution
Android 1.5
7.9%
39
3.PRESTUDY
Android 1.6
15.0%
Android 2.1
40.8%
Android 2.2
36.2%
One big problem the last year has been that Android 1 and 2 had about the
same amount of users.So you could not target Android 2 without excluding half
of the users in the world.
But as the pie chart shows,77%of active phones are now running some version
of Android 2.And the rest is running 1.5 or 1.6.And this group is getting smaller
and smaller for every month.So targeting Android 2 is not longer a huge trade
o,it only excludes about 23% of users.
The next android version Android 2.3(Gingerbread) has been announced for
a late November/December 2010 launch.A detailed description of what features
this release will contain has yet to be conrmed as of mid November 2010.
3.7 Android NDK and JNI Bindings
3.7.1 JNI
The Java Native Interface is a programming framework that allows Java code
running in a Java Virtual Machine (JVM) to call and to be called[45] by native
applications (programs specic to a hardware and operating system platform)
and libraries written in other languages,such as C,C++and assembly.
3.7.2 Android NDK
The NDK is the native development kit.It allows programmers to compile native
C/C++ code to a shared library le.This shared library's functions can be
called from java by the use of JNI.The use of the NDK together with JNI can
signicantly increase the performance of an android program.But this does not