A NOVEL AJAX USER INTERFACE FOR TELEROBOTIC CONTROL

duewestseaurchinAI and Robotics

Nov 14, 2013 (3 years and 10 months ago)

134 views

A NOVEL AJAX USER INTERFACE FOR TELEROBOTIC CONTROL
_______________
A Thesis
Presented to the
Faculty of
San Diego State University
_______________
In Partial Fulfillment
of the Requirements for the Degree
Master of Science
in
Computer Science
_______________
by
Andy Ngo
Spring 2011


iii
Copyright © 2011
by
Andy Ngo
All Rights Reserved



iv
DEDICATION
This thesis is dedicated to my family and friends for their everlasting support and
encouragements.


v
ABSTRACT OF THE THESIS
A Novel AJAX User Interface for Telerobotic Control
by
Andy Ngo
Master of Science in Computer Science
San Diego State University, 2011

Recent improvements to the World Wide Web and wireless technologies have led to a
plethora of ideas and concepts that either improve upon existing implementations or are
original altogether. In this thesis, we propose a novel approach for an overall telerobotic
system for controlling the iRobot Create® mobile robot that will enhance a user’s experience
by being intuitive to use, ergonomically friendly, accessible, portable and scalable. This
novel concept involves the creation of Robajax; a Web application built using an AJAX
methodology. Previous applications for telerobotic control were PC based applications that
had to be installed as a monolithic desktop application. Telerobotic control involves a series
of mouse clicks to direct a robot’s trajectory. The Robajax Web User Interface, being a Web
application, renders local installation unnecessary, as the Robajax Web User Interface is
accessible to anyone over an Internet connection. Mouse clicking to direct the robot to move
is not needed as the user interface tracks mouse movement in a circular widget to direct
motion. Thus, users can easily control the iRobot Create through obstacles for an extended
period of time without witnessing hand fatigue, making the Robajax Web User Interface
more ergonomically friendly when compared to a PC application counterpart. This overall
system uses DHCP to acquire a dynamic IP address for the iRobot. A dynamic IP address
allows this system to be portable across different wireless networks because the iRobot can
be placed on any wireless network, including wireless free-Wi-Fi hotspots, and still connect
to the Robajax Web Application Gateway. Scalability is solved by using the Robajax Web
Application Gateway Web Service. The Robajax Web Application Gateway maintains a one-
to-many TCP session relationship with the Web browsers of remote users and a one-to-one
session relationship between the iRobot and the Web Application Gateway. The purpose of
restricting the number of sessions over the WLAN to a single session is to prevent the
streaming of redundant video to multiple parties, thus allowing the wireless channel to carry
a single video stream to the gateway with the highest possible frame rate or lowest possible
quantization value, depending on the available wireless data rate. Using this design approach,
the Robajax Web User Interface we have developed is scalable, portable, accessible and
ergonomically friendly. Design details regarding the use of AJAX, multithreaded SOAP-
based Web Services, and algorithms used to prevent deadlock conditions in Web-based
telerobotic control are presented.


vi
TABLE OF CONTENTS
PAGE
ABSTRACT ...............................................................................................................................v
LIST OF TABLES ................................................................................................................. viii
LIST OF FIGURES ................................................................................................................. ix
ACKNOWLEDGEMENTS ..................................................................................................... xi
CHAPTER
1 INTRODUCTION TO TELEROBOTICS ....................................................................1
 
1.1 Acronyms and Terminology ..............................................................................1
 
1.2 What is Telerobotic Control? .............................................................................3
 
1.3 Previous Research ..............................................................................................4
 
2 OVERVIEW OF THE SAN DIEGO STATE UNIVERSITY ROBAJAX
WEB USER INTERFACE DEVELOPMENT PLATFORM........................................8
 
2.1 Motherboard .......................................................................................................9
 
2.2 Compact Flash Memory .....................................................................................9
 
2.3 Digital Camera ...................................................................................................9
 
2.4 Ajax Based Control ..........................................................................................14
 
2.5 The Robajax Web Application Gateway .........................................................16
 
2.6 GlassFish ..........................................................................................................17
 
2.7 The iRobot .......................................................................................................17
 
3 USER INTERFACE DESIGN .....................................................................................20
 
3.1 Button Based UI ...............................................................................................20
 
3.2 Slider Based UI ................................................................................................22
 
3.3 Circle Based UI ................................................................................................24
 
4 PROBLEMS ENCOUNTERED ..................................................................................28
 
4.1 Incorrect Status Being Displayed.....................................................................28
 
4.2 Website Pictures not Being Loaded .................................................................28
 
4.3 Out of Order SOAP messages ..........................................................................29
 
4.4 SDSU Internal WLAN Firewall .......................................................................31
 


vii
4.5 Loss of Wireless Signal ...................................................................................34
 
4.6 The Robajax Web Application Gateway Deadlock Condition ........................36
 
5 TELEROBOTIC DESIGN...........................................................................................39
 
5.1 Circle UI Design ..............................................................................................39
 
5.1.1 Algorithm ................................................................................................39
 
5.1.2 Static Class Diagram ...............................................................................42
 
5.2 The Robajax Web Application Gateway Design .............................................42
 
5.2.1 Algorithm ................................................................................................42
 
5.2.2 Static Class Diagram ...............................................................................47
 
5.3 The iRobot Design ...........................................................................................47
 
5.3.1 Algorithm ................................................................................................49
 
5.3.2 Static Class Diagram ...............................................................................51
 
6 CONCLUSION ............................................................................................................56
 
7 FUTURE DIRECTIONS .............................................................................................58
 
7.1 Proportional-Integral-Derivative (PID) Controller ..........................................58
 
7.2 Centered Video View .......................................................................................59
 
7.3 Obstacle Avoidance .........................................................................................59
 
7.4 Intelligent Access Point Hopping ....................................................................61
 
7.5 Multiple User Support......................................................................................62
 
7.6 Security System ...............................................................................................64
 
7.7 Lazy Initialization ............................................................................................64
 
7.8 Message Protocol .............................................................................................65
 
REFERENCES ........................................................................................................................66
 




viii
LIST OF TABLES
PAGE
Table 2.1. Specification of the Migrus Motherboard ...............................................................10
 
Table 2.2. Specification of the Kingston 4GB Compact Flash Memory .................................12
 
Table 2.3. Specification of the Unibrain Fire-i™ Digital Camera...........................................12
 
Table 3.1. Button Based UI Summary .....................................................................................22
 
Table 3.2. Slider Based UI Summary ......................................................................................24
 
Table 3.3. Circle Based UI Summary ......................................................................................27
 
Table 5.1. Circle UI’s Static Classes .......................................................................................42
 
Table 5.2. The Robajax Web Application Gateway’s Static Classes ......................................48
 
Table 5.3. The iRobot’s Static Classes ....................................................................................54
 



ix
LIST OF FIGURES
PAGE
Figure 2.1. Overview of the Robajax Web User Interface system. ...........................................8
 
Figure 2.2. Picture of the Migrus motherboard. .........................................................................9
 
Figure 2.3. The Kingston 4GB compact flash card. ................................................................11
 
Figure 2.4. The Robajax Web Application Gateway eliminates redundant video traffic
between itself and the iRobot. ......................................................................................16
 
Figure 2.5. The iRobot will generate more traffic if the Robajax Web Application
Gateway is not used. ....................................................................................................18
 
Figure 2.6. The iRobot Create
TM
Vehicle. ...............................................................................18
 
Figure 3.1. Screenshot of the button based UI. ........................................................................21
 
Figure 3.2. How the iRobot turns based on the radius. ............................................................22
 
Figure 3.3. Screenshot of the slider based UI. .........................................................................23
 
Figure 3.4. Screenshot of the circle based UI. .........................................................................25
 
Figure 3.5. A decision diagram for tracking mouse movement within a web browser. ..........26
 
Figure 3.6. A SOAP message is sent immediately when the mouse enters the red
circle. ............................................................................................................................26
 
Figure 4.1. SOAP request messages being processed out of order. .........................................29
 
Figure 4.2. SOAP messages being processed by in order by using sequence ids....................30
 
Figure 4.3. New expected sequence ID due to an out of order SOAP message. .....................31
 
Figure 4.4. Connection initiator before implementation of the SDSU firewall. ......................32
 
Figure 4.5. SDSU internal WLAN firewall restricts PCs on the wired LAN from
initiating a connection. .................................................................................................32
 
Figure 4.6. The iRobot is now the connection initiator to adapt to SDSU firewall
restrictions. ...................................................................................................................33
 
Figure 4.7. Sample GetStatus JSON message. ........................................................................34
 
Figure 4.8. Heartbeat between the WriteThread and the iRobot is still alive. .........................35
 
Figure 4.9. Connection restart due to a lost heartbeat message. ..............................................35
 
Figure 4.10. Deadlock scenario caused by a lost TCP connection. .........................................36
 
Figure 5.1. A UI handshake sequence diagram. ......................................................................40
 
Figure 5.2. A UI send command sequence diagram. ...............................................................40
 


x
Figure 5.3. Sample of a SOAP request message. .....................................................................41
 
Figure 5.4. The circle UI’s static class diagram. ......................................................................43
 
Figure 5.5. The Robajax Web Application Gateway start server sequence diagram. .............44
 
Figure 5.6. The Robajax Web Application Gateway close server sequence diagram. ............45
 
Figure 5.7. Sample of a handshake SOAP request message. ...................................................46
 
Figure 5.8. The Robajax Web Application Gateway handshake sequence diagram. ..............46
 
Figure 5.9. The Robajax Web Application Gateway forward sequence diagram. ..................47
 
Figure 5.10. The Robajax Web Application Gateway write task sequence diagram. .............47
 
Figure 5.11. The Robajax Web Application Gateway static class diagram. ............................48
 
Figure 5.12. The iRobot open port sequence diagram. ............................................................49
 
Figure 5.13. The iRobot open socket sequence diagram. ........................................................50
 
Figure 5.14. The iRobot initiating a TCP socket connection. ..................................................51
 
Figure 5.15. The iRobot parse forward sequence diagram. .....................................................52
 
Figure 5.16. The iRobot task sequence diagram. .....................................................................52
 
Figure 5.17. The iRobot close socket sequence diagram. ........................................................53
 
Figure 5.18. The iRobot static class diagram. ..........................................................................55
 
Figure 7.1. A view of the sensor assembly platform. ..............................................................60
 
Figure 7.2. A sample of the histogram grid. ............................................................................60
 
Figure 7.3. Timeout penalties incurred because of a lack of multiple user support. ...............63
 



xi
ACKNOWLEDGEMENTS
I would like to thank my family for their encouragement and support. Without their
encouragement I would have never would have pursued an advanced degree. I would also
like to Dr. Christopher Paolini for giving me guidance and supervision throughout this thesis.
Furthermore, I would also like to thank Dr. Bill Root and Dr. Carl Eckberg whose courses
have helped me immensely to complete this thesis.


1
CHAPTER 1
INTRODUCTION TO TELEROBOTICS
In this thesis, we developed a telerobotic system for controlling the iRobot Create
TM

Vehicle® consumer robot appliance [1]. This chapter introduces the acronyms and
terminology that will be used throughout this thesis and define what telerobotic control is and
discuss previous work on telerobotic control.
1.1

A
CRONYMS AND
T
ERMINOLOGY

User Interface (UI) – A place where interaction between humans and machines occurs. The
goal of interaction between human and a machine at the user interface is effective operation
and control of the machine, and a feedback from the machine which aids the operator in
making operational decisions.
Graphical User Interface (GUI) – A type of user interface item that allows people to
interact with programs in more ways than typing; with images rather than text commands. A
GUI offers graphical icons and visual indicators, as opposed to text-based interfaces, typed
command labels or text navigation to fully represent the information and actions available to
a user.
Widget – An element of a GUI that displays an information arrangement changeable by the
user, such as a window or a text box. The defining characteristic of a widget is to provide a
single interaction point for the direct manipulation of a given kind of data. Widgets are basic
visual building blocks which, combined with an application, hold all the data processed by
the application and the available interactions on the data.
HyperText Markup Language (HTML) – Predominate markup language for web pages.
HTML provide a means to create structured documents by denoting semantics for text such
as headings, paragraphs, lists, links, quotes and other items. It allows images and objects to
be embedded and can be used to create interactive forms.


2
Extensible Markup Language (XML) – A set of rules for encoding documents in a
machine-readable form; XML was originally designed to emphasize simplicity, generality
and usability over the Internet.
Web Services Description Language (WSDL) – Is an XML-based language that provides a
model for describing Web Services. The WSDL define services as a collection of network
endpoints, or ports. The WSDL specification provides an XML format for documents for this
purpose. The abstract definitions of ports and messages are separated from their concrete use
or instance, allowing the reuse of these definitions.
Simple Object Access Protocol (SOAP) – A protocol specification for exchanging
structured information in the implementation of Web Services in computer networks. SOAP
uses XML as its message format.
JavaScript Object Notation (JSON) – A lightweight text-based open standard designed for
human-readable data interchange.
Web Service – Is typically an application programming interface (API) that is processed via
HTTP and executed on a remote system hosting the requested service. Web Services tend to
fall into one of two camps: "big Web Services" or SOAP based Web Services and
Representational State Transfer (RESTful) Web Services. A "Big Web Service" uses XML
messages that follow the SOAP standard and have been popular with traditional enterprises.
In such systems, there is often a machine-readable description of the operations offered by
the service written in WSDL.
JavaScript – A scripting language typically used to enable programmatic access to
computational objects within a host environment. JavaScript is primarily to provide enhanced
user interfaces and dynamic web websites.
XMLHttpRequest (XHR) – An API available in web browser scripting languages such as
JavaScript. XHR is used to send Hypertext Transfer Protocol (HTTP) or Hypertext Transfer
Protocol Secure (HTTPS) requests directly to a web server and load the sever response data
directly back into the script.
Asynchronous JavaScript and XML (AJAX) – A methodology to develop web
applications, this methodology consists of using cascading style sheets (CSS), dynamic
document object model (DOM), asynchronous XHR and the use of JavaScript to manipulate
the DOM [2]. Web applications developed using AJAX methodology places less strain on the


3
server when compared to conventional web applications written on complete HTML pages.
This is because, user actions on conventional web applications causes the entire page to be
re-loaded by the server. User actions with web applications using AJAX only re-load the
changed information.
GlassFish – An open source application server developed by Sun Microsystems for the Java
Enterprise Edition Platform. It uses a derivative of Apache Tomcat as the servlet container
for serving Web content, with an added component called Grizzly and uses Java New I/O for
scalability and speed.
The iRobot – The iRobot Create
TM
Vehicle, a mobile robot based on the Roomba platform (a
commercial robotic vacuum cleaner). Unlike the Roomba, the iRobot Create was designed
for robotics development [1]. In place of the vacuum hardware, the iRobot Create includes a
cargo bay which houses a 25 pin port that can be used for digital and analog input and output.
The Robajax Web User Interface – Is a website (http://www.robajax.com), where any user
can go to remotely control the iRobot. This website (hosted on godaddy.com) masks
maxwell.sdsu.edu, where the code for the UI based control resides.
The Robajax Web Application Gateway – A SOAP based Web Service that acts a gateway
between the many browsers logged onto the Robajax Web User portal and the iRobot
appliance.
Maxwell.sdsu.edu – Is a server that hosts GlassFish and the JavaScript code for the Robajax
Web User Interface.
1.2

W
HAT IS
T
ELEROBOTIC
C
ONTROL
?
Telerobotic control is the area of robotics concerned with the control of robots from a
distance, chiefly using wireless communications. Telerobotics is a sophisticated form of
teleoperation (doing work at a distance) [3] and combines two major subfields: teleoperation
and telepresence (feeling like you are somewhere else). Two major components of
telerobotic control are the visual and control applications. A remote camera provides the
visual representation of the view from the robot. This remote camera will offer an operator
the robot’s view, allowing intuitive control of the robot. Telerobotic control has not been
fruitful as time delay, resolution, and bandwidth required have only recently been adequate to
give the ability to control a robot in any meaningful way. This is especially true in the case of


4
ground-based control of space robots where the effects of time delays and bandwidth
restrictions are extremely significant [4]. In such a space system, the time delays (from the
ground to the space station) can be up to 3 seconds and this delay plays a major role in the
stability control of the entire system. Bandwidth restrictions limit the availability of video
and audio channels. Therefore, less information passes between a user and the space-based
robot.
1.3

P
REVIOUS
R
ESEARCH

The field of telerobotics has been around for the past 40 years. Many research
endeavors have been dedicated to this field. Computer science and engineering curricula
commonly use the iRobot Create
TM
Vehicle (the iRobot) because of its low cost. The iRobot
is particularly attractive because of the easy to learn Create Open Interface (OI) that uses the
RS232 serial protocol for sending command packets and is the platform of choice to
stimulate student interests and improve retention rates in many colleges. Weiss et al.
identified that the iRobot as an ideal platform for researches seeking affordable and durable
solutions for teaching introductory robotic courses [5]. Wellman et al. at the University of
Alabama found that using the iRobot in Computer Science (CS) courses has helped boost
enrollment and improve retention rates in the underrepresented minority groups and women
[6]. Other research by Dodds in the Harvey Mudd College CS Department has found that
using the iRobot positively stimulate students interest in CS and improves retention rates,
especially in female students [7]. One of the more challenging areas students have focused
attention on is the integration of visual classification to automated robot task planning where
image profiles consisting of pixel-intensity sums computed across subsets of a streaming
video sample can be used for visual odometrical control of the iRobot, where the use of
images taken from the iRobot's surroundings is used estimate change in position over time
[8].
Telerobotics is not just used to stimulate student interests but this field also has real
world applications as well. One of the most famous examples is the twin Mars Exploration
Rovers (MER) project. The MER vehicles have a very sophisticated suite of autonomous
capabilities to explore unknown terrain and automatically collect data, such as scientifically
interesting events [9]. The vehicles use hazards to populate a surrounding map of terrain


5
traversability; from this map the best route is chosen to be executed. The vehicles repeat the
process of detecting hazards and then executing the appropriate motion command until the
goal is reached. In order to maintain a local navigation map the vehicles combine wheel
odometry, linear accelerations, angular velocities and optional vision-based pose estimates
[10].
Other research has focused on developing a system that combines both human
interactions along with autonomous control, a concept called supervised autonomous robots
[4]. In such a system there would a functional distribution of tasks between a user and the
robot. The user would be responsible for: sensing or perceiving the work volume, reasoning
and understanding the work volume, building a world model, generating a complete task
description, task analysis and understanding, task decomposition, error detection, anomalies,
error recovery, and re-planning. The robot would then be responsible for collision free
trajectory generation and execution of run time functions and skills.
Another use of telerobotics is on the Internet, focusing on the telesurgery application
domain. King et al. developed a protocol called Interoperable Teleoperation Protocol (ITP) to
solve the problem of using a shared data interface between telerobotic master-slave systems
[11]. To simplify cooperation, ITP uses a low-overhead binary UDP packet structure.
Telesurgery master-slave systems are connected with a master manipulator being operated by
a surgeon in order to control the slave (patient-side robot).
Many research projects have taken advantage of the Internet to develop highly
accessible telerobotic systems. Hartfiel et al. designed a system to tackle issues of
semiautonomous telerobotics, communication time delays, sensory feedback and assignment
and modification of tasks [12]. In Hartfiel’s research an electronic storyboard was the chosen
interface because of the cooperative environments between humans and robots. The story
board provided the following benefits: real time group problem solving, easy visibility of the
system’s main functions, multiple operations can be performed simultaneously, and the
interface encouraged user interaction and ease of understanding. Zakaria et al. decided on an
Internet based system to control their XR-4 series Robot Arm because of the many
advantages the Internet had to offer: the Internet has extensive geographical reach, is network
and platform independent, is standards-based and open, and a wave of technological
developments, from high bandwidth networks to software technologies, is revolutionizing the


6
Internet [13]. Zakaria’s Internet based system consisted of a robot, a camera and a personal
computer. One of the biggest concerns was that the robot should not collide with other
objects to avoid damage to both the robot and the objects. The user interface for this
telerobotic system was a button based browser with a live video stream embedded in the
browser. To control the robot’s arms, a user would click buttons to perform a certain
command while watching a video stream to see actions take place in real time.
Using the Internet for teleoperation is not without challenges. The Internet faces
problems such as unreliable packet delivery and unpredictable latencies. Lakshmanan and
Rajkumar offered a solution to these problems in their research [14]. The main principle in
their research is to achieve reliable communication through redundancy, while maintaining
low end-to-end latency. They realized that individual links of the Internet can fail due to
network saturation or physical failures. Thus, they employed multipath routing techniques to
solve such problems. This was done by sending duplicate data to the same target through
different paths. Thus, the network’s communication reliability increases with the degree of
statistical independence between the chosen paths. Studies have been undertaken to
characterize the correlation between paths [15]. Proactive re-transmissions were used to fix
packet errors due to tail-drop effects at router buffers and interference spikes in the wireless
channel. The problem with waiting for acknowledgements before retransmitting data is that
the resulting average packet latency is normally unacceptable. This was solved by pro-
actively sending a duplicate copy of the data after a statistically estimated time delay. The
delay was needed to accommodate the durations of burst traffic in the network. To achieve
reliable low-latency communication they used proxy forwarding, which also fixed problems
caused by an unreliable wireless link. Long latency in the Internet is mainly due to long
distances, which prevents acknowledgment based communication over the entire path. A
proxy node connected through the wired network could communicate over the Internet with
the controller with high reliability and without requiring acknowledgements.
Another big field in telerobotics is autonomous mobile manipulating robots that
would automatically perform desired tasks without human intervention. Xu et al. attached a
robotic arm on the iRobot so it can be used to grasp objects [16]. The motivation behind Xu’s
work was that it could be used to help people with motor impairments. Thus, the robotic arm
attached would allow the iRobot to grasp objects on the floor. Some of the challenges


7
encountered were how to model such a robotic arm so that it can be used to grasp common
household objects that come in different shapes and sizes. In this research, the robotic arm
was modeled to emulate a human hand. To pick up an item people normally use their finger
nail as a wedge to get under the item or fingers or the palm of the hand as a surface to push
an object and apply force to the opposite side to grasp the item. Thus, the robotic arm in this
research has a flat plate with a leading wedge to slide under an object, and a 2-link compliant
finger to apply force to grasp an object.
Another research augmented the iRobot to enable service tasks that require spatial
reasoning such as delivery, surveillance, and map maintenance [17]. This setup consists of
mounting an Axis207 Ethernet camera and six Sharp® GP2D12 Infrared sensors (IR) on top
of the iRobot. The Wiimote game controller was used to direct the iRobot. Autonomous
exploration was handled by a spatial-reasoning module. The software developed by Koziol et
al. allowed a user to click on a goal location within the robot’s known map, and then the
system automatically planned and executed a path to a target location.
As mobile robots become more common, the need for robots to operate smoothly and
collision free arose. Research done by Snape et al. introduces a concept called hybrid
reciprocal velocity obstacles for collision avoidance among autonomous robots [18]. This
concept is an extension of reciprocal velocity obstacles [19]. Most avoidance algorithms
extrapolate where a moving object might be based on the object’s current velocity. However,
treating another robot as a general obstacle using such a technique may not be the best
approach because this technique does not take into account the reciprocity between robots
(robots might react the same way to each other). Hybrid reciprocal velocity is a mix between
reciprocal velocity obstacles and velocity obstacles [20]. Reciprocal velocity obstacles place
50% of the responsibility on one robot (robot A) and 50% on the other (robot B) to avoid a
collision, while for velocity obstacles it has one robot choosing a velocity that will avoid a
collision. The hybrid formulation has a consequence that if robot A attempts to pass on the
wrong side of robot B, full priority will be given to robot B to avoid collision. However, if
the correct side is chosen, full cooperation with robot B is assumed and both robots retain
equal priority.


8
CHAPTER 2
OVERVIEW OF THE SAN DIEGO STATE
UNIVERSITY ROBAJAX WEB USER INTERFACE
DEVELOPMENT PLATFORM
In order to start telerobotic control, a user will log on to the Robajax Web User
Interface (http://www.robajax.com) Web portal. The Robajax Web User Interface portal’s
main page contains JavaScript code that creates the UI needed to control the iRobot. When
the UI detects that a user wants to move the iRobot, the UI will construct a SOAP Request
and use an instantiated XHR object to transmit a SOAP Request to GlassFish (a SOAP
container). GlassFish will then marshal the SOAP Request and invoke an operation on the
Robajax Web Application Gateway Web Service to call the correct Web operation. The
Robajax Web Application Gateway will then construct a JSON object and send the serialized
JSON object via a TCP socket connection to the iRobot to perform the requested command
(see Figure 2.1).

Figure 2.1. Overview of the Robajax Web User Interface system.


9
2.1

M
OTHERBOARD

The motherboard used for this thesis is a Migrus C787 DCF-P with a 1.2GHz Eden
ULV Processor [21, 22]. The view (see Figure 2.2) and specifications (see Table 2.1) of the
motherboard are provided.

Figure 2.2. Picture of the Migrus motherboard.
2.2

C
OMPACT
F
LASH
M
EMORY

CompactFlash (CF) is a mass storage device format used in portable electronic
storage. For storage, CompactFlash normally uses flash memory in a standardized enclosure
[23]. For this telerobotic system a Kingston 4GB compact flash card is used (see Figure 2.3)
and its specifications are provided in Table 2.2.
2.3

D
IGITAL
C
AMERA

A Unibrain Fire-i™ Digital Camera is used in this telerobotic system [24]. It is a
lightweight camera suitable for mobile applications. The main features of this camera are two
400Mbps Firewire ports (up to 640x480 resolution) and a maximum of 30 frames per second.
A detailed specification of the camera is provided in Table 2.3.


10
Table 2.1. Specification of the Migrus Motherboard
Model Name Migrus-C787-1.2E-DCF-P
Processor VIA 1.2GHz Eden ULV Processor
Chipset
VIA CN700 North Bridge
VIA VT8237RP High Bandwidth Vlink Client South Bridge
System
Memory
1 x 240-pin DDR2 Module Sockets Support Max 1GB, 2.5V DDR2
400MHz SDRAM
8/16/32/64MB frame buffer
VGA
16/32/64MB frame buffer using system memory
Internal AGP 8X performance
128bit 2D/3D VIA S3 Graphics UniChrome Pro Processor
24bit true-color RAMDAC up to 250MHZ pixel
Expansion
Slots
1xPCI Slot (PCI 2.2)
Onboard SATA
2 Serial ATA Port provides 150MB/s (Max.) Data transfer rate
RAID0, RAID1 function Onboard IDE
2 x Ultra DMA 133 IDE Interface(Up to 4 IDE Devices & 133MB/s
transfer rate)
ATAPI IDE CD-ROM, CD-R,CD-RW and LS-120 Support
Onboard LAN
IEEE 802.3 10/100 BASE T Standard
Boot On LAN function
Onboard Audio
Software Audio Controller with Onboard 6-Channel CODEC Compliant
to AC97
3D Surround & Positioning, Full Duplex
Onboard TV
Out
VIA VT1622A TV encoder
NTSC/PAL TV system S-Video/AV Output
Onboard 1394
Compliant with 1394a OHCI specification Version 1.0
Compliant with IEEE 1394a-2000 standard, support 400Mbit/s
Back Panel I/O
1 PS2 mouse port
1 PS2 keyboard port
1 RJ-45 LAN port
1 1 Serial port
2 USB 2.0 ports
1 VGA port
1 RCA port (S/PDIF or TV out)
1 S-Video port
(table continues)


11
Table 2.1. (continued)

3 Audio jacks: line-out, line-in and mic-in (Horizontal, Smart 5.1
Support)
1 1394
Onboard I/O
Connectors
3 USB connector for 6 additional USB 2.0 ports
2 SATA Connector
1 Front-panel audio connector (Mic and Line Out)
1 COM2
1 Parallet port headers 25pin
1 Front panel audio connector
1 1394B1 connector
1 CD-in connector
2 Fan connectors: CPU/Sys FAN
ATX Power Connector
Form Factor
Mini-ITX
17 cm x 17 cm
Features
One 16-bit (PCMCIA2.1) card socket and one CF socket
PC2001 Design Guide Compliant
Compliant with PC card and Standard Release 8.0 Specification
Compliant with i82365SL compatible register set/ExCA
Plug and play support

Figure 2.3. The Kingston 4GB compact flash card.







12
Table 2.2. Specification of the Kingston 4GB Compact Flash Memory
CF card Name Kingston
Model Number

9904168 0 053.A01LF
3578146-1033191 X01
Capacities 4GB
Dimensions
1.43'' x 1.68'' x 0.13'' (36.4mm x 42.8mm x 3.3mm) - CF Type
I
Operating
Temperature 0
o
to 60
o
C / 32
o
to 140
o
F
Storage Temperature -20
o
to 85
o
C / -4
o
to 185
o
F
Voltage 3.3v / 5v
Standardized
complies with CompactFlash Association specification
standards
Small One-third the size of a full-size PC card
Easy Plug-and-play
Guaranteed Lifetime warranty
Versatile compatible with PC Card Type II adapters
Economical Auto sleep mode preserves system battery life
Table 2.3. Specification of the Unibrain Fire-i™ Digital Camera
Device type
IIDC 1394-based Digital Camera, V1.04 Specification compliant
Compliance v 1.04
Interface
IEEE-1394a (FireWire) 400 Mbps, 2
ports ( 6 pins )
Speed 400 Mbps
Sensor type
Sony Wfine* ¼" CCD with progressive scan (ICX-098BQ)
Scanning Progressive
Effective pixels ( H x V ) 659 x 494
Pixel shape ( H x V ) square, 5.6 x 5.6 µm
Color yes, RGB filtering, Bayer
Resolution TV-lines ( H x V ) 640 x 480
Optics
f 4.65 mm built-in
Focus Manual, from 5 mm to infinite
Iris Fixed
Horizontal view angle 42 º
Vertical view angle 32 º
Coating Anti-reflective
Picture
modes progressive VGA uncompressed, modes selection by 1394 link
(table continues)


13
Table 2.3. (continued)

Resolutions 640x480, 320x240, 160x120
Codings YUV, RGB, Monochrome
Video Modes / Frame rates 30fps 15fps 7.5fps 3.75fps
640x480 RGB-24 (24 bits) X X X
640x480 YUV 4:2:2 (16 bits) X X X
640x480 YUV 4:1:1 (12 bits) X X X X
640x480 Y-Mono8 (8 bits) X X X X
320x240 YUV 4:2:2 (16 bits) X X X X
160x120 YUV 4:4:4 (24 bits) X X X
Picture
settings
Settings selection by 1394 link
Gain Auto / Manual 0 - 30 dB
Shutter Auto / Manual 1/3400s – 1/31s
Black level Auto / Manual
Gamma OFF(linear) = 1 / ON (visual) = 0.45
Backlight compensation 6 modes + OFF
Sharpness OFF(linear) = 0 / ON (visual) = manual
White Balance Auto / Manual
Color Saturation Manual
Reference picture generator Bars and ramps, all modes. ON/OFF
Power
supply
8 to 30 VDC, software switched by 1394 link
Source By 1394 link / external jack input
Standby power 400 mW typ., 450 mW max.
Operation power 900 mW typ., 1 W max.
General Temperature / Humidity,
operation -10 to 50C / 20 to 80 % rel., non cond.
Temperature / Humidity,
storage -20 to 60C / 20 to 95 % rel., non cond.
Dimension (W x H x D ) 62 x 62 x 35 mm without clip
Weight 60 gr camera + 32 gr clip
Housing plastic, transparent
Attachment Spring clip, laptop suitable
Accessory Fire-i™ software included




14
2.4

A
JAX
B
ASED
C
ONTROL

The Robajax Web User Interface is an application developed using an AJAX
methodology. The AJAX methodology represents a group of technologies that can be used to
create a web application that communicates with a server without having to interfere with the
current stage of the web application. Jesse James Garrett explained the following
technologies that can be used [2]:
1. HTML or eXtensible HyperText Markup Languague (XHTML) and Cascading Sytle
Sheets (CSS) for presentation
2. Document Object Model (DOM) for dynamic display of and interaction with data
3. XML for the interchange of data, and Extensible Stylesheet Language
Transformations (XLST) for its manipulation
4. XHR object for asynchronous communication
5. JavaScript for bringing these technologies together
Development and user experience would differ drastically if developed under
standard HTML forms [25]. Using HTML forms the UI would contain buttons. Pressing a
button would generate either a POST request method or a GET method that would cause data
to be sent to the server to be handled, which in turn would cause the iRobot to perform the
requested action. Afterwards, the server would then respond by sending back a complete web
page, triggering the browser to re-render the website. Clicking buttons at a fast rate
(navigation through a series of obstacles) would cause many page refreshes. This would
saturate the server causing communication between the server and the web page to become
very slow. This is due to the increased communication time due to continuous sending POST
requests and re-rendering the browser, which would make it very distracting to a user while
trying to watch the video stream. In order to stream live videos, HTML frames would have to
be used (one to control the iRobot and one to stream live video) [26].
Using an AJAX methodology, the web page rendering a user interface would never
need to be refreshed. Server responses are no longer full HTML web pages but rather small
messages in the form of XML. The responses are sent asynchronously using XHR, resulting
in a much faster response time due to the communication model being asynchronous and the
data exchange being much smaller than the HTML forms counterpart. A fast response time is
always desired as it is a key component to all UI based controls.


15
With AJAX, XHR objects are used to invoke Web Services asynchronously. This is
achieved because XHR objects are invoking Web Services and waiting for a response in the
browser’s background process. The invocation of a Web Service is done by sending a SOAP
Request message and waiting for a SOAP Response message. Since this communication
happens in the browser’s background process and decoupled from the main browser event
loop, data can still be sent and received while a user is performing any main browser action.
With a traditional HTML forms method, a user would have to click a button to submit the
request and wait for the HTML form to be reloaded before continuing. However, during this
reload process the browser is blocked prevent any action from taking place.
Another benefit offered by AJAX is reduced network traffic on the server. With a
HTML forms model every browser connected to the Robajax Web User Interface would
connect to the server and the server would have to send a complete HTML web page to every
browser. Imagine the scenario where every single browser begins to direct the iRobot to
move, the server would again have to send a complete HTML web page to every browser.
Using an AJAX methodology, the JavaScript code on the browser is responsible for
rendering the web page. The server now has a smaller job because now the server only has to
respond with small messages. This reduced network traffic on the server allows the Robajax
Web User Interface to scale very well, as the added network traffic between one browser
session connected to the server versus many browser sessions is very minimal. Because of
these advantages, using an AJAX based methodology is a superior technology to use for
Web-based for telerobotic control system interfaces.
Ease of access is important to all UI based controls. If the Robajax Web User
Interface were to be developed as a monolithic personal computer (PC) application, the
Robajax Web User Interface would only be accessible to users with a PC. Any UI based
control wants to expand its potential list of users to as many users as possible. Being a web
page, the Robajax Web User Interface is readily available to anyone with Internet access.
With net books and smart phones rising in popularity, the Robajax Web User Interface will
have more and more potential users. Devices ranging from PCs to personal digital assistants
(PDAs) to an Android Phone can logon to the Robajax Web User Interface and control the
iRobot.


16
2.5

T
HE
R
OBAJAX
W
EB
A
PPLICATION
G
ATEWAY

The Robajax Web Application Gateway is a Web Service designed to reduce the
amount of network connections (sessions) to the iRobot. When the iRobot boots up, the
iRobot will connect to the nearest access point. The iRobot will then send a broadcast request
looking for a Dynamic Host Configuration Protocol (DHCP) server to answer. The router
will direct the request to the correct DHCP server and that DHCP server will then assign an
Internet Protocol (IP) address to the iRobot. Afterwards the iRobot will establish a TCP
socket connection with the Robajax Web Application Gateway. Suppose there are multiple
users on multiple networks that want to control the iRobot. Each user would open a web
browser and log on to the Robajax Web User Interface to interact with the UI. Every SOAP
Request constructed by the UI will be transmitted to the Robajax Web Application Gateway;
the Robajax Web Application Gateway will then invoke a web operation to create the correct
JSON message to transmit to the iRobot. Thus, the Robajax Web Application Gateway can
have many sessions, but the iRobot will have only one session (see Figure 2.4).

Figure 2.4. The Robajax Web Application Gateway eliminates redundant
video traffic between itself and the iRobot.


17
The main purpose of restricting the number of sessions over the WLAN to one is to
prevent the streaming of redundant video to multiple parties, thus allowing the wireless
channel to carry a single video stream to the Robajax Web Application Gateway with the
highest frames/sec or lowest QScale value.
Without the Robajax Web Application Gateway, communication between the iRobot
and the UI would look vastly different. One possible way is to store the IP address assigned
by the DHCP server as a Fully Qualified Domain Name (FQDN). With this implementation,
after the DHCP server assigns an IP to the iRobot, it would communicate this information to
the DNS on gammabot.sdsu.edu and the DNS will store the IP address would be stored as a
FQDN. Now every SOAP Request constructed by the JavaScript code by the UI will be
transmitted to a GlassFish server on the iRobot (see Figure 2.5), while any status information
from the iRobot would be transmitted to each user as well. With the same scenario as above
(multiple user opening a browsers), this would multiply the amount of sessions on the iRobot
as a function of the number of users. As stated above, sessions maintained by the iRobot
should be minimized. Thus, this second implementation without using the Robajax Web
Application Gateway is not an ideal solution because it does not allow this telerobotic system
to be scalable.
2.6

G
LASS
F
ISH

Since this system architecture uses a Web Service (the Robajax Web Application
Gateway) to decrease the amount of network connections to the iRobot, there were two
choices between which protocol to use for exchanging information, SOAP or REST. Because
there is not a standard for RESTful Web Services, SOAP was the chosen Web Service.
Another benefit of using SOAP is that there already exists a powerful SOAP container
(GlassFish) that comes packaged with NetBeans, the integrated development environment
(IDE) used for this thesis.
2.7

T
HE I
R
OBOT

The iRobot Create
TM
Vehicle (the iRobot) consists of electronic and software
interfaces for controlling the iRobot’s behavior and reading sensors (see Figure 2.6). The
electronic interface includes a 7 pin Mini-DIN connector and a DB-25 connector in the cargo


18

Figure 2.5. The iRobot will generate more traffic if the Robajax Web Application
Gateway is not used.


Figure 2.6. The iRobot Create
TM
Vehicle.



19

bay for connecting hardware and electronics for sensors. The software interface allows
manipulation of the iRobot’s behavior and reading sensor data through a series of commands.
Because of limitations in storage space and processing power of the iRobot, the
aforementioned motherboard (Refer to Section 2.1) was attached to the iRobot. The iRobot
Create
TM
Vehicle was chosen because of the affordability and flexibility in extending the
model’s functions.




20
CHAPTER 3
USER INTERFACE DESIGN
The challenge of every telerobotic system is to create a user interface that is both
ergonomically friendly and easy to use. Many UIs were developed during this research effort.
The many revisions of the UI, ranging from the beginning button based UI to the final
implementation of the circle based UI, are discussed in this chapter.
3.1

B
UTTON
B
ASED
UI
The button based UI (see Figure 3.1) was the very first design implemented to control
the iRobot platform. Its design allowed the user to control the iRobot in a very simplistic and
easy to understand manner. The Forward/Backward buttons allowed the user to move at a
constant speed of 100 mm/s, either forward or backwards, while the left/right buttons
allowed the user to move at a speed of 100 mm/s while turning at a radius of 100 mm. The
radius is measured from the center of the turning circle to the center of the iRobot; the longer
the radius the straighter the iRobot will drive before turning (see Figure 3.2) [1]. A stop
button was used to order the iRobot to halt. However, during test runs the following issues
were discovered:
1. Lack of speed control: Because of this UI’s simplistic design, the user wasn't able to
control how fast they wanted the iRobot to move. This UI only allowed the user to
move the robot at a constant speed of 100 mm/s. This created various issues if the
user wanted to have the iRobot move faster, a likely scenario if the user wanted the
iRobot to cover a vast area quickly.
2. Inability to control wheel angular velocity and turning radius: Similar to the lack of
speed control, this design didn't offer the user an ability to control the turning speed
of the iRobot. The iRobot was only able to be controlled to move at constant speed
and turning radius. This deficiency prevented the iRobot from being able to perform
sharp turns, which is problematic if there are a lot of obstacles in the iRobot's area.
3. Cumbersome to use: Even though the button based GUI was easy to use and
understand, the amount of buttons made it an encumbrance. While controlling the
iRobot, if any turn commands were needed, the user had to click the left/right button
repetitively. This is especially frustrating to a user if there are a lot of obstacles in the
iRobot's area. With the amount of buttons needed to control the iRobot, users can


21

Figure 3.1. Screenshot of the button based UI.
quickly develop hand and finger fatigue. This would cause this telerobotic system to
not be non-ergonomic.
The video stream’s low resolution is caused by a high QScale. High QScale reduces
the quality of the video stream, inversely; a low QScale will increase quality. High frame rate
allows fast video streaming, whereas a low frame rate will make the video stream appear
choppy to a user. The best video stream would consist of a low QScale and a high frame rate
for fast streaming high quality video. These two values, QScale and Frame/s, must combine
to stay within in the Encoding Bitrate [27].
Depending on the scenario, the QScale and Frame/s can be varied. For example, when
driving the iRobot, emphasis should be placed on a fast video stream but not quality. So, both
the QScale and Frame/s should be high. However, when a remote user wants to look at an
object in front of the iRobot, emphasis is placed on quality and not how fast the frame rate is,


22

Figure 3.2. How the iRobot turns based on the radius.
because it is not useful to see a fast stream if it is hard to determine what object is in front of
the iRobot. Thus, in this scenario, both the QScale and frame rate should be low.
While this design served as a good starting point as a GUI to control the iRobot, the
button based UI contained a lot of key problems that needed to be addressed (see Table 3.1)
to make it ergonomically user-friendly.
Table 3.1. Button Based UI Summary
Advantages Disadvantages
Easy to use/understand
Lack of Speed Control
Lack of Turn Speed/Radius
Cumbersome to use

3.2

S
LIDER
B
ASED
UI
The Slider based UI (see Figure 3.3) was the second design. It was created to address
the issue of not being able to control the speed of the iRobot. The slider in conjunction with
directional buttons allowed a user to better control the motion of the iRobot. The slider


23

Figure 3.3. Screenshot of the slider based UI.
controlled the straight-line speed of the iRobot; it allowed the user to set the speed of the
iRobot up to 0.5 m/s (500 mm/s).
Clicking the left and right button made the robot turn at a radius of 100 mm and move
forward at a speed of 100 mm/s. However, once the mouse button was released, another
command was issued, causing the iRobot to move forward at the previous speed requested.
For example, say the slider was positioned at 0.25 m/s and then the user clicked the right
button. The iRobot performed the requested turn at a speed of 0.1 m/s. Once the user released
the mouse button, the iRobot continued moving straight ahead at a speed of 0.25 m/s.
Holding the left and right button caused the iRobot to turn for a longer time period before
traveled straight at the previous speed. Thus, the iRobot can be issued to move in a circle.
Even though, this design resulted in a greater control of the iRobot, test runs revealed
the following issues:
1. Lack of Turn Speed/Radius: Since the UI features the same left/right button as the
Button GUI (refer to section 3.1), this problem wasn’t fixed, limiting the user in that
the robot could only be made to turn at a constant velocity.
2. Inefficient Left/Right maneuvering: Once the user clicked the left/right button, a
SOAP Request is sent to GlassFish on maxwell.sdsu.edu, which then creates a JSON
object and passes the JSON object along to gammabot.sdsu.edu to perform the
requested turn command. After the mouse up event (when a user lets go of a mouse
button), another SOAP Request is sent to command the iRobot to move forward at the
most recent slider invoked speed, as explained earlier. This is inefficient as it makes
both maxwell.sdsu.edu and gammabot.sdsu.edu process two requests. Thus every turn


24
command caused the iRobot to perform two motion requests. If a user is navigating
the iRobot through a series of several obstacles, both servers can be inundated with
messages that will saturate the network.
3. Hard to Stop Using Slider: While the slider allows the user to easily control the speed
of the iRobot, it was hard to set the slider at an exact speed. Normally being a couple
of millimeters off did not pose a big problem. However, trying to adjust the slider
value to be exactly 0mm/s to force the iRobot to stop was challenging. This is
worrisome as there are cases where the iRobot needed to stop immediately to prevent
the iRobot from colliding with a rigid body or driving off a ledge. This problem was
fixed by using both the stop button in conjunction with the slider. However, this slider
based UI is an inefficient design to force the user to use two different mechanisms for
controlling the same feature.
4. Cumbersome to Use: While controlling the iRobot, a user had to move the slider to
make the iRobot move forward, click the turn buttons to make the iRobot adjust to the
left or the right, and use the stop button to halt the iRobot. All these UI sequences
required the user to perform a lot of mouse clicking and mouse movement which
could be exhausting for a user.
Once again, another design was need to address these issues (see Table 3.2).
Table 3.2. Slider Based UI Summary
Advantages Disadvantages
Able to control the speed
Lack of Turn Speed/Radius
Inefficient Left/Right maneuvering
Hard to Stop Using Slider
Cumbersome to use
3.3

C
IRCLE
B
ASED
UI
The Circle based UI (see Figure 3.4) was the third and final design. Issues that
plagued the first two designs, namely the inability to vary the turning velocity and radius
were solved. In the Circle based UI the user controls the iRobot by maneuvering the mouse
within a circle. The circle has a top view point of reference. The mouse position within the
circle is polled every second by the JavaScript code, from that mouse position a vector is
drawn using the center as the initial point. This creates two key attributes (1) robot velocity is
proportional to the magnitude of the vector; (2) orientation (turning radius) is a function of
theta, the angle of the vector. So the position in the circle completely defines the motion of
the iRobot.


25

Figure 3.4. Screenshot of the circle based UI.
This design can be best thought of as if you are driving a car. If the vector is a straight
line forward, the iRobot will go straight. If the vector is a straight line backward, the iRobot
will drive in reverse. If there’s a slight angle, then the iRobot will turn its wheel and move in
the direction specified. Like making a turn with a car, if you do not straighten out the wheels,
the iRobot will continue to drive in a circle.
The mouse position is polled every second, if there is a significant change (refer to
Section 5.1) since the previous mouse position, the iRobot will be told of the new vector to
move (see Figure 3.5). Moving the mouse over the red dot (see Figure 3.6) will cause the
iRobot to immediately stop by sending an immediate stop message, even if the polling period
has not matured (T
mature
) as in indicated in Figure 3.6. If the user decides that polling every
second is not sufficient, the user can click within the circle. This will cause the iRobot to
immediately perform the requested movement command. Since the vector can only be
determined within the circle, moving the mouse outside of the circle will cause the iRobot to
immediately stop.


26

Figure 3.5. A decision diagram for tracking mouse
movement within a web browser.

Figure 3.6. A SOAP message is sent immediately when the mouse enters the
red circle.


27
The Circle based UI design addresses every previous issue encountered. The user can
still control the speed of the iRobot with the added benefit of controlling the turning radius,
thus it is more efficient as the request to the iRobot is now a single command (see Table 3.3).
Stopping is now trivial as once the mouse moves over the red dot, the iRobot will
immediately stop. A user does not have to use a mixture of user interface widgets (sliders and
left/right buttons) to control robotic motion. In addition, a user does not have to continuously
click a mouse button, so it is ergonomically friendlier. Controlling the iRobot using the circle
based UI is similar to driving a car, so it is easy to understand as it is based on a concept
many users are familiar with.
Table 3.3. Circle Based UI Summary
Advantages Disadvantages
Controls speed/turn radius in 1 command
Easy to stop the iRobot motion
Ergonomically friendly
Mouse clicking is optional
Easy to use
One second polling might not fit user's needs
Mouse clicking might still be needed

The only issue encountered is the mouse polling period of one second. It is difficult to
guess what the user’s needs are in advance, so the default period is configured to be one
second. However, the mouse click feature was added in case the user cannot wait for an
entire second.



28
CHAPTER 4
PROBLEMS ENCOUNTERED
The Spiral software development model [28] was used to develop the iRobot’s user
interface. During the software testing phase many issues were encountered. These issues
were fixed in subsequent iterations. The following problems encountered and a discussion of
the methods used to fix them is presented in this chapter.
4.1

I
NCORRECT
S
TATUS
B
EING
D
ISPLAYED

IRobot provides certain status information that would be useful for a remote user.
Status such as battery capacity, battery temperature, current charge state, voltage and more
are all pertinent information when remotely controlling a remote robot. The user would be in
a potentially dangerous predicament is a remote robot is not responsive because of a dead
battery without being issued any type of information for them to monitor.
However, status information coming back at one stage in the development process
was nonsensical. Charge states of unknown value when the iRobot was charging and battery
temperature readings of 114 degrees Celsius (impossible as that battery would be burning)
were being returned initially. It was thought that the Java Communications 3.0 API was the
problem because it could not provide accurate ways of reading data [29]. So many solutions
were tried, solutions varying from using the example code of reading data from the serial port
(by applying an event handler) to manually reading the serial port, byte by byte. These initial
strategies did not work.
Finally, it was determined that the MAX232 chip designed to connect to the iRobot’s
serial port was wired wrong. The error was fixed and status information is now coming back
correctly.
4.2

W
EBSITE
P
ICTURES NOT
B
EING
L
OADED

After the slider based UI (refer to Section 3.2) was developed it was noted that the
slider picture was not being loaded correctly. Even though the static html code was written


29
correctly, the picture would only load correctly in the Internet Explorer but not the FireFox
browser.
This problem was inadvertently solved using AJAX methodology. The static html
code was being changed to be dynamically created in JavaScript when the website loaded.
Now the pictures load correctly in either FireFox or Internet Explorer.
4.3

O
UT OF
O
RDER
SOAP
MESSAGES

During test runs it was noted that SOAP messages coming from the Robajax Web
User Interface were sometimes arriving out of order. At the time, every SOAP message
received by the Robajax Web Application Gateway was blindly processed in the order
received. This caused problems because the out of order SOAP messages caused the iRobot
to move erratically. For example, with the slider based UI, clicking the left or right button
sends two SOAP messages to the Robajax Web Application Gateway. The first SOAP
message tells the iRobot to turn, the second SOAP message (which the Robajax Web User
Interface sends later) tells the iRobot to drive straight. However, sometimes the Robajax Web
Application Gateway receives the second SOAP message before the first SOAP message (see
Figure 4.1). This causes the iRobot to continually turn in a circle until another motion
command is sent in.

Figure 4.1. SOAP request messages being processed out of order.


30
To alleviate such problem, a message queue was implemented. SOAP messages
coming from the Robajax Web User Interface would have a sequence ID attached. When a
user first logs onto the Robajax Web User Interface UI page, a sequence ID (0-65535) would
be randomly generated. The Robajax Web User Interface will then establish a handshake
with the Robajax Web Application Gateway to send the sequence ID. The sequence ID is
used to make sure SOAP messages are processed in the order they were generated, not the
order the Robajax Web Application Gateway received them (see Figure 4.2).

Figure 4.2. SOAP messages being processed by in order by using sequence ids.
SOAP messages received by the Robajax Web Application Gateway would be stored
in the priority queue. If the messages are arriving in sequential order, the messages would
immediately be processed. However, if messages are arriving out of order, then the queue
would wait until a certain timeout interval has expired and then execute the out of order
messages (see Figure 4.3). The timeout interval was put in place in hope that the correct
SOAP message would be sent during that time and re-synchronize the sequence ID.


31

Figure 4.3. New expected sequence ID due to an out of order SOAP message.
4.4

SDSU

I
NTERNAL
WLAN

F
IREWALL

The initial design had the iRobot act as a server; taking incoming JSON messages
from a client, the Robajax Web Application Gateway (see Figure 4.4). However, during fall
of 2009 an internal WLAN firewall was erected by SDSU’s IT department that rendered such
operation invalid. This firewall blocked any programs on maxwell.sdsu.edu from establishing
an initial connection to devices on the wireless LAN (see Figure 4.5), essentially forcing any
program on maxwell.sdsu.edu to act as a client. Thus, the firewall allows TCP sessions to be
established if the session originator is a device on the wireless network. This created a
problem because the Robajax Web Application Gateway and the iRobot could no longer
communicate with each other.


32

Figure 4.4. Connection initiator before implementation of the SDSU firewall.

Figure 4.5. SDSU internal WLAN firewall restricts PCs on the
wired LAN from initiating a connection.


33
The only fix available was to make the Robajax Web Application Gateway a server
and have the iRobot initiate the connection (see Figure 4.6). The iRobot and the Robajax
Web Application Gateway initially used SOAP messages to communicate with another.
During this time, it was decided that SOAP messages were too data intensive to use for such
trivial task like directing the iRobot where to move. Instead of using SOAP messages, the
iRobot and the Robajax Web Application Gateway would now communicate using light-
weight JSON messages through a TCP socket.

Figure 4.6. The iRobot is now the connection initiator to adapt to SDSU firewall
restrictions.
The GlassFish sever on the iRobot was removed because the Robajax Web
Application Gateway now only uses JSON messages to communicate with the iRobot.
Because of this change, messages were being processed at a much faster rate and delay
periods noted during earlier iterations from using GlassFish and SOAP were negligible.
Initially it was thought that UDP sockets would be ideal for this re-architecture,
because UDP is a lighter weight transport protocol and the extra connection oriented features
of TCP were not needed. A few lost motion messages would not be detrimental to the iRobot
because a user can easily perform the same actions to have those messages sent again.
Additionally, the overhead to maintain a TCP socket was not needed. Unfortunately, the


34
SDSU firewall seems to block UDP protocols between the wired and wireless LAN, so TCP
had to be used instead.
4.5

L
OSS OF
W
IRELESS
S
IGNAL

Successful transmission between the iRobot and the Robajax Web Application
Gateway is highly coupled with the wireless signal strength. During test runs it was noted
that occasionally, in areas of low signal strengths or areas covered by a lot of metal
infrastructure, some messages sent to the iRobot were never received. Problematic areas tend
to be regions containing a lot of metal infrastructure and farther away from the access point.
Thus, a remote user would sometimes be required to manually recover the iRobot from an
area of poor signal strength. Using debugging tools it was noted that this problem started to
become more frequent as the wireless bit rate got closer to 1 Mb/s. Thus, a mechanism was
needed to determine when the iRobot and the Robajax Web Application Gateway are no
longer connected, so the connection could be restarted. This problem was solved by using a
“heartbeat” message.
The “heartbeat” refers to a message, requesting the iRobot’s status (see Figure 4.7),
that the Robajax Web Application Gateway will send to the iRobot every 3 seconds (see
Figure 4.8). IRobot has 5 seconds to respond to the signal; otherwise the Robajax Web
Application Gateway will close the existing TCP socket connection, open a new connection
and wait for the iRobot to re-establish connection (see Figure 4.9). The delay period was
needed to give ample time for the JSON response message to be returned. 5 seconds was
chosen because test runs showed such a delay provided ample time for a JSON response
message to be returned. A user will know when this has happened, because the status “No
connection to the iRobot” will be displayed on Robajax Web User Interface. Ideally, there
would be some native Java library that can be used to tell if there is still an existing
connection between the iRobot and the Robajax Web Application Gateway, because the
response might just take a long time because of low connectivity. However, we were not able
to find such a library. So, the only solution is to assume the iRobot is either down or has lost
connection to the Robajax Web Application Gateway.

Figure 4.7. Sample GetStatus JSON message.


35

Figure 4.8. Heartbeat between the
WriteThread and the iRobot is still alive.

Figure 4.9. Connection restart due to a lost heartbeat message.


36
On the iRobot’s side, it expects the “heartbeat” to be sent by the Robajax Web
Application Gateway every 5 seconds. After the time elapse, it will assume communication
to the Robajax Web Application Gateway is lost and continually try to re-establish a
connection. This autonomous solution allows the iRobot to function in areas of low
connectivity with no user intervention. It should be noted that areas with very low
connectivity will quickly drain the iRobot’s battery at a faster rate as it continually trying to
re-establish connection.
4.6

T
HE
R
OBAJAX
W
EB
A
PPLICATION
G
ATEWAY
D
EADLOCK
C
ONDITION

The use of TCP socket along with a “heartbeat” to monitor the health of the
connection worked well during trial runs. However, leaving the connection open for an
extended amount of time (overnight) would cause the Robajax Web Application Gateway
and the iRobot to lose their established connection. As a side effect the Robajax Web
Application Gateway would also enter a dead lock scenario (see Figure 4.10).

Figure 4.10. Deadlock scenario caused by a lost TCP connection.


37
The problem occurred when the Robajax Web Application Gateway sent a
“heartbeat” message to the iRobot (thus entering a blocking Input Output (IO) state), whom
during this time just happens to lose the established TCP socket connection. Because the
iRobot has just lost the connection, a response message cannot be sent back to the Robajax
Web Application Gateway. Complicating matters is that Java’s native socket interface has no
way of determining when an established socket connection has been lost. Thus, the Robajax
Web Application Gateway will continue to be stuck in its blocking IO state, waiting for an
IO response that will never happen.
This problem was fixed by using an updated socket interface, SocketChannel [30].
SocketChannel provides the ability to interrupt a blocking IO call, allowing control to be
regained if it is determined that the established socket connection is no longer viable. Now,
every time the Robajax Web Application Gateway sends any message (like the “heartbeat”)
to the iRobot, a timer is started. If the iRobot does not respond in 5 seconds, the timer will
wake up and interrupt the blocking IO call. The Robajax Web Application Gateway will then
close any existing socket connections and open another one waiting for the iRobot to re-
establish the connection.
On the iRobot, a timer is also used. If the iRobot does not receive a message from the
Robajax Web Application Gateway in 5 seconds, the message’s timer wakes up and interrupt
the blocking IO. The iRobot will then close off any existing socket connection, issue a stop
command (if the iRobot is not on the home base, a precaution taken just in case the iRobot is
moving) and attempt to re-establish a connection with the Robajax Web Application
Gateway after a wait period. The 5 second wait period is needed because when the iRobot’s
wireless connection is dropped, the iRobot needs some time to re-initialize the wireless link.
Attempting to establish a connection with the Robajax Web Application Gateway before the
link has been re-initialized will cause ICMP network unreachable errors.
The reason why the timer wakes up in 5 seconds is because the heartbeat is set to be
transmitted every 3 seconds, and 2 seconds is added on top of that to give some lag time for
transferring messages across a socket. Since this on average only allows enough time for one
message to be through, all messages sent between the Robajax Web Application Gateway
and the iRobot must be accounted for. The rationale being, if the wireless connection is


38
dropped, we do not want the iRobot to continue moving as there would not be a way to stop
it remotely. The risk is too great for the iRobot to collide with a rigid object or drive off a
ledge. It is much safer to have both the Robajax Web Application Gateway and the iRobot
stop its current task and re-initialize the connection to get back to a known state.


39
CHAPTER 5
TELEROBOTIC DESIGN
This chapter talks about the overall design of this telerobotic system. Designing a
telerobotic system with these ideas will make a telerobotic system scalable, portable,
ergonomically friendly, and accessible.
5.1

C
IRCLE
UI

D
ESIGN

This section talks about the design of the Circle UI. The main focus of the Circle UI
is to make the UI ergonomically friendly and accessible.
5.1.1 Algorithm
When a user logs on to Robajax Web User Interface, the very first thing that needs to
be done is synchronize the sequence ID with the Robajax Web Application Gateway (see
Figure 5.1). This sequence ID is needed to ensure SOAP Requests are processed by the
Robajax Web Application Gateway in the order they were submitted by a user through a
browser (Refer to Section 4.3). Afterwards, the two timers are started; the first timer is used
to poll the Robajax Web Application Gateway for the iRobot’s status. The status is important
because the status contains relevant information (battery temperature, battery charge, etc) a
user might need to determine the next best course of action. For example, by monitoring the
battery charge, a user could best determine when is the best time to direct the iRobot to return
to the home base to recharge the battery. An unresponsive the iRobot due to an empty battery
is a high nuisance to a user, as they would have to manually retrieve and transport the iRobot
back to the home base to re-charge, which would defeat the purpose of telerobotic control.
It is imperative that every mouse movement that occurs within the circle based UI is
not translated as a motion command, causing JavaScript to create a SOAP message and send
the SOAP message to the Robajax Web Application Gateway (see Figure 5.2). This is
because when dealing with any hardware, like the iRobot, requests that cause physical
motion need time to be processed. Otherwise, both servers (the iRobot and GlassFish running


40

Figure 5.1. A UI handshake sequence diagram.

Figure 5.2. A UI send command sequence diagram.


41
on the gateway server) will become saturated and can malfunction. Thus, mouse movements
within the circle will be polled every second (a time period chosen because it offered just
enough time to not saturate the Robajax Web Application Gateway and the delay did not
cause complaints during demonstrations with inexperienced users) to give the iRobot enough
time to process any request it receives. Even though, mouse movement within the circle
based UI is polled every second, this does not mean a SOAP message is issued every second.
Issuing the SOAP messages every second when the mouse has not moved is not efficient. To
further help reduce the amount of data transmitted over the network, SOAP messages will
only be sent if the current mouse position is greater than or equal a Euclidean distance equal
to three pixels than the previous mouse position used for the most recent SOAP message.
Test runs have shown that three pixels yielded acceptable results of not saturating the
network. During test runs, users did not even notice this restriction, because most of the time
they are moving the mouse beyond this arbitrary movement threshold.
Sometimes a user needs the iRobot to respond immediately and they cannot tolerate a
1 second delay period or the 3 Euclidean distance threshold. To alleviate such issue, if a user
clicks within the circle, a SOAP message (see Figure 5.3 for a sample SOAP message) is
immediately created and sent to the Robajax Web Application Gateway. Stop commands
(moving inside the stop region or outside the circle) also have the same effect. It is a bad idea
to enforce a delay period before stop commands because the iRobot could possibly collide
with a rigid object or drive off a ledge during such delay period.

Figure 5.3. Sample of a SOAP request message.


42
5.1.2 Static Class Diagram
To be consistent with object oriented programming (OOP), the following classes (see
Table 5.1) were created as part of this thesis work (see Figure 5.4).
Table 5.1. Circle UI’s Static Classes
Class Name High Level Description
Main
Application class that coordinates the rendering of all UI controls.
This class also tracks mouse movement and forwards such
movement to the circle control where the movement will either
processed as a motion command or ignored if the mouse
movement is outside of the circle control.
IRobot
Communication class used to send SOAP messages to the Robajax
Web Application Gateway and process responses returned.
Circle
Utility class used to render the circle user interface. This class also
determines if mouse movements should be processed as a motion
command (if the movement is within the circle UI rendered).
String Utility class used for manipulating strings.
Picture
Utility class used to create dynamic html code for inserting
pictures.
Dial
Utility class used to render the dial controls; the controls monitor
the iRobot's speed and direction.

5.2

T
HE
R
OBAJAX
W
EB
A
PPLICATION
G
ATEWAY
D
ESIGN

This section discusses the design of the Robajax Web Application Gateway. The
main focus of the Robajax Web Application Gateway is to make this telerobotic system
scalable in terms of the number of supported remote users and remote viewers.
5.2.1 Algorithm
The Robajax Web Application Gateway is used as an intermediate service point
between the Robajax Web User Interface and the iRobot. By design, the iRobot has a
dynamic IP address because this increases the portability of this Telerobotic system. With a
dynamic IP address the iRobot can be placed in any wireless network, receive an IP address
(through a DHCP server) and communicate with the Robajax Web Application Gateway. If a
dynamic IP address is not used, a user would have to configure a subnet mask and


43

Figure 5.4. The circle UI’s static class diagram.



44
a default gateway address and one or more name server addresses for every wireless network
the iRobot wants to use. Since the iRobot has a dynamic IP address, to communicate with
Robajax, an intermediate gateway with a static IP address is needed.
Since, SDSU’s firewall prevents the Robajax Web Application Gateway from
initiating a connection to devices on the campus’ wired network, the Robajax Web
Application Gateway is forced to act as a server. The Robajax Web Application Gateway’s
first course of action is use Transmission Control Protocol (TCP) to open a socket (SDSU’s
firewall prevents User Datagram Protocol (UDP) from being used) and wait for the iRobot to
establish a connection (see Figure 4.6 and Figure 5.5). During this non-connected state, any
SOAP messages sent by the Robajax Web User Interface are not processed by the Robajax
Web Application Gateway.

Figure 5.5. The Robajax Web Application Gateway start server sequence diagram.
To prevent unknown programs from establishing a connection with the Robajax Web
Application Gateway, the session needs to be authenticated with a shared secret key. So, the
iRobot would first establish a connection with the Robajax Web Application Gateway and
then send the shared secret key to authenticate the session. Once the correct password has
been sent, the Robajax Web Application Gateway will transition to a ready connection state


45
(NOT_READY → READY) and start to process SOAP messages sent by a user through the
AJAX application running within a user’s browser. Since it is important to know the status of
the iRobot, every 3 seconds, the Robajax Web Application Gateway will send a JSON
message to request for the iRobot’s status. This periodic polling is also used to determine if
there is a valid connection with the iRobot. An invalid connection will result in the TCP
Socket being closed and re-opened (see Figure 5.6).

Figure 5.6. The Robajax Web Application Gateway close server sequence diagram.
A sequence ID, arg0, of the SOAP message (see Figure 5.7 for a sample handshake
SOAP message) sent by the Robajax Web User Interface to the Robajax Web Application
Gateway, a process known as handshaking (see Figure 5.8), to ensure commands are
processed in order. These sequence ids ranges from [0-65535] and are always sequential
(except when wrapping from 65535 to 0). SOAP messages sent to the Robajax Web
Application Gateway are stored in a priority queue. Messages in the queue that arrived in
order are processed immediately. Messages that arrive out of order incur a 2 second penalty
before the message is processed; with the hope of the correct message being sent during that
delay period to re-synchronize the message order. If the correct messages do not arrive, the
out of order messages are processed (see Figure 4.3). Furthermore, the priority queue is


46

Figure 5.7. Sample of a handshake SOAP request message.

Figure 5.8. The Robajax Web Application Gateway
handshake sequence diagram.
designed to ignore any messages with a smaller sequence ID than the most recent message
processed (with the exception of the wrap around case).
Since motion commands to the iRobot causes the iRobot to move, it is important to
ensure that messages sent to the iRobot are not dropped, due to network saturation or the
message queue being full, (see Figure 5.9). This is because if a user notices that the iRobot is
about to collide with a rigid object or drive off a ledge, they would want to divert the iRobot
away from danger immediately. Thus, every command sent to the iRobot is attached to a
timer (see Figure 5.10). If the iRobot does not respond before the timer matures (5 seconds),
the Robajax Web Application Gateway will transition to a non-connected state, close off any
existing connections and wait for the iRobot to establish another connection. The value for
the timer has to be between the heartbeat timer (3 seconds) and an ample time when a
response is expected. Test runs have shown that 5 seconds yielded acceptable results.


47

Figure 5.9. The Robajax Web Application Gateway forward sequence diagram.

Figure 5.10. The Robajax Web Application Gateway write task sequence diagram.
5.2.2 Static Class Diagram
To be consistent with object oriented programming (OOP), the following classes (see
Table 5.2) were created as part of this thesis work (see Figure 5.11).
5.3

T
HE I
R
OBOT
D
ESIGN

This section discusses the design of the iRobot user interface. The main focus of
designing the iRobot UI is to make the interface portable.


48
Table 5.2. The Robajax Web Application Gateway’s Static Classes
Class Name High Level Description
Gateway
Is a Web Service that contains web operations which are
invoked when a SOAP message intended for the Robajax
Web Application Gateway is received.
WriteThread
A thread that creates and maintains the connection
between the Robajax Web Application Gateway and the
iRobot. This thread determines when to process a SOAP
Request the Robajax Web Application Gateway receives
by storing it in a queue and handles sending/receiving
JSON messages from the iRobot.
ConnectionState
A class used to store the connection state between the
Robajax Web Application Gateway and the iRobot.
SoapRequest
A container for the SOAP Request received. When a
SOAP Request is received, it will be bundled with a time
stamp. This time stamp allows the WriteThread to
determine when to process the SOAP Request.
WriteTask
A class dedicated to determine the status of a JSON
message sent by the Robajax Web Application Gateway to
the iRobot. This status is used to determine whether the
connection between the iRobot and the Robajax Web
Application Gateway is still valid.
Parser
A class used to parse the IP address of the Robajax Web
Application Gateway.

Figure 5.11. The Robajax Web Application Gateway static class diagram.


49
5.3.1 Algorithm
The very first action item for the iRobot is to open a serial port on the iRobot (see
Figure 5.12). Because of the importance of the iRobot’s status information, the iRobot will
periodically send a request for the iRobot’s status to the iRobot’s Micro Controller Unit
(MCU) over the serial port every 3 seconds. This is done frequently and autonomously
because the iRobot’s status is a very important piece of information that needs to be
refreshed constantly regardless of whether there is a user logged on to the Robajax Web User
Interface or not. This is done for 2 reasons: (1) Since the Robajax Web Application Gateway
always knows the status when a user logs onto to Robajax, the status can immediately be
displayed and (2) the Robajax Web Application Gateway automatically logs the status in a
history file allowing for diagnostic post processing and analysis.

Figure 5.12. The iRobot open port sequence diagram.
Afterwards, the iRobot will then need to establish a wireless TCP socket connection
with the Robajax Web Application Gateway. This is done quite easily because the Robajax
Web Application Gateway has a static IP address. In order to establish a TCP socket
connection, the iRobot needs to know what port the Robajax Web Application Gateway is
listening on (see Figure 5.13). A SOAP Request for the port number is sent to GlassFish on
port 8080 which will invoke a Web Service on the Robajax Web Application Gateway. The