Copyright by Seth Michael Holloway 2011 - Repository Home - The ...

moneygascityInternet και Εφαρμογές Web

8 Δεκ 2013 (πριν από 3 χρόνια και 10 μήνες)

235 εμφανίσεις

Copyright
by
Seth Michael Holloway
2011
The Dissertation Committee for Seth Michael Holloway
certies that this is the approved version of the following dissertation:
Simplifying the Programming of Intelligent
Environments
Committee:
Christine Julien,Supervisor
Randolph Bias
Sarfraz Khurshid
Miryung Kim
Dewayne E.Perry
Simplifying the Programming of Intelligent
Environments
by
Seth Michael Holloway,B.S.,M.S.
DISSERTATION
Presented to the Faculty of the Graduate School of
The University of Texas at Austin
in Partial Fulllment
of the Requirements
for the Degree of
DOCTOR OF PHILOSOPHY
THE UNIVERSITY OF TEXAS AT AUSTIN
May 2011
Dedicated to loved ones lost along the way.If they could see me now...
Acknowledgments
A big\Thanks!"to everyone who helped along the way.In particular,
my wonderful wife,Andrea,who allowed me to concentrate solely on my work.
Also to my immediate family:Pat and Jay Holloway,Allison Holloway,Lane
Holloway,Eddie and Ritamay Mire,Dean Mire,Kate and Ryan Mire,and all
of our pets (Iko,Gumbo,Einstein,Ada,Michaela,and Lola).
None of this would be possible if not for my committee.Thank you for
your time,support,feedback and expertise.It has been fun!
v
Simplifying the Programming of Intelligent
Environments
Publication No.
Seth Michael Holloway,Ph.D.
The University of Texas at Austin,2011
Supervisor:Christine Julien
In the future,computers will be virtually everywhere:carried by ev-
eryone and integrated into the environment.The increased computation and
communication capabilities will enable intelligent environments that react to
occupants through automated decision-making.Devices (sensors and actu-
ators) are the key to making intelligent environments a reality.We believe
that devices must be made more approachable for average users.Existing ap-
proaches to application development for intelligent environments require de-
tailed knowledge about devices and their low-leveling programming interfaces,
which greatly limits the number of potential users.Instead of limiting users,
we must enable everyone to programthe devices around them.Intelligent envi-
ronments will not be commonplace until average people can set up and manage
the hardware and software necessary for their personalized applications.
In simplifying the programming of intelligent environments,we rst
made sensors and actuators accessible to average programmers then extended
vi
our work to end-users.We term the former contribution Sensor Enablement
for Average Programmers (SEAP);the latter work is Sensor Enablement for
End-Users (SEEU).In our experience,devices'disparate,niche programming
languages and communication protocols presented great diculty in devel-
oping intelligent environments.To ease the development eort for average
programmers,we abstracted and standardized complex sensor and actuator
interactions,allowing users to instead think in terms of well-understood web
applications.Users have said that SEAP is easy-to-use and exciting.But what
about average people,end-users?
We found that end-users are incredibly interested in intelligent envi-
ronments.By engaging end-users we can create intelligent environments even
faster and allowdomain experts to tailor their environment.This dissertation's
second contribution,Sensor Enablement for End-Users (SEEU) provides a vi-
sual programming interface that allows users to create personalized automated
behaviors given available devices and data.We performed several user stud-
ies to uncover people's desires for intelligent environments and determine the
best interface for managing an intelligent environment.SEEU combines an
intuitive interface with the power and exibility of SEAP.SEEU is a usable
end-user programming framework that allows average people to create useful
applications for their intelligent environments.
With SEEU and SEAP,we simplied the development of intelligent
environments,reducing barriers to adoption of emerging sensing and actuation
technologies.We demonstrated the feasability with a series of user studies.
vii
Table of Contents
Acknowledgments v
Abstract vi
List of Tables xii
List of Figures xiii
Chapter 1.Introduction 1
1.1 Existing Approaches.......................3
1.2 Solution..............................5
1.3 Contributions...........................6
1.4 Impact...............................7
Chapter 2.SEAP
Sensor Enablement for Average Programmers 9
2.1 Sensor Enablement In Context..................13
2.2 SEAP Architectures........................15
2.2.1 Underlying Infrastructure.................17
2.2.2 SEAP Architecture:Devices...............19
2.2.2.1 Sensors.......................20
2.2.2.2 Actuators......................21
2.2.2.3 Reconguration..................23
2.2.3 SEAP Architecture:Application Server.........25
2.3 SEAP-Based Implementations..................27
2.3.1 SEAP Applied to an Existing Application........28
2.3.2 SEAP From Scratch....................30
2.4 SEAP Evaluation.........................32
2.4.1 Participants........................32
viii
2.4.2 Setting...........................34
2.4.3 Study Tasks........................35
2.4.4 Questions to Answer....................37
2.5 Study Results...........................38
2.5.1 Browser-based Tasks:1a,1b,1c.............40
2.5.2 Programming Tasks:2a,2b................41
2.5.3 Device Tasks:3a,3b....................43
2.5.4 Diculties.........................44
2.5.4.1 Devices......................44
2.5.4.2 Server Programming Language.........46
2.5.4.3 Platform......................46
2.5.5 Lessons Learned......................47
2.5.5.1 Devices......................47
2.5.5.2 Server Programming Language.........47
2.5.5.3 Platform......................48
2.5.5.4 Working with Human Subjects.........48
2.6 Discussion.............................49
2.6.1 Challenges.........................50
2.6.1.1 Speed and Precision...............50
2.6.1.2 Security......................51
2.6.2 Barriers to Adoption....................53
2.6.2.1 Non-conforming Devices.............53
2.6.2.2 Lack of Examples.................53
2.7 The Potential Power of SEAP..................54
Chapter 3.SEEU
Sensor Enablement for End-Users 57
3.1 Related Work...........................61
3.2 The SEEU Approach.......................65
3.3 Learning How Users Think About Intelligent Environments..65
3.3.1 Choosing the Test Method................67
3.3.2 Questions for the Users..................67
3.3.2.1 Participation Agreement.............68
ix
3.3.2.2 Demographic Information Collected.......68
3.3.2.3 Questions about Smart Homes..........69
3.3.3 Deploying the survey...................71
3.3.4 Survey Results.......................71
3.3.4.1 Demographics...................71
3.3.4.2 Answering our research questions........74
3.3.4.3 General observations...............78
3.3.4.4 Discussion of Survey Results...........78
3.4 Developing an Eective Interface.................80
3.4.1 Interface Research Questions...............82
3.4.2 Tasks............................82
3.4.3 The Interfaces.......................83
3.4.3.1 Puzzle.......................84
3.4.3.2 Magnetic Poetry.................86
3.4.3.3 Mad Libs.....................87
3.4.4 Administering the Test..................89
3.4.5 Interface Study Results..................90
3.4.5.1 Interface:Puzzle.................92
3.4.5.2 Interface:Magnetic Poetry............96
3.4.5.3 Interface:Mad Libs................99
3.4.5.4 The Interface We Chose.............102
3.4.5.5 Lessons Learned.................103
3.5 The SEEU System........................106
3.5.1 SEEU System Architecture................106
3.5.2 Technologies Employed..................107
3.5.3 Database Design......................107
3.5.4 Rules Processing Engine..................108
3.5.5 A Tour of the SEEU System...............112
3.5.5.1 Homepage.....................112
3.5.5.2 Sensors and Actuators..............112
3.5.5.3 Sensor Readings..................114
3.5.5.4 Actuator Commands...............115
x
3.5.5.5 Rules Creation..................115
3.6 SEEU User Study.........................117
3.6.1 System Setup........................118
3.6.2 Tasks............................119
3.6.3 Questions to Answer....................119
3.6.4 SEEU User Study Results.................120
3.6.5 Findings..........................122
3.6.6 User Suggestions......................124
3.7 Discussion.............................126
3.7.1 Complex Actuators....................126
3.7.2 Actuators are Sensors...................128
3.7.3 Do we need OR?......................128
3.7.4 Errors in Rules.......................130
3.7.5 System Stability......................131
3.7.6 Potential Impact......................132
Chapter 4.Conclusion 133
Appendices 136
Appendix A.SEAP User Study 137
Appendix B.Smart Home Survey 150
Appendix C.SEEU Interfaces User Study 152
Appendix D.SEEU User Study 186
Appendix E.Code Samples 190
Bibliography 192
Vita 210
xi
List of Tables
2.1 Experience for the SEAP user study participants........33
2.2 Development experiences for the SEAP user study participants.33
2.3 Languages that SEAP participants used to create web applica-
tions.................................33
2.4 Languages used by SEAP participants to create mobile appli-
cations...............................34
2.5 User feedback on the SEAP study.................39
2.6 Minutes spent for each task in the SEAP study.........40
2.7 Minutes spent for each task in the SEAP study with incom-
pletely timed entries removed...................41
3.1 User rankings of the three interfaces (lower is better)......91
3.2 Number of user ratings of the three interfaces..........92
3.3 Feedback on the ease-of-use of the SEEU system........121
3.4 User desire for a smart home system like SEEU.........121
A.1 Experience for the SEAP user study participants........137
A.2 Development experiences for the SEAP user study participants.138
A.3 Languages that SEAP participants used to create web applica-
tions.................................138
A.4 Languages used by SEAP participants to create mobile appli-
cations...............................138
A.5 User feedback on the SEAP study.................138
A.6 Minutes spent for each task in the SEAP study.........139
C.1 User rankings of the three interfaces (lower is better)......153
C.2 Number of user ratings of the three interfaces..........153
xii
List of Figures
1.1 Our contribution,SEAP/SEEU,related to other research...4
2.1 A high-level view of the SEAP architecture...........17
2.2 Data ow in the SEAP architecture...............20
2.3 Example user interface for sensor conguration.........25
2.4 Data ow for UbiCoee,before and after SEAP.........30
2.5 The SPOT-to-Bot data ow....................31
2.6 A Sun SPOT device........................36
2.7 The code for task 2a,number.jsp.................42
2.8 The code for task 2b,color.jsp..................42
2.9 TIOBE programming language popularity [112].........54
3.1 The locations of our continental United States participants...72
3.2 Histogram of participant ages (range = 19-85;mean = 41.7)..73
3.3 Self-assessed technological expertise across ages (mean = 2.32).
A value of 1 denotes low ability,while a value of 3 is high ability.73
3.4 The Puzzle interface as users encountered it...........85
3.5 The rule creation area for the Puzzle prototype interface....85
3.6 A sensor puzzle piece........................86
3.7 An actuator puzzle piece......................86
3.8 Magnetic Poetry in the real world................87
3.9 The Magnetic Poetry interface..................87
3.10 A Mad Libs example........................88
3.11 SEEU - Mad Libs Interface....................89
3.12 The paper prototype setup....................90
3.13 A creative example of the Puzzle interface............93
3.14 A simple rule created using the Puzzle interface.........94
3.15 A complex Puzzle rule.......................95
xiii
3.16 A Scratch-like interface drawn on the whiteboard........95
3.17 The Magnetic Poetry interface as users encountered it.....97
3.18 An example rule created using the Magnetic Poetry interface.98
3.19 A complex rule created using the Magnetic Poetry interface..99
3.20 Another example rule created using the Magnetic Poetry interface.100
3.21 A temperature rule created with the Mad Libs interface....100
3.22 Two potentially con icting rules created by one user with the
Mad Libs interface.........................101
3.23 Abstract language employed in the Mad Libs interface.....101
3.24 A complex,abstract rule created with the Mad Libs interface.102
3.25 The entity-relationship diagram for SEEU............109
3.26 An example rule as stored in the database............111
3.27 The basic rule grammar in Backus-Naur form..........111
3.28 The homepage of the SEEU system showing user-created sen-
sors,actuators,and rules.....................113
3.29 Adding a new sensor to the SEEU system............113
3.30 Paginated sensor readings in the SEEU system.........114
3.31 The command\on"displayed for humans in the SEEU system.115
3.32 The command\on"display for devices in the SEEU system..115
3.33 Creating a rule in the SEEU system...............116
3.34 The SEEU user study with seven SPOT devices.........117
3.35 An potential grammar for allowing sensors and actuators in
conditions..............................128
3.36 An incredibly hard to parse temperature rule..........129
A.1 The code for task 2a,number.jsp.................137
A.2 The code for task 2b,color.jsp..................137
A.3 SEAP user study notes for participant 1.............140
A.4 SEAP user study notes for participant 2.............141
A.5 SEAP user study notes for participant 3.............142
A.6 SEAP user study notes for participant 4.............143
A.7 SEAP user study notes for participant 5.............144
A.8 SEAP user study notes for participant 6.............145
xiv
A.9 SEAP user study notes for participant 7.............146
A.10 SEAP user study notes for participant 8.............147
A.11 SEAP user study notes for participant 9.............148
A.12 SEAP user study notes for participant 10............149
B.1 The locations of our continental United States participants...150
B.2 Histogram of participant ages (range = 19-85;mean = 41.7)..151
B.3 Self-assessed technological expertise across ages (mean = 2.32).
A value of 1 denotes low ability,while a value of 3 is high ability.151
C.1 The Puzzle interface as users encountered it...........152
C.2 The rule creation area for the Puzzle prototype interface....154
C.3 A sensor puzzle piece........................154
C.4 An actuator puzzle piece......................154
C.5 SEEU interface:User 1 - Mad Libs example 1..........155
C.6 SEEU interface:User 1 - Mad Libs example 2..........156
C.7 SEEU interface:User 1 - Mad Libs example 3..........157
C.8 SEEU interface:User 1 - Mad Libs example 4..........158
C.9 SEEU interface study notes:User 1................159
C.10 SEEU interface:User 2 - Mad Libs example 1..........160
C.11 SEEU interface study notes:User 2................161
C.12 SEEU interface:User 3 - Mad Libs example 1..........162
C.13 SEEU interface study notes:User 3................163
C.14 SEEU interface:User 4 - Mad Libs example 1..........164
C.15 SEEU interface study notes:User 4................165
C.16 SEEU interface:User 5 - Mad Libs example 1..........166
C.17 SEEU interface study notes:User 5................167
C.18 SEEU interface:User 6 - Mad Libs example 1..........168
C.19 SEEU interface:User 6 - Puzzle example 1...........169
C.20 SEEU interface study notes:User 6................170
C.21 SEEU interface - User 7 - Puzzle example 1...........171
C.22 SEEU interface - User 7 - Puzzle example 2...........172
C.23 SEEU interface - User 7 - Magnetic Poetry example 1.....172
xv
C.24 SEEU interface - User 7 - Mad Libs example 1.........173
C.25 SEEU interface - User 7 - Mad Libs example 2.........174
C.26 SEEU interface study notes:User 7................175
C.27 SEEU interface - User 8 - Magnetic Poetry example 1.....176
C.28 SEEU interface study notes:User 8................177
C.29 SEEU interface - User 9 - Magnetic Poetry example 1.....178
C.30 SEEU interface study notes:User 9................179
C.31 SEEU interface:User 10 - Mad Libs example 1.........180
C.32 SEEU interface study notes:User 10...............181
C.33 SEEU interface:User 11 - Mad Libs example 1.........182
C.34 SEEU interface study notes:User 11...............183
C.35 SEEU interface:User 12 - Mad Libs example 1.........184
C.36 SEEU interface study notes:User 12...............185
D.1 SEEU user study notes part 1...................186
D.2 SEEU user study notes part 2...................187
D.3 SEEU user study notes part 3...................187
D.4 SEEU user study notes part 4...................188
D.5 SEEU user study notes part 5...................189
xvi
Chapter 1
Introduction
Sensors (devices that gather data) and actuators (devices that perform
actions) are currently too complex to be widely adopted [38,47,96].Pro-
prietary programming languages,communication protocols,and data formats
compound to create a complex domain that few people can understand.This
reality is at odds with the power that sensors and actuators can provide for
all people.Intelligent environments powered by sensors and actuators have
been part of the public consciousness for several decades,but the dream is not
yet a reality.In 1935,Popular Mechanics magazine had an article entitled,
\The House That Runs Itself"[70];in 1951 popular science ction author
Ray Bradbury described the Happylife Home,a residence that took care of
its inhabitants [8].This story is still not technically possible.In 1991,Mark
Weiser espoused the idea of being immersed in technology and brought ubiqui-
tous computing to researchers [122].In the many years since Weiser described
his vision for sensors and actuators embedded in the environment,researchers
have successfully tackled a number of open issues,but people were still dis-
appointed by the lack of intelligence in modern environments [13].Sensors
and actuators|everything from thermometers to cameras to gyros|are now
common in homes,oces,and electronic devices like the iPhone.The next
1
step is to empower owners to use these devices in a unied system.
Modern descriptions of intelligent environments have evolved along
with technological capabilities and societal needs:sensors and actuators are
poised to revolutionize many parts of our lives including environmental mon-
itoring,construction management,and healthcare.There are innumerable
applications yet to be programmed by the people who would use them.With
talk about global warming,people are now more interested in data about their
environment like carbon dioxide output,general air quality,and various tem-
peratures.By embedding devices in their homes,people can monitor their
consumption and make informed decisions about how to live a more ecologi-
cally friendly life.Similarly,businesses could reduce electricity consumption
by automatically controlling heating,cooling,and lights.An intelligent con-
struction site would allow the supervisor to track valuable assets,including
materials and crew members,thereby increasing safety and reducing theft.
The supervisor could also easily keep track of properties that he would other-
wise have to check in-person,properties such as the structural integrity of a
building,the time for concrete to dry,etc.Home healthcare will be vital with
a massive aging population.Aided by a handful of sensors in the home,elderly
people could retain a level of care similar to that in a nursing home while still
enjoying the freedomand dignity of living on their own.Aging-in-place is only
one piece of the home healthcare puzzle.If we arm people with inexpensive
medical sensors and allow them to program their residences,everyone could
benet from routine (constant) check-ups.Diagnoses will be made with more
2
data,and emergency situations can be averted,or at worst,discovered quickly.
Each intelligent application will have dierent requirements,so it is imperative
that inhabitants manage their own environment.
1.1 Existing Approaches
Given the incredible possibilities,it is no surprise that intelligent envi-
ronments and their components are heavily researched.These are large-scale
sensor network systems that have to be usable by diverse populations.Re-
searchers have contributed knowledge to virtually every aspect of intelligent
environments;however,we believe that we are the rst project that solely fo-
cused on simplifying the programming of intelligent environments.We blend
research from ubiquitous computing,human computer interaction,sensor net-
working,and end-user software engineering to create an end-user programming
framework for intelligent environments.
Figure 1.1 provides a brief overview into the state of the art.We
included\Involves Users in Design"because we believe that user-centered
design will lead to frameworks that are more usable and well-liked;along the
same lines,we feel projects that target end-users should be tested by end-users
so we included this heading.Fewother projects have built a usable systemthat
provides an\End-to-End Solution"from devises to interface and deployment,
yet intelligent environments necessitate a completely working system.Our last
heading,\Uses Existing Infrastructure"is important because reusing existing
infrastructure will speed adoption.
3
Involves
Users in
Design
Tested on
Users
End-to-End
Solution
Uses Existing
Infrastructure
SEAP/SEEU
YES
YES
YES
YES
CAMP [116]
YES
NO
NO
YES
CASAS [85]
NO
YES
YES
YES
slog [14]
NO
NO
NO
YES
pRest
[26]
NO
NO
NO
NO
Atlas [54]
NO

YES

YES

YES

MediaCubes
[5]
YES

YES

NO

NO

Web of Things [101]
NO

NO

YES

YES

Figure 1.1:Our contribution,SEAP/SEEU,related to other research.
Existing research leaves a gap in creating and demonstrating the feasabil-
ity of an end-user programming framework for intelligent environments.While
we could have created a heavyweight system that redenes communication
protocols and programming languages,we chose to apply cutting-edge web
programming techniques to create a system that works today.We demon-
strate that our system allows end-users to express the actions that they desire
now.
We provide a more thorough coverage of work related to SEAP in Sec-
tion 2.1 and work related to SEEU in Section 3.1.
4
1.2 Solution
This dissertation reduces the complexities currently found in sensors
and actuators making them easier to integrate into applications.We found
that people understand sensors and actuators,but they cannot interact with
the devices because of obscure communication protocols and programming lan-
guages.Our rst major contribution,sensor enablement for average program-
mers (SEAP) addresses the device interaction challenge by raising the level of
abstraction.We model low-level devices as well-understood web components.
Given the current prevalence and future visions of the Internet,modeling intel-
ligent environments as web applications is a logical choice.There is a wealth
of information already available for web applications,and many programmers
already know a web language such as PHP,Java,or Python.We show that
by translating the problem to a well understood domain,developing intelli-
gent environments is tractable for a much larger population of programmers.
Programmers we talked to were very interested in using this research.
By focusing on average programmers (a termwe use to mean beginning
programmers and hobbyists,as opposed to sensor network researchers or vet-
eran programmers) we dramatically lower the barrier for developing intelligent
environments.To reach even more people,we must allow non-programmers
to specify behaviors.With our second major contribution,sensor enablement
for end-users (SEEU),we reduce intelligent environment complexities for end-
users (people with no formal education or experience in the computer sciences)
by oering a usable graphical interface.Research in this dissertation shows
5
that people are excited about automating their environments and many users
have a similar mental model of the process.End-users are able to program
whenever the interface is suciently simple and the end goal is suciently
worthwhile [11,65].In fact,there are over 55 million end-user programmers
using simplied interactions,like spreadsheets,to program without a com-
puter science degree [93].This is at least one order of magnitude larger than
the number of professional programmers.Given this precedent,we believe a
usable end-usable programming framework for intelligent environments would
reach millions of people.
1.3 Contributions
This dissertation addresses the challenges related to widespread deploy-
ment of intelligent environments.Specically,we
 Research Task 1.Dened a unied sensor/actuator model that makes
existing and future devices usable by average programmers.
 Research Task 2.Created a working embodiment of the sensor/actuator
model as a middleware,SEAP (Sensor Enablement for Average Program-
mers).
 Research Task 3.Evaluated the usefulness of SEAP on a selection of
novice software developers.
 Research Task 4.Discovered how end-users think about a tangible
6
intelligent environment,smart homes,and how they express desired
computer-controlled home behaviors.
 Research Task 5.Designed a programming interface that is accessible
to end-users for creating customized intelligent environment behaviors.
 Research Task 6.Created a software system,SEEU,that allows end-
users to specify behaviors without in-depth knowledge about the under-
lying technologies.
 Research Task 7.Evaluated the eectiveness of SEEU on end-users.
1.4 Impact
With SEAP and SEEU,pervasive computing applications are easier
to understand and develop.Programmers have the full range of expression
provided by sensors and actuators|while using their favorite web program-
ming language.End-users too can program their own intelligent environment
behaviors.
From an industrial standpoint,device manufacturers have a simple
specication to follow in order to create usable,interoperable devices.Intel-
lectual property and competitive advantages are maintained while users gain a
simpler interface.This dissertation represents a necessary step towards stan-
dardization for the embedded device industry that currently oers immense
capabilities through myriad complex,individual methods.
7
In addition,this work can help focus and speed future research.The
work blends software engineering,hardware specication,and usability.Oth-
ers can adopt the approach and explore alternatives,particularly by augment-
ing the system (for example by creating domain-specic user interfaces or
augmenting individual components of the system).The SEAP architecture
and the SEEU system can both be extended in many worthwhile ways.
This document has two major sections.We will rst discuss a system,
Sensor Enablement for Average Programmers (SEAP),that easily allows\av-
erage programmers"to create pervasive computing applications from scratch
and quickly integrate devices into existing applications.SEAP is covered in
detail in Chapter 2.In Chapter 3 we discuss an end-user programming frame-
work,Sensor Enablement for End-Users (SEEU),that opens the power of
pervasive computing to everyone.SEEU was designed with\average people"
in each step of the development process.Finally,Chapter 4 concludes.
8
Chapter 2
SEAP
Sensor Enablement for Average
Programmers
In this chapter we discuss SEAP:Sensor Enablement for Average Pro-
grammers.We will attempt to motivate the need for accessible sensors with
an example about George,an average programmer who wants a smart home
of his own.
George is an undergraduate in Computer Science.After completing
some college courses,he has learned many programming concepts and basic
Java;George is an average programmer.
Recently,George has started reading about intelligent environments|
an intriguing new idea that would make his life easier.George wants to use
his Java programming skills to automate his own apartment.He hopes to
start small and continue to augment the system as money allows.George's
initial goal is to automatically turn on lights in his current location.Later,he
can play music,control the A/C and fans,open and close blinds,brew coee,
and numerous other additions.George is really excited!With a little bit of
programming and a few pieces of hardware,he will never have to use a pesky
9
light switch again!
George begins by searching for solutions,but,after extensive research
George cannot nd anything that satises his requirements.There are inex-
pensive single-purpose solutions like a motion sensitive light switch [63],but
these replacements cannot be altered or extended.To achieve his initial goal
of illuminating only the occupied room,George could replace his light switches
with the motion sensitive version;however,the motion sensitive light switch is
not enough to allow George to achieve his future goals of,for example,turning
on the inhabitant's favorite radio station.The company selling the motion sen-
sitive light does not have all the devices to fulll George's dream.At the other
end of the spectrum,George nds costly,complex,multi-purpose solutions like
those fromCrestron [22].Using commercially available products,George could
automate his entire apartment and control the site with a state-of-the-art re-
mote;unfortunately,it would cost thousands of dollars and require extensive
knowledge about a specic brand.Even worse,George could be locked into
products that communicate using undocumented protocols.George wants a
cheap,extensible solution that he can program using a language he already
knows.George realizes the smart home reality is much more complicated than
buying a few new components and writing some Java code.
George understands the application logic perfectly (\if someone is present
then turn on lights"),but he does not know how to interact with the necessary
pervasive computing elements:sensors and actuators,like motion sensors and
lighting controls.A task as straightforward as reading the temperature turns
10
out to be very complex.Since there are no satisfactory solutions,George gives
up.
While George may be ctional,this scenario is real.Currently,only
highly-trained developers and sensor researchers have the knowledge necessary
to create intelligent environments,but there are many\average programmers"
who would develop intelligent environments if they could easily interact with
the necessary sensors and actuators.The potential impact of intelligent en-
vironments is commiserate with the number of people who can create them.
Devices must be usable by average programmers like George.
Commercial approaches ll a niche by supplying and supporting func-
tionality that is otherwise unavailable.Unfortunately,commercial products
solve only one problem and cannot interact with other devices.Current com-
panies [22,100,107,110] utilize proprietary protocols,eectively preventing
people from mixing and matching brands.For example,in the case of the
motion sensitive light switch,George cannot easily reuse the motion sensor
from his light switch;in the case of the Crestron complete home system,
George cannot integrate generic sensors from other companies into his in-
telligent apartment.Interacting with a simple device like the aforementioned
motion sensor (that only has two states:motion or no motion) would require
specialized knowledge of the unique communication protocol.More than that,
it would require disentangling it from the light.Combining multiple devices
would require knowledge of the many specialized protocols and languages as
well as prociency in an intermediary language that could accept the varied
11
inputs and generate distinct outputs.We found that many people understand
the basic application logic;they simply lack the in-depth knowledge currently
necessary to develop intelligent environments.To enable programmability of
pervasive computing applications,we propose a paradigm shift that empowers
the average programmer rather than highly specialized device programmers.
We explored the following research question:\How can we em-
power average programmers to develop their own intelligent en-
vironments?"To answer this question we completed the following three
research tasks:
 Research Task 1.Dene a unied sensor/actuator model that makes
existing and future devices usable by average programmers.
 Research Task 2.Create a working embodiment of the sensor/actuator
model as a middleware,SEAP (Sensor Enablement for Average Program-
mers).
 Research Task 3.Evaluate the usefulness of SEAP on a selection of
average programmers.
We classify this work as sensor enablement for average programmers,
or SEAP [45,46,103].We review previous eorts by other researchers related
to SEAP in Section 2.1 before going into our approach.The SEAP architec-
ture combines sensors,actuators,an application server,and web programming
languages;details are in Section 2.2.We have used SEAP to develop new ap-
plications and to evolve existing web applications into ubiquitous computing
12
applications.We detail these implementations in Section 2.3.To further eval-
uate SEAP,we performed a user study with\average programmers."We
discuss the user study in Section 2.4.In Section 2.6 we discuss SEAP and the
lessons learned during research and development.Section 2.7 concludes.
2.1 Sensor Enablement In Context
The notion of sensor and web integration is not entirely new,and sev-
eral related projects have paved the way for our approach.CoolTown allows
networked mobile devices to publish data on the web through a variety of
tailored protocols [4].Data published includes information about a device's
characteristics (e.g.,location),enabling a degree of content- and context-based
discovery.Another project created a centralized website that accepts sensor
data generated worldwide [14],a technique titled slog (sensor log).Sensors
communicate their data to a base station that funnels sensor data to a clear-
inghouse.The slog approach is adequate for collecting data frommany sources,
but does not help users create personalized intelligent environments.
Other current approaches to sharing sensor data build on web services
using SOAP,WSDL,and XML.The Open Geospatial Consortium (OGC)
allows access to sensors using SensorML,a sensor-specic language that denes
an XML schema to remotely interact with sensors [6].Microsoft's SenseWeb
project [92] also provides a generic method to push sensor data online.The
Atlas service oriented architecture [54] allows users to interact with devices,
but requires new hardware and software to fulll the vision.These approaches
13
rely on SOAP web services,which can be in exible,slow,hard to maintain and
manage,and heavyweight [57].SOAP web services require a great deal of setup
and pass large XML les that are much larger than the data being passed.
Pervasive computing deployments include resource-constrained devices that
benet from ecient,streamlined solutions,so we prefer to avoid the large
overhead present in these approaches.
While rooted in dierent technologies,there are a number of other
designs to reduce the eorts required to develop pervasive computing appli-
cations.For example,Weis et al.created a programming language,Visual-
RDK [121],to reduce the learning curve typically required for creating new
pervasive computing applications.The language is meant to speed pervasive
computing development by allowing developers to focus on application logic.
Other approaches [3,41] provide additional layers of abstraction to manage
complexities that can be hidden fromthe developer.These approaches are not
easily extended because they rely on a vocabulary dened by the authors.New
devices would not work until the authors hand-coded the vocabulary.These
techniques may complement the SEAP middleware architecture and might be
combined to further ease software development for pervasive computing.
Our goal is to reduce the apparent complexity of devices and enable
programmers to develop pervasive computing applications.We do this by rely-
ing on a simple,expressive method to transfer data between disparate devices,
standard HTTP.Our approach is consistent with representational state trans-
fer (REST) principles [29] in an eort to be lightweight and exible.REST
14
is a stateless client-server architectural style that promotes the transmission
of data over standard HTTP.Users interact with resources using HTTP verbs
(GET,POST,PUT,DELETE).Some work has been done to apply REST
to pervasive computing [26,66],however,this work violates REST principles
by redening HTTP.It also requires a great deal of a priori knowledge that
reduces scalability innate in the initial REST proposal.We believe that device
interactions can be achieved without altering HTTP,allowing us to inherit the
benets of both past and future work on HTTP.
Recent advances in cloud computing [12] have generated research simi-
lar to ours;this eld is called the\web of things,"a relative of the\Internet of
Things."A number of projects fromacademia and open-source are attempting
to make all devices usable online,such as JXTA-C [114],the RESTful plug
and play experience [102],putting things to REST [123],the RESTlet [79],
Sun Cloud APIs [106],SOA-based integration [101],and RESTful web ser-
vices specication [89].These projects share our goal (getting devices online
with a simple interface) and use similar technologies (HTTP),but none solve
the same problem in the same way that we do.Our research provides a well-
dened interface focused on average programmers.
2.2 SEAP Architectures
In this section,we describe the SEAP approach in detail.Specically,
we present the necessarily simple software architecture that underlies the sens-
ing,actuation,communication,and interaction capabilities as we approach our
15
rst research task:Research Task 1.Dene a unied sensor/actuator model
that makes existing and future devices usable by average programmers.
It takes humans approximately 10 years to master programming and
four years to gain basic competency [124];other approaches that aim to sim-
plify the programming of intelligent environments still require users to under-
stand sensors,actuators,and intelligent environments.These approaches re-
quire developers to learn the new domain (embedded hardware and software).
We hope to overcome this problem by modeling intelligent environments in
a way that average programmers are more likely to understand already.We
reduce the amount to learn by allowing programmers to apply the knowledge
they already have.Using SEAP,a web guru will be an intelligent environment
application guru as well,while the programmer would have a lot to learn with
other research projects.
To understand what average programmers already know,we looked
at trends in computing and course oerings.We found that most Electrical
Engineering and Computer Science students at The University of Texas at
Austin are learning a web capable programming language (namely,Java or
Python);this is in-line with the prevalence of web programming and forecasts
for continued expansion [24,84,112].
By relying on well-established programming standards,SEAP brings
the seemingly unapproachable task of programming pervasive computing ap-
plications into the hands of domain programmers who are experts in their
applications'requirements.SEAP hides complexities associated with data
16
collection and actuator command with familiar web programming patterns,
using lightweight software components deployed on resource-constrained de-
vices to manage the distributed coordination tasks.Through SEAP's abstrac-
tions,a developer can quickly tailor device and network congurations to a
particular task by adjusting parameters in the application.At the same time,
participating remote devices can use standard posting procedures to exchange
sensor data and actuation commands in simple formats.A broad overview of
the SEAP architecture can be seen in Figure 2.1.Here,\Server"refers to a
centralized host that may or may not be connected to the Internet.
Interface
Interface
User
User
Sensor
Actuator
Actuator
Server
Server
Configurations
Sensor
Configurations
Figure 2.1:A high-level view of the SEAP architecture.
We divide our description of the SEAP architecture into two aspects:
the behavior of devices participating in a SEAP supported application,and the
application server that hosts the applications.Before discussing these topics,
we will brie y discuss the underlying infrastructure.
2.2.1 Underlying Infrastructure
The SEAP architecture relies on HTTP to communicate.To help the
reader understand SEAP interactions,we will describe the basics as seen on
the Internet.While SEAP applications do not have to be online,the ability
17
to seamlessly integrate with the Internet is a huge benet.The Internet is a
network of computers connected using TCP/IP.The world wide web,WWW,
contains hypertext documents known as web pages.Typical web pages are
presented using hypertext markup language (HTML) and accessed using the
hypertext transfer protocol (HTTP).A\web address"is technically known as
a uniform resource identier (URI),or more generally,as a uniform resource
locator (URL).The URI is an alias for the IP address of the page.Let us
consider an example URI,http://www.sethholloway.com.The URI starting
with http://means that the resource we are seeking is available via HTTP.
Next comes www,which means that the resource is on the world wide web.
The remaining portion,sethholloway.com,is an alias for a specc IP address.
Whenever a person tries to access http://www.sethholloway.com using a web
browser,the browser retrieves the HTML information from a web server at
the specic IP address;the HTML is rendered as text and images.
Above,we described the process of retrieving a web page.Here we
discuss richer interactions with web pages,namely inputting information.In-
putting information on the web is achieved using HTML forms.Forms are a
generic construct for input;these include text boxes,radio buttons,drop-down
menus,and more.To make this idea more concrete,let us consider the ubiqui-
tous search-provider,Google.Google's minimalist page contains a single input
form:a text box form that accepts your search term.Whenever you search,
the form input is fed to the web server as a URI.For example,searching for
\Seth Holloway"generates the following URI:
18
http://www.google.com/search?hl=en&q=seth+holloway.
As with the above example of http://www.sethholloway.com,the Google
page is on the world wide web accessible using HTTP.After google.com we
see search.Search is the application that is being called.The question mark,
?,signals the beginning of parameters.Parameters are key-value pairs of the
form key=value separated by ampersands,(&);these are the variables that
the application requires.In this example there are two parameters,hl=en
and q=seth+holloway.The rst parameter,hl=en,denes the language,En-
glish,abbreviated as en.The second parameter,q=seth+holloway,denes the
search query,q,as\seth holloway"with the space represented as a plus sign.
Other web applications will have dierent parameters,but forms always input
information in the aforementioned style,using URIs.
Having described the supporting infrastructure provided by HTTP and
HTML,we now move into dening the SEAP architecture.
2.2.2 SEAP Architecture:Devices
As SEAP aims to be generally applicable to a wide variety of pervasive
computing applications,a goal of the SEAP architecture is therefore to ensure
that the functionality required for these devices is kept to a minimum.To this
end,devices are not required to accept arbitrary inbound connections;instead
each device controls its own communication costs through the outbound con-
nections it creates.This includes both the transmission of data requested by
an application and the reception of conguration and actuation commands.
19
The SEAP data ow depicted in Figure 2.2 provides a high-level view of the
relationships between dierent hardware components in a SEAP system.
Req. Configuration
Sensor
Data
User Code
Req. Command
Server
Req. Configuration
Configuration
Actuator
Configuration
Command
Figure 2.2:Data ow in the SEAP architecture.
SEAP participating devices are classied as sensors (data producers)
or actuators (command consumers).While it is possible for a node to perform
both sensing and actuation duties simultaneously,this distinction allows clarity
in the architectural description.
2.2.2.1 Sensors.
A device that is creating its own data or gathering data from the en-
vironment to send to the application is considered a sensor.SEAP enables
applications to collect this information from sensors in an intuitive fashion.
SEAP,running on the sensor/actuator,creates an interface to the accepted
web-based programming approaches.Specically,when a sensor has data to
20
send,SEAP packages the data and opens an HTTP connection to a precon-
gured Uniform Resource Identier (URI).The sensor data is encapsulated
as parameters in the connection and is transferred to the application.The
high-level SEAP algorithm that runs on sensors is shown below.
while(true) f
connect to data-report-uri
send readings
disconnect
sleep data-report-delay
g
The connection to the server is successful when communication paths
are available (SEAP delegates communication to an underlying network in-
frastructure).When the central web server receives the connection and its pa-
rameters,SEAP deployments use freely-available,widely-used web application
servers (e.g.,Apache httpd,Tomcat,nginx,IIS,lighttpd) to handle commu-
nication,parse the parameters,and present the data to the user's application.
Because the user's application appears to be interacting only through web
programming constructs,interaction with sensor data is reduced to operations
that are familiar and routine for an average programmer.
Source code for a Sun SPOT device [97,105] congured as a SEAP-style
temperature sensor can be found in the Appendix.
2.2.2.2 Actuators.
Adevice that receives commands froman application to alter the logical
or physical environment is considered an actuator.As before,SEAP's goal
21
is to allow applications to pass commands to remote actuators using simple
and intuitive techniques.Like sensors,a remote SEAP actuator manages an
HTTP connection and invokes native functions.A basic SEAP component for
a remote actuator could use the following algorithm.
while(true) f
connect to command-uri
while (connection open) f
read command
apply command
g
sleep command-retry-delay
g
Here,the while (connection open) loop is used to eliminate the re-
source usage of continuously polling the server for new commands.Devices
read sequences of commands from a single connection as they are sent by the
server.Assuming the read command above is able to parse the commands as
they are delivered,the inner while loop explicitly provides support for batching
many commands together.Any sequence of commands returned to the device
will be processed in order,without encountering the sleep statement.
However,even batching commands together may still not provide the
immediate response that some applications desire.If the connection is closed
after each batch of commands is processed,the device must still wait for the
command-retry-delay before receiving a new batch of commands.This concern
is easily addressed by allowing the read command to block until new commands
are received.The server simply distributes commands with arbitrary delays,
22
each of which is read and applied as soon as it is received.When the end-to-
end connections are stable (compared to the command-retry-delay),this gives
the application much ner control over the timing of actuation commands.
Rather than polling (closing and re-opening the HTTP connection) for
every command,we could leave the connection open.This communication
style is commonly referred to as Comet [21],but is also known as Ajax push,
reverse Ajax,two-way-web,HTTP-streaming,and HTTP server push.In this
approach the HTTP connection persists allowing the web server to push data
to the receiver.The increased responsiveness comes at a price:the actuator
and server now has to keep an HTTP connection alive for every actuator.Each
connection requires a small amount of memory and processing power;this is
probably not a problem for the server,but the devices may not be able to
handle the extra load.
2.2.2.3 Reconguration.
Both sensors and actuators rely on a small set of variables to properly
integrate and support a SEAP application.These variables are set by the pro-
grammer,ideally when the device is rst placed in the environment.However,
while the application is running,it may become necessary to change the values
of such a variable on a remote device.For example,a temperature monitoring
application may request temperatures once every ve minutes until a thresh-
old is met,then the application will request temperatures once every thirty
seconds.To enable these scenarios,devices also host a SEAP component that
23
downloads conguration changes in much the same way that commands are
retrieved by an actuator.This allows a developer to programmatically alter
any of the variables mentioned in this section,even the reconguration URI,
at runtime.
In the example sensor and actuator algorithms above,the uri and
delay variables can be reassigned by providing new values on the webserver
as a simple properties-style conguration page like the one below.Updating
the uri means the device will interact with the server at the new address.
Updating the delay will aect how often the device connects to the web server.
http://my.host.name/sensor7/conguration.jsp
configuration-uri=http://my.host.name/sensor7/configuration.jsp
configuration-delay=60
data-report-uri=http://my.host.name/sensor7/temperature.jsp
data-report-delay=3
command-uri=http://my.host.name/sensor7/alarm
status.jsp
command-retry-delay=10
In this case the conguration le contains the URIs and delays for a
ctitious device sensor7.The device parses the conguration and updates its
internal variables.Once the sensor has applied this conguration,we expect
it to post readings every three seconds,and download its conguration every
60 seconds.
Even devices that can be recongured still require initial settings;a pro-
cess we call bootstrapping.When a device participating in SEAP is initially
put into service for an application,the user congures the device by specify-
ing variables necessary to coordinate with the web application.An example
24
conguration utility,Figure 2.3,demonstrates the interaction that is required
for this process.Here,a device with a temperature sensor and an LED is con-
gured to publish temperature readings and retrieve commands to control the
LED.The nature of this component depends on the particular target device;
its main purpose is to translate between the low-level hardware and protocols
and the simple SEAP interface.As the nal step of the conguration utility,
the SEAP component is loaded on the device (either through a direct connec-
tion or through existing over-the-air programming capabilities [27,60,61,86]).
seconds
10
http://my.host.name/sensor7/configuration.jsp
Read from:
Configuration
3
Every
Read from:
Configuration Utility: Sensor 0000.0661 
seconds
Temperature Sensor
60
LED Display
Every
ubmit
seconds
http://my.host.name/sensor7/alarm_status.jsp
x
C
http://my.host.name/sensor7/temperature.jsp
Post to:
ancel
S

Every
Figure 2.3:Example user interface for sensor conguration.
2.2.3 SEAP Architecture:Application Server
Because average,hobbyist programmers do not have device-specic
knowledge,we shift the application logic to the application server where pro-
grammers are more comfortable.By centralizing the logic to an application
server we mitigate complexities of coordinating distributed devices.This also
25
allows us to reuse existing high-quality tools and techniques that are not avail-
able to similar research projects that develop their own application servers and
communication protocols.Because SEAP reduces interactions with both sen-
sors and actuators to a standard web-style interaction paradigm,constructing
applications becomes more familiar as we show in Section 2.5.By relying on
standard HTTP,SEAP applications can leverage the wealth of existing knowl-
edge about web applications.Research approaches that are built on anything
but standard HTTP will not be able to use existing tools.
A web application framework (e.g.,Struts,Django,Ruby on Rails,
Google Web Toolkit,Yii,ASP.NET MVC,etc.) simplies the design of the
server by addressing many of the traditional server-programming concerns
(e.g.,connection management,data-stream parsing,etc.).Many of these
frameworks also provide support for advanced features such as load balanc-
ing and clustering that would be useful to future large-scale pervasive com-
puting applications;these features,enabled specically by our approach and
not present in other research,would allow pervasive computing to scale us-
ing standard methods.While other approaches would necessitate research
into optimization and scaling,SEAP applications can grow and evolve using
the readily-available features.By including web application frameworks in
the SEAP architecture,we are encouraging the separation of non-application
concerns from the user's code base and into a purpose-built tool.Pragmat-
ically,the key advantages of using a web application framework are derived
from re-framing poorly understood intelligent environment problems into well
26
understood web applications.This enables developers to use any of the pop-
ular web programming languages and the manuals,tutorials,and guides that
are available for them.Developers also benet from the high-quality tools
available for testing,debugging,and documenting web applications.
While we have primarily provided details about dynamic web pages,
SEAP also works with simple web servers with static content.Commands
and congurations requested by remote devices could be read directly from
the central server's le system.Any web server would be sucient for this
purpose.Updates to the les made by hand or by other applications would be
immediately re ected by the web server and propagated to the devices.
2.3 SEAP-Based Implementations
Being interested in more than theory,we implemented the model de-
scribed above as our second research task:Research Task 2.Create a work-
ing embodiment of the sensor/actuator model as a middleware,SEAP (Sensor
Enablement for Average Programmers).
SEAP eases software development by moving the logic of pervasive
computing applications to an environment that is accessible and familiar to
average programmers.In this section we detail how this is done in practice for
two dierent applications.One application was developed before the SEAP
architecture was conceived and then adapted to it after the fact.The other was
developed from scratch using the SEAP patterns to build a robust, exible,
and easily testable application.We mention the testability because average
27
programmers are less skilled at debugging than their senior counterparts [55];
thus straightforward verication is critical to their success.
2.3.1 SEAP Applied to an Existing Application
Graduate students in the department formed an unocial coee coop-
erative to make coee more readily available and ecient (brewing a personal
portion felt wasteful and inecient).At rst,the student who brewed coee
would send an email to others in the group.This solution worked,but a gradu-
ate student developed a web application,UbiCoee,to replace the emails and
show up-to-date coee levels.Originally,this was a simple web application
where users updated the amount available,but the site and coee level would
get out of sync when users got interrupted before updating the site.To get
immediate feedback,we needed something closer to the coee pot.
We decided to place two buttons near the coee pots to indicate the ad-
dition and removal of coee.Instead of directly inputting the number of cups
of coee available,users would simply input the change.Users would press a
button labeled\+1"several times when brewing new coee (8 presses to indi-
cate brewing 8 cups) or press the\-1"button when taking a cup from the pot.
To provide this functionality,we selected Sun SPOT [97] devices that have two
buttons|perfect for our\+1"and\-1"idea.However,this choice left us with
a problem:our SPOT devices needed a custom application in order to com-
municate with our web server and the devices'personal area network messages
would need to be transferred to local area networking.The device manufactur-
28
ers foresaw the latter problem,so they allow SPOT devices to be congured
as basestations.These basestations are plugged in to a personal computer
and use the host machine's network to translate messages.SEAP solves the
former problem of communicating with the web server.Using JavaME [75],
we developed an application on the SPOT that created an HTTP stream and
sent it to our Ubicoee server through a basestation SPOT connected to a
personal computer.This task was not overly dicult;however,in the future,
device manufacturers could easily provide this functionality out-of-the-box.
With Sun SPOTs now able to communicate with the web server,we
could replace the user input with Sun SPOT inputs.To start,we extended
the UbiCoee application to include an\amount changed"in addition to the
existing\amount available."The amount available is the sum of the amount
changed.Figure 2.4 shows the data ow for UbiCoee before and after SEAP.
Because the SEAP version of the application can interact with users and de-
vices,it is more powerful than the standard,device-free application.We gain
this power while the logic remains virtually unchanged|exactly what\average
programmers"need.
The second step was programming the Sun SPOT to behave as a simple
pervasive computing device with\+1"and\-1"buttons.When the buttons
are pressed,the device posts the appropriate value to a given web URI as a
given parameter.To link our new pervasive device to our web application,
we simply provided the URI of the new UbiCoee page and the parameter
name to the device.Concretely,this means that the Sun SPOT sends an
29
Figure 2.4:Data ow for UbiCoee,before and after SEAP.
HTTP packet to http://192.168.0.1/UbiCoffee?amountChanged=1.The
resulting system now accepts data from users via the Sun SPOT device or the
traditional web interface.
2.3.2 SEAP From Scratch
We have access to a wide variety of robotic components like remote-
control vehicles,pan-and-tilt servos,cameras,infrared sensors,sound sensors,
GPS systems,and passive- and active-RFID equipment [104].SEAP is an
ideal method to interconnect these disparate devices.We decided a demo that
remotely controlled the vehicles would be interesting,and SPOT-to-Bot was
born.Shown in Figure 2.5,the Spot-to-Bot application [103] was developed
by an undergraduate researcher in our lab to steer a Roomba iRobot [51,115]
30
using the accelerometer on a Sun SPOT.In this case,the SEAP architecture
was used to break the complete application into three distinct parts,each
independently testable and exchangeable.
SPOT-to-
Bot

SPOT
Accelerometer
(X, Y, Z) Input
Angle
Output
Roomba

Speed
Output
SEAP

Application
Logic
Figure 2.5:The SPOT-to-Bot data ow.
The rst of the three components is a web application with three pages.
The rst page is a web form that accepts three values:\X",\Y",and\Z".
These values can be provided by the Sun SPOT accelerometer,but during
development these values were provided by submitting the form from a web-
browser.The second and third pages of the web application display one value
each:the requested speed and turning angle for the robot.The web application
thus contains the logic to translate the raw data provided by the sensor into
the commands given to the actuator.
The second component of the system delivers accelerometer data from
the Sun SPOT device to the web server.This component is a simple exten-
sion of the program developed for the UbiCoee application described in the
previous section.The third system component drives the Roomba robot.We
already had code to drive the Roomba given speed and angle,so the majority
of this work fell to retrieving and parsing values provided by the web server.
31
The resulting application uses three components that can be indepen-
dently tested and veried.We are able to easily test each component because
we use standard protocols;our model for sensors and actuators takes the place
of standard web forms,so users can debug using familiar web forms rather than
obscure pervasive computing devices.The resulting application is exible and
easily extended to accept new sensors,actuators,or behaviors.In fact,the
three components can even use dierent programming languages,each speci-
cally suited to the component's primary task (e.g.,JavaME for the Sun SPOT,
PHP for the server,C++ for the Roomba controller).
2.4 SEAP Evaluation
We created several applications using SEAP,but,as the creators,our
experience is less informative than the experiences of our intended audience:
average programmers.To evaluate SEAP,we performed a user study to com-
plete the nal research task for this work:Research Task 3.Evaluate the
usefulness of SEAP on a selection of novice software developers.
2.4.1 Participants
We studied 10\average programmers"in a user study that mirrored the
SEAP development process as it would occur outside the study.We recruited
ve undergraduate students and ve graduate students with a rst-come rst-
served letter targeted at Electrical and Computer Engineering students.Par-
ticipants were all male and ranged in age from 19 to 29.
32
< 1
1 to 2
3 to 4
> 4
Years programming
1
2
2
5
Programming courses taken
0
3
1
6
Table 2.1:Experience for the SEAP user study participants.
YES
NO
Created website?
8
2
Created web application?
6
4
Created mobile application?
4
6
Table 2.2:Development experiences for the SEAP user study participants.
Table 2.1 shows the years spent programming and number of university
programming courses taken for SEAP study participants.Many of the users
had been programming for years in school,but only one had held a full-time
programming position.This ts our\average programmer"denition.
As Table 2.2 shows,the majority of our participants had created a
website using static HTML/CSS and a majority had programmed a dynamic
web application using the languages detailed in Table 2.3.
Web Application Language
Number of Users
Perl
1
PHP
4
Python
1
Ruby
4
.Net
1
Java
2
Table 2.3:Languages that SEAP participants used to create web applications.
33
Mobile Application Language
Number of Users
iOS (Objective C)
2
TinyOS
2
Embedded Linux (shell)
2
Table 2.4:Languages used by SEAP participants to create mobile applications.
Fewer users had created mobile applications (the languages used are
in Table 2.4),but the iPhone had drawn two users in while the Agilla middl-
ware [32] enabled another user to explore TinyOS [62] and embedded linux.
Another user had completed a project that used TinyOS and an embedded
linux robotics platform for his pervasive computing class.
2.4.2 Setting
Studies took place in a quiet room with the participant and the author.
All necessary materials were clearly visible.The items present were
 a MacBook Pro laptop running all necessary software,
 Flip SlideHD video recorder,
 three Sun SPOT devices,
 study cover letter,
 consent form,
 printed pre-test,
 printed post-test,and
34
 a primer (available in print or PowerPoint)
Participants were instructed to follow the primer,which included back-
ground information and directions for each task,and ask questions only when
necessary.The primer and tasks were meant to emulate what average pro-
grammers would encounter in the real world.
2.4.3 Study Tasks
The study required users to interact with sensors and actuators and
alter a running SEAP application.The application used a SPOTaccelerometer
to change the color and number of LEDs illuminated by another SPOT device.
(A Sun SPOT device is shown in Figure 2.6
1
.) Many intelligent environment
visions include automatically adjusting lights,so this was a fairly realistic
application.
We sought to test the basic SEAP actions by asking participants to rst
use the system using only the web browsers,then alter the server logic,then
use real sensors and actuators.To eliminate confusion about when and how
the tasks were to be completed,we included the context in each task.\Safari"
refers to the web browser that was open on the test machine.\Eclipse"is an
integrated development environment (IDE) that was open on the test machine.
Sun SPOT devices are identied by a 16 digit string that is printed near the
top;by convention,we identied the test devices with the last eight digits
1
http://johnsjavapda.blogspot.com/2007/05/sun-spot-newbie-time.html
35
Figure 2.6:A Sun SPOT device.
(e.g.,0000.47DD),although only the last four digits are currently in use.
The seven tasks were the following:
 Task 1a.In Safari navigate to accelerometer.jsp and input values from
[-1,1] for X,Y,and Z
 Task 1b.Change the color shown in color.jsp by inputting data through
the form on accelerometer.jsp
 Task 1c.Change the displayed number by inputting data through the
form on accelerometer.jsp
 Task 2a.There is an error in the output of number.jsp...values should
be in the range [0,8]...Fix number.jsp in Eclipse
 Task 2b.In Eclipse,alter color.jsp to incorporate Z in the color output
36
 Task 3a.Move Sun SPOT 0000.47DD to change the values in color.jsp
and number.jsp
 Task 3b.Move Sun SPOT 0000.47DD to change the number of LEDs
and their color displayed on Sun SPOT 0000.342E
Combined,these tasks forced participants to program and debug the
hardware and the software as they would outside the laboratory setting.
2.4.4 Questions to Answer
We hypothesized that students would understand the basic logic,strug-
gle with the devices,and performmoderately well with web programming.Our
primary question was,\Is SEAP usable by average programmers?"This ques-
tion could be answered\yes"if each participant could successfully complete
the tasks before getting frustrated or hitting the deadline:one hour.(The
hour deadline was chosen because it is a round number and informal tests
to tune the study were completed in about an hour by non-programmers.)
Answering\no"would occur if no average programmer could complete the
tasks.There are many shades in between these end-points and we thought
documentation might transform any failures into successes,so we had the fol-
lowing question in mind:\If the user failed to complete a task,would improved
documentation/support mitigate the issue?"
In general we expected that users would be able to use the system
without a deeper understanding of sensors or the Internet.To verify this
37
hypothesis we planned to answer the following questions for each participant:
 Did the participant complete the test successfully?
 How long did it take the participant to complete the test?
 What questions did the participant ask?
Examined singly,the above questions will not be particularly illumi-
nating;however,the aggregate data will help us explore these meta-ideas:
 Can average programmers use SEAP to understand and interact with
pervasive computing applications?
 Were there any common questions asked?
 Was there a step that slowed a majority of people?
 Is the framework usable in its current form?
 If users struggled,would improved documentation remedy the problem?
2.5 Study Results
Every participant successfully completed the study within the time pro-
vided (further discussion of time on task is below).The study indicates that
SEAP is not only usable,it is also very exciting for everyone who was exposed
to it.As shown in Table 2.5 feedback was overwhelmingly positive.Users felt
38
Strongly Agree
Agree
Disagree
Strongly Disagree
Tasks doable in
time provided
10
0
0
0
Further interest
in SEAP?
5
5
0
0
Table 2.5:User feedback on the SEAP study.
the tasks necessary to edit an example application and the devices associated
with that application were completely doable in the time given.
We did not explicitly quiz subjects on their understanding of the sys-
tem,but based on discussion and observation we believe that 7/10 subjects
learned the fundamental concepts by the time they completed the test.Through-
out the test we would ask users why they had performed an action or why the
output was the way it was.People would explain how the pieces t together
and why events had transpired;they would explore the system more deeply
and make accurate predictions about what would happen.This yields two
noteworthy conclusions:users without a deeper understanding can still suc-
ceed,and the SEAP system is easy to learn.
To start each study,we provided the primer then asked the participants
to work through the tasks at their own pace.At the beginning of each task,
we would note the time rounded to the closest 15 second interval.Table 2.6
shows the time spent on each task in the SEAP user study;dashes appear
2
The sum is skewed for Users 4,6,and 10 due to missing values.We remove them in
Table 2.7
39
Task
User
1a
1b
1c
2a
2b
3a
3b
Sum
2
1
2.75
3.00
1.00
9.25
3.50
1.25
2.25
23.00
2
1.00
1.00
2.00
4.75
11.00
11.75
5.00
36.50
3
0.75
3.25
1.50
8.75
6.00
1.75
18.50
40.50
4
0.50
-
-
-
-
7.50
7.50
15.50
5
2.00
3.50
1.50
5.00
9.00
4.00
4.75
29.75
6
0.50
0.75
0.75
1.50
2.75
-
1.50
7.75
7
2.75
1.00
1.50
3.25
6.50
3.50
3.00
21.50
8
0.25
7.25
1.25
7.75
24.25
2.00
2.25
45.00
9
0.25
0.75
0.50
5.25
6.75
5.75
2.25
21.50
10
1.00
0.50
7.75
3.50
2.75
-
7.75
23.25
Mean
1.18
2.33
1.97
5.44
8.06
4.69
5.49
29.15
Table 2.6:Minutes spent for each task in the SEAP study.
where the time could not be recorded.User 4 completed tasks very quickly,
and we were not watching closely enough to accurately note the time e.g.,by
the time we were done writing notes for Task 1c he was done with 2a;times
for Users 6 and 10 were not recorded because we were engaged in discussion
about the system and it was not clear when Task 3a ocially began.Table 2.7
provides only the completely timed studies and shows that users spent about
21 minutes minutes to 45 minutes on the entire study with an average around
31 minutes.
2.5.1 Browser-based Tasks:1a,1b,1c
The rst tasks (1a,1b,1c) had people input X,Y,and Z values through
a web browser.These inputs led to update colors and numbers found on
another page.Users moved through the browser-based tasks quickly.Variance
40
Task
User
1a
1b
1c
2a
2b
3a
3b
Sum
1
2.75
3.00
1.00
9.25
3.50
1.25
2.25
23.00
2
1.00
1.00
2.00
4.75
11.00
11.75
5.00
36.50
3
0.75
3.25
1.50
8.75
6.00
1.75
18.50
40.50
5
2.00
3.50
1.50
5.00
9.00
4.00
4.75
29.75
7
2.75
1.00
1.50
3.25
6.50
3.50
3.00
21.50
8
0.25
7.25
1.25
7.75
24.25
2.00
2.25
45.00
9
0.25
0.75
0.50
5.25
6.75
5.75
2.25
21.50
Mean
1.39
2.82
1.32
6.29
9.57
4.29
5.43
31.11
Table 2.7:Minutes spent for each task in the SEAP study with incompletely
timed entries removed.
is based on how quickly the user read,how eectively they switched from task
description to web browser,and how deeply they tested the system.The
quickest users generally tried the fewest number of input values.
2.5.2 Programming Tasks:2a,2b
Task 2a and 2b required the user to program;these tasks were per-
formed slower than the previous browser-based tasks.
Task 2a required the user to debug the number output:\There is an
error in the output of number.jsp...values should be in the range [0,8]..."
The code for number.jsp is shown in Figure 2.7.The quickest x simply
scaled the output by 8.0 instead of 100.0.More advanced answers required
mathematical calculations.Users tried a range of solutions including modulo,
%,absolute value,Math.abs(),multiplication and division.We saw people
scale the input (getY()) and the entire output (by introducing parentheses
41
around the entire equation).Participants were slowed by choosing the right
operation then fully verifying the x.It was this task that led a participant
to ask about JSP syntax with respect to absolute values.
<%= (int)(spot47dd.getY() * 100.0) %>
Figure 2.7:The code for task 2a,number.jsp.
The longest task,2b,asked users to incorporate Z into the color dis-
play (\In Eclipse,alter color.jsp to incorporate Z in the color output").Users
were told that colors may be in the set fBLUE,GREEN,ORANGE,RED,WHITE,
YELLOW,CHARTREUSE,CYAN,MAGENTA,MAUVE,PUCE,TURQUOISEg,which
are the colors supported by the SPOT LEDs.Code for this task is shown
in Figure 2.8.Notice the two getX() methods.The simplest solution we saw
simply replaced X with Z (as in getX() =)getZ()).
String color ="YELLOW";
if (spot47DD.getX() < -0.2) f
color ="RED";
g
if (spot47DD.getX() > 0.2) f
color ="GREEN";
g
Figure 2.8:The code for task 2b,color.jsp.
From the time on task data,we see that User 8 spent an extended
period (over 24 minutes) on task 2b.User 8 quickly replaced X with Z,but
then attempted to incorporate every possible color.He performed a number
of calculations to determine evenly spaced intervals then meticulously coded
42
and tested each state.After several minutes of unexpected results,the user
discovered that his precise intervals were being interpreted as integer (whole)
numbers rather than oating point (decimal) numbers;instead of 12 evenly
spaced ranged from -1 to 1,the machine was seeing only three choices:-1,
0,and 1.After xing this bug,he exclaimed triumphantly and continued
development.
2.5.3 Device Tasks:3a,3b
The last two tasks required the users to handle the SPOT devices.Task
3a asked users to aect color.jsp and number.jsp by moving the input SPOT.
Almost the entire time on task was spent guring out the SPOT.Users who
could accurate predicted the outcomes of the system (they would say things
like,\when I input.1 for XI expect the color to be yellow") posted the shortest
times,while the longest time on task came from a user who tried all manner
of manipulations in an eort to understand the system.
The second longest task,Task 3b,asked users to manipulate the input
SPOT then verify the output using the LEDs on the other SPOT.Task 3a
provided an opportunity to learn the input device,so Task 3b isolated the
output device and was generally nished quickly.Taking almost 19 minutes,
User 3 was an exception to this rule.After accomplishing the basic task,this
user continued to play with the system by altering number.jsp and color.jsp
and experimenting with the SPOTs.The timing should have stopped much
earlier as the user clearly understood the task;however,we were not sure
43
exactly when the user was done,so the entire duration was recorded.
As the end of a section,Task 2b and 3b,were natural stopping points
where people would attempt to understand everything they had done up until
that point.It was not uncommon for users to revisit earlier tasks or attempt
to nd corner-cases in an ad hoc debugging/penetration testing session.
The study was a great success,but it was still imperfect.Next,we
detail diculties we encountered and the resulting lessons learned.
2.5.4 Diculties
We asked users to discuss the most dicult part of the study.One user
provided high praise by saying that his biggest hurdles occurred because he was
\too excited about the project".Other users admitted a lack of knowledge in
three general areas:unknown devices,the web server software,and the study
platform.The rst two issues are central issues in intelligent environments,
while the third was an issue with the test setup.All three areas are explained
below.
2.5.4.1 Devices
The piece of feedback that stood out was that the\lack of knowledge
about the sensors made it dicult to determine if my problems were in soft-
ware or hardware."This sentiment captures one of the biggest challenges in
intelligent environments.
Part of this confusion lies in the test setup.The Sun SPOT input
44
device was congured to use the X,Y,and Z dimensional accelerometer
3
.Ev-
ery participant struggled to understand the accelerometer and one wrote that
\dealing w/the Sun SPOTs accelerometer was a bit confusing".Many users
slowly and uniformly moved the device;when queried about their thoughts,
they said they were trying to change the values relative to the starting po-
sition,so we had to explain that the device was not a position sensor like
GPS,instead it was an accelerometer.Usually,this led users to then tilt the
device and check the values.When the user got frustrated or asked what was
going on,we would explain that the device was not a tiltometer,it was an
accelerometer.
Once people conceptually understood an accelerometer,many had a
hard time determining dimensions and expected values.For example, inging
the device towards the oor did not necessarily send a negative Y acceleration.
Leaving the device untouched on the table would also generate non-zero val-
ues.This led to feedback like,\understanding the axis of the actuator"and
\guring out the sun spot orientation."
Another frustration that we observed arose from the 10 second delay
programmed for each device.In the nal task,3b,users expected immediate
results.They would verify the value from the webpage then say something
like,\the lights should be cyan now."If the SPOT LEDs did not illuminate
properly before the user tried a dierent tact,we would intervene and ask for
3
an accelerometer measures acceleration,a dierence in speed or direction.
45
patience.
In some tests,the SPOT devices would lose connection and have to be
restarted.From our experiences with the devices,we believe there is an issue
with the manufacturer's connection algorithms.Disconnections seem to occur
after periods of inactivity,usually caused by repeated collisions.In an end-
user's intelligent environment,a aky connection would be very frustrating;
however,in the lab setting it was simply something to watch for and intervene
as necessary.
2.5.4.2 Server Programming Language
A couple of the less experienced programmers noted that their biggest
hurdle was that they\do/did not understand JSP language"and were slowed
by a\lack of familiarity with JSP."We knew that users would probably not
know the JSP language and these comments conrmed that.
2.5.4.3 Platform
There were some issues with the platform used for the study.Studies
were performed using a MacBook Pro running the Mac OS X operating sys-
tem.Participants were asked to use the Safari web browser and Eclipse IDE.
One subject said that their greatest obstacle was,\ignorance of Mac OS X"
though other users mentioned they were unfamiliar with Safari and Eclipse.
We saw users try to refresh with the F5 key,which works in many browsers
and operating systems,but not OS X (F5 aects the keyboard backlight).
46
Confusion about keyboard shortcuts were common;we observed alt+tab to
scroll through open applications and control+c/control+v to copy/paste.Ad-
ditionally,the trackpad proved unintuitive.Users could move the mouse,but
many had to learn how to left click,right click,and click-and-drag.
2.5.5 Lessons Learned
In designing and administering this study,we learned a great deal.Be-
low we discuss the general lessons learned and those based on the frustrations
we observed.
2.5.5.1 Devices
If we repeated this study,we would congure the SPOT to be used
a tilt sensor that is more easily explained and immediately understood.To
reduce the jitter when the device was held gingerly or positioned on a solid
surface,we would also adjust the sensitivity,probably by reporting changes
above a certain threshold.The 10 second refresh rate should be reduced to
increase response time.
2.5.5.2 Server Programming Language
Despite not knowing the language a priori,the participants were able
to complete the tasks quickly and successfully.We did not anticipate this
challenge,but there is little we could do without introducing heavy selection
bias (for example,by only testing programmers with JSP experience).Despite
47
these concerns,users completed the tasks quickly.Very few users consulted
documentation or asked questions specic to JSP.
Based on our results,we believe that average programmers could adapt
to other web languages;however,to simplify the programming of intelligent
environments,users should be allowed to use whatever language they are most
comfortable with.
2.5.5.3 Platform
Issues with the platform were only minor slow downs,but we would
like to alleviate this sort of anxiety in the test setting.One option is to oer
a greater variety of platforms,although we believe too many options could
lead to choice paralysis [94] and negatively impact the results.Instead,we
would provide a short introduction to performing the tasks using the software
provided.
2.5.5.4 Working with Human Subjects
Before beginning the study we wanted to measure time on tasks and
take note of the number and type of questions.Once the study began,we
found it dicult to accurately note these quantities.Notable distractions
arose when users asked questions,required assistance,or went o course (e.g.,
they continued on a previous task).One potential x would be to use software
like Morae
4
that can simultaneously record the user's screen and their face.
4
http://www.techsmith.com/morae.asp
48
Armed with this information,we could review the sessions to time actions and
take additional notes.
A computer based recording program might also have reduced di-
culties that we encountered with recording the user sessions.However,we
believe the presence of the camera intimidated some participants and may not
be worthwhile at all.As such,we do not feel that we have missed anything
due to a lack of recordings.
We did not notice a problem,but the data collection sheet included
a section entitled\Errors."This section was aptly named,but it had the
potential to make participants uncomfortable if they saw the data collection
sheet.Instead,we should have simply made the\Comments"section larger.
It was very interesting to observe people as they explored the system;
some sought deep understanding while others simply strove to nish.Under-
standing users without impeding their progress or biasing them was a tremen-
dous experience.We learned a lot about the process,but the lessons are too
subtle to express.
2.6 Discussion
SEAP provides three major research contributions:a method for mod-
eling embedded devices as web-compatible objects including a mechanism for
dynamic conguration,the embodiment of these concepts as a middleware,
and a user study that demonstrates that SEAP can be understood and used
49
by average programmers.We have worked extensively with SEAP,develop-
ing several applications and exploring the possibilities of the middleware.We
have developed applications from scratch and altered existing web applica-
tions,allowing them to interact with sensors and actuators dynamically.We
have also demonstrated and presented the research,which generated a wealth
of interesting discourse.
2.6.1 Challenges
SEAP addresses usability issues that currently impede average pro-
grammers.SEAP is applicable to a large number of projects;however,it is
not ideal for environments where security,speed,or precision are of utmost
importance.
2.6.1.1 Speed and Precision
SEAP does not specify a communication protocol,but we believe that
wireless protocols should be used to keep intelligent environments tidy and
easy to use.IEEE 802.11(a/b/g/n) wireless communication is a realistic choice
because of its ease of use,widespread understanding,and general availability.
The Compact Application Protocol (CAP) [23] is bringing the same ease-of-use
to the increasingly common suite of IEEE 802.15.4 devices,so personal area
networking protocols like Zigbee may also be suitable.Wireless communication
provides many benets,but physical limitations mean that radio transmission
will always be slower than wired communication [108].
50
Factory automation and other real-time systems do not generally use
wireless communication,opting instead for lean,custom communication pro-
tocols transmitted over wire (Ethernet,copper,ber,etc.);this maximizes
communication speed and reduces overhead.SEAP-style HTTP interactions
are unlikely to disturb the status quo here.However,recent research shows
that wireless communication could be used for factory automation [82],so
SEAP may be viable.Further research is necessary to conrm or deny.
2.6.1.2 Security
Wireless security is hard,but there are many ways to mitigate the
risks [83].We believe that security,usability,and speed are at odds in system
design.SEAP is highly usable while still being as fast and secure as other
web applications.Because we use standard HTTP,we can piggyback on any
security improvements that nd their way into communication protocols,web
servers,and devices.
One option is to communicate using HTTPS (HTTP Secure) [87].The
aesthetic dierences are minor:a dierent URI (\https"versus of\http")