and Human-Computer Interaction

builderanthologyΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 4 χρόνια και 20 μέρες)

248 εμφανίσεις

5
-
1

5. Computer Science

and Human
-
Computer Interaction


5.1. Motivation


5.2. User Interface Software Concepts


5.2.1. Design Process for Interactive Systems


5.2.2. Interaction Techniques


5.2.3. Interaction Styles


5.2.4. User Interface Management Systems


5.2.5. User Interface Description Languages


5.3. User Interface Software Tools


5.3.1. Window Systems


5.3.2. Toolkits


5.3.3. UIMS


5.4. Future Directions

5
-
2

5.1. Motivation


Easier to use, but harder to program [Myers]


Multiprogramming


Real
-
time programming


Robustness


Testability


Modularization


Need to support iterative design and testing


Software tools

5
-
3

5.2. User Interface

Software Concepts


Dialogue independence


Interface (syntax) vs. application (semantics, business
logic)


Model
-
View
-
Controller architecture


Interaction techniques


Basic interaction styles


Design levels: conceptual, semantic, syntactic,
lexical


Generate UI from abstract description


Early research in user interface management
system (UIMS) and user interface description
languages (UIDL)


User interface software tools


New interaction styles (non
-
WIMP)

5
-
4

5.2.1. Design Process

for Interactive Systems


Partition into Four Levels [Foley & Wallace]




Conceptual


Semantic




Syntactic


Lexical

5
-
5

5.2.2. Interaction Techniques


Interaction task


Interaction device


Interaction technique


A way of using a device to perform a generic HCI task


Widget in a toolkit


Two information processors communicating via
constrained interface [Tufte]


Goal: Increase bandwidth across channel


5
-
6

5.2.3. Interaction Styles: Current


Command Language


Question and Answer


Form
-
based


Menu


Natural Language


Direct Manipulation

5
-
7


Virtual and augmented reality


Tangible interfaces


Ubiquitous, pervasive, handheld


Lightweight, tacit, passive, non
-
command, context
-
aware


Affective computing


Multi
-
modal interfaces


Reality
-
based interfaces


Brain
-
computer interaction


Interaction Styles: Emerging

5
-
8


Dialogue Independence [Hartson & Hix]


User Interface Management System (UIMS)
Approach [Olsen]


Before





After






Levels of design:

Conceptual [Johnson &
Henderson], Semantic, Syntactic, Lexical

5.2.4. User Interface Management
Systems Research





Application



and its own



User Interface



USER



USER



UIMS



Application



(Syntactic+Lexical)



(Semantic)





5
-
9

Classes of Programmers

and Tool Users



Runtime

Application

End



User



UIMS

UIMS



Tools



User

Interface

Designer

Application

Programmer

(UIDL)

5
-
10


Same UIMS for many applications







Different interfaces to same application







Interface to null application (rapid prototyping)


Dialogue Independence

USER

Application

1

Application

2

Application

3

UIMS

User

Interface

1

User

Interface

2

User

Interface

3

Application

USER

5
-
11

5.2.5. User Interface

Description Languages


How does the user interface designer describe
the desired user interface to the UIMS or toolkit?


Layout


Behavior


With a User Interface Description Language
(UIDL)


May be interactive


Generate UI from abstract description


UIDL describes user
-
visible behavior, but not
implementation


More in Section 5.3.3


Other uses of UIDL (evaluation)

5
-
12


Goal: Support dialogue independence and
separation of programming roles [Foley, van
Dam, Feiner, & Hughes]


5.3. User Interface Software Tools

Toolkit

Application

Window system/graphics package

Operating system

Hardware

UIMS

5
-
13

5.3.1. Window Systems


Manage shared resources (pixels, keyboard,
mouse), provide clipping, translation


Tells client when exposed, resized, or other
damage


Could implement in different places


Single built
-
in window manager or allow separate


Dispatcher









(N.B. Security classifications shown are simulated only.)

AccessWindowMouseEvent (Event *e) {


. . .

FileWindowMouseEvent (Event *e) {


If (e
-
>type==MS_LEFT) . . .


5
-
14

5.3.2. Toolkits


Library of reusable routines for implementing
human
-
computer interface features


Basic components


Set of widgets


"Intrinsics"


Does not try to describe or support user
interface as a whole


Standard look and feel


Encourage consistency


But imposes limits on interaction style

5
-
15

Toolkits (cont'd)


Examples


Visual Studio (Windows)


Interface Builder (Macintosh)


GTK (X/Unix/Linux)


Based on controls or "widgets"











(N.B. Security classifications shown are simulated only.)

AccessCheckBox (Boolean val)

ClassifTypein (char *val)

DisplayMsgFile ()

5
-
16

5.3.3. UIMS


Higher level than procedure library


May include main program, provide external
control of application


Call semantics/business logic as subroutines


Message to component software


Design time, run time, and evaluation time tools
[Myers]


Module partitioning: Runtime vs. interface design
time partitioning


5
-
17

Model
-
View
-
Controller


Design pattern, originally Smalltalk


Found, in some form, in most UI toolkits


Matches dialogue independence,
syntactic/semantic split


Extend to web browser



USER



Model

Controller

View

5
-
18

User Interface

Description Language


Designer creates it, UIMS executes it


Specific language or technique suitable for describing user
interfaces


Describe user
-
visible behavior, but not implementation


Goal: Generate a UI from high
-
level, abstract
description


Platform
-
independent if possible (Windows, Mac, X)


Platform
-
type
-
independent if possible (desktop, cellphone,
voice)!


Model
-
based UI development [Paterno; Jacob, Limbourg, &
Vanderdonckt]


Current GUI practice


Has evolved from general purpose UIDL into small, stable set
of widgets and classes


Change with new UI styles, portability needs


Choice of design language at heart of UIMS

5
-
19


Based on techniques for specifying static
languages


Both can be modified for interactive languages


BNF

send
-
cmd::= SEND DRAFTNAME RECIPNAME


State Transition Diagram




Alternate Form of State Transition Diagram

start: SEND
-
>getdr

getdr: DRAFTNAME
-
>getre

getre: RECIPNAME
-
>end

Basic Classes of UIDLs

start

getdr

getre

end

SEND

DRAFTNAME

RECIPNAME



5
-
20

Early UIMS/UIDL Research


Language
-
based


Menu Network [Kasik]


BNF, grammar
-
based [Olsen]


State transition diagram [Newman]


Event language


Declarative


Non
-
language
-
based


Interactive builder (bottom up)


Current state of practice, use GUI techniques to specify GUI


Plug together components with
pre
-
defined

interactive behavior


Automatic generation (top down) [Foley]


By demonstration [Myers]


Other categories


Constraint
-
based [Hudson]


User
-
oriented [Moran]

5
-
21

Example:

Visual Programming for UI


Layout vs. Behavior


UIMS, with state diagrams for behavior








[Jacob, “A state transition diagram language...”]


Visual Basic, pre
-
programmed behavior




5
-
22

Example: State Diagram as UIDL




Command
-
driven Graphical Report Editor












[Jacob, “Using formal specifications...”]


5
-
23

UI Evaluation


Directly from UIDL


Evaluate interface without building system


Emerging area, but lack underlying formal knowledge
of UI design


Analyze user interface to detect specific properties
[Darlington et al.]


Apply human performance models [Card]


Automated screen design critique [Tullis]


Prove safety
-
critical properties


Screen mockup, prototyping tools


“Slide show" of canned screens based on user inputs


UIMS sans application (use stubs for semantics)


Visual Basic, Powerpoint, plain HTML also used this
way

5
-
24

UI Tools: Standardization


Early:


Many desktop platforms


No standardization


Current:


~3 major platforms, standard widgets


Settled into stable toolkits, widget libraries, GUI class
libraries


UIMS seems like overkill


Emerging:


Mobile, automobile


Diverse platforms, no standardization


Wider range of interaction techniques


Model
-
based development, UIMS redux


Future:


Same issue for new non
-
WIMP interaction styles


Still wider range of possible interactions

5
-
25

UI Tools: Trends


UI toolkits, libraries


Become important, integral part of software
development tool sets


e.g., Macintosh Cocoa UI features are well integrated


Good uptake despite unusual programming language


Web browser as UI platform


Cloud computing


Chrome
-
only OS


New tools for web UIs, but still clumsy, poorly
integrated


Making cross platform toolkits less significant

5
-
26

UI Tools: Mobile Devices


Basic HCI development and design principles


Largely GUI, not new interaction style + gestures, GPS, ...


Use context: Full attention or not?


Development tools


Early: Clumsy tools, restricted UI designs


Much like early GUIs, function key
-
driven (pre
-
mouse) GUI


Current: UIs and tools very similar to standard GUI


iPhone: Interface Builder (Mac, Objective C)


Droid: Java + Swing (minor differences)


UI Consistency


Low level consistency


High level models, UIDL could help


Example: IMAP mail clients

5
-
27

5.4. Future Directions:

Emerging Interaction Style


Parallel, Highly
-
interactive (non
-
WIMP)


Lightweight, Context
-
aware, Non
-
command


Continuous (+ Discrete)


Tangible Interfaces


Microprocessors, 3D forms


Ubicomp, Pervasive, Context
-
aware


Sensors, networks


Reality
-
based Interaction


Brain
-
computer Interaction





5
-
28

Implications for Software

Current


Future


Single thread I/O


Parallel, asynchronous
dialogues; may be interrelated


Discrete tokens


Continuous inputs and
responses (plus discrete)


Precise tokens


Probabilistic input, not easily
tokenized


Sequence, not time


Real
-
time requirements,
deadline
-
based


Explicit user

commands


Passive monitoring of user


5
-
29

Reality
-
Based Interaction


Connects several emerging interaction styles


Understand together as a new generation of HCI
through RBI [Jacob et al., “Reality
-
Based Interaction...”]


Computer interaction more like rest of world


Exploit skills and expectations user already has about
simple, everyday, non
-
digital world


Reality + extensions


Design tradeoffs:

5
-
30

Tangible User Interfaces


Simple, transparent
mechanical structures


Use knowledge of
physical world to
operate


Augment physical
objects with digital
meaning


Combine physical +
digital representations


Exploit advantages of
each



[Jacob et al., “A Tangible Interface...”]

[Zigelbaum et al., The Tangible Video Editor...”]


5
-
31

Technology for TUI


Sensors


Produce signal in response to change in surroundings


Actuators


Produce physical change in response to signal


Microcontroller


Communicates with devices and with main computer


Other technologies


RFID


Computer vision


...


Wider range of I/O events than GUI


Development methodology, tools


None!


5
-
32


A Visual Language for
Modeling Tangible User
Interfaces [Shaer]


TAC paradigm


Each TUI consists of token
within a constraint


Same object may
sometimes be token,
sometimes constraint


Two tier model fits well


Dialogue (states,
storyboard)


Interaction (especially
continuous)


TUIML

5
-
33

Conclusions:

1. Research Background


Background for following current research in
user interface software


Interaction styles


Dialogue independence


Four levels of design


User interface management system (UIMS)


User interface description language (UIDL)


Goal: Generate UI from abstract description


Types of software tools


New interaction styles (non
-
WIMP) and requirements


Easier to use, harder to program?


5
-
34

Conclusions:

2. Practice


What research ideas can be applied in building
user interfaces today?


Separate interface from application programming


Model
-
View
-
Controller architecture


Use specialized tools (UIMS) for leverage


Use specialized languages (UIDL) for perspicuity and
portability

5
-
35

Sources


Textbooks


Olsen


Foley, van Dam, Feiner, & Hughes


Shneiderman, Plaisant, Cohen, & Jacobs


History


Early UIMS research [Buxton et al.; Foley and Wallace; Jacob;
Kasik; Newman; Olsen]


Classic surveys: ACM Transactions on Computer
-
Human
Interaction [Myers, Hudson, & Pausch], Computing Surveys
[Hartson & Hix]


Current research


ACM UIST Conference on User Interface Software and
Technology (annual conference)


ACM EICS Symposium on Engineering Interactive Computing
Systems (conference)


CADUI Conference on Computer
-
Aided Design of User
Interfaces (conference)


ACM Transactions on Computer
-
Human Interaction (journal)


interactions (ACM magazine)

5
-
36

References

W. Buxton, M.R. Lamb, D. Sherman, and K.C. Smith, ''Towards a Comprehensive User Interface Management
System,'' Computer Graphics 17(3) pp. 35
-
42 (1983).

S.K. Card, T.P. Moran, and A. Newell, ''The Keystroke
-
Level Model for User Performance Time with Interactive
Systems,'' Comm. ACM 23 pp. 396
-
410 (1980).

J. Darlington, W. Dzida, and S. Herda, ''The Role of Excursions in Interactive Systems,'' International Journal of
Man
-
Machine Studies 18 pp. 101
-
112 (1983).

J. Foley, W.C. Kim, S. Kovacevic, and K. Murray, ''Defining Interfaces at a High Level of Abstraction,'' IEEE
Software 6(1) pp. 25
-
32 (January 1989).

J.D. Foley, A. van Dam, S.K. Feiner, and J.F. Hughes, Computer Graphics: Principles and Practice, Addison
-
Wesley, Reading, Mass. (1990).

J.D. Foley and V.L. Wallace, ''The Art of Graphic Man
-
Machine Conversation,'' Proceedings of the IEEE 62(4)
pp. 462
-
471 (1974).

H.R. Hartson and D. Hix, ''Human
-
computer Interface Development: Concepts and Systems for its
Management,'' Computing Surveys 21(1) pp. 5
-
92 (1989).

S.E. Hudson, ''Graphical Specification of Flexible User Interface Displays,'' Proc. ACM CHI'89 Human Factors
in Computing Systems Conference pp. 105
-
114 (1989).

R.J.K. Jacob, "A Specification Language for Direct Manipulation User Interfaces," ACM Transactions on
Graphics, Vol. 5(4) pp. 283
-
317 (1986) http://www.cs.tufts.edu/~jacob/papers/tog.pdf].

R.J.K. Jacob, ''A State Transition Diagram Language for Visual Programming,'' IEEE Computer 18(8) pp. 51
-
59
(1985) [http://www.cs.tufts.edu/~jacob/papers/ieeecomputer.pdf].

R.J.K. Jacob, "Using Formal Specifications in the Design of a Human
-
Computer Interface," Communications of
the ACM, Vol. 26(4) pp. 259
-
264 (1983) [
http://www.cs.tufts.edu/~jacob/papers/cacm.pdf
].

R.J.K. Jacob, A. Girouard, L.M. Hirshfield, M.S. Horn, O. Shaer, E.T. Solovey, and J. Zigelbaum, "Reality
-
Based Interaction: A Framework for Post
-
WIMP Interfaces," Proc. ACM CHI 2008 Human Factors in
Computing Systems Conference pp. 201
-
210, ACM Press (2008)
[http://www.cs.tufts.edu/~jacob/papers/chi08.pdf].

R.J.K. Jacob, H. Ishii, G. Pangaro, and J. Patten, "A Tangible Interface for Organizing Information Using a
Grid," Proc. ACM CHI 2002 Human Factors in Computing Systems Conference pp. 339
-
346, ACM Press
(2002) [http://www.cs.tufts.edu/~jacob/papers/chi02.pdf].

R.J.K. Jacob, Q. Limbourg, and J. Vanderdonckt (Eds.), Computer
-

Aided Design of User Interfaces IV, Kluwer
Academic Publishers, Dordrecht (2005).

J. Johnson and A. Henderson, "Conceptual Models: Begin by Designing What to Design," interactions, Vol. 9(1)
pp. 25
-
32 (January 2002).

D.J. Kasik, ''A User Interface Management System,'' Computer Graphics 16(3) pp. 99
-
106 (1982).

T.P. Moran, ''The Command Language Grammar: A Representation for the User Interface of Interactive
Computer Systems,'' International Journal of Man
-
Machine Studies 15 pp. 3
-
50 (1981).

B.A. Myers, ''User Interface Software Tools,'' ACM Transactions on Computer
-
Human Interaction 2(1) pp. 64
-
103 (March 1995).

B.A. Myers, Creating User Interfaces by Demonstration, Academic Press, Boston (1988).

B. Myers, S.E. Hudson, and R. Pausch, "Past, Present, and Future of User Interface Software Tools," ACM
Transactions on Computer
-

Human Interaction, Vol. 7(1) pp. 3
-
28 (March 2000). Also reprinted in Human
-
Computer Interaction in the New Millenium, ed. J.M. Carroll, Addison
-
Wesley/ACM Press, Reading, Mass.,
2002, pp. 213
-
233.

W.M. Newman, ''A System for Interactive Graphical Programming,'' Proc. Spring Joint Computer Conference
pp. 47
-
54, AFIPS (1968).

D.R. Olsen, Building Interactive Systems: Principles for Human
-
Computer Interaction, Course Technology Inc.
(2009).

D.R. Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann, San Mateo,
Calif. (1992).

F. Paterno, Model
-
Based Design and Evaluation of Interactive Applications,, Springer
-
Verlag, London (2000).

O. Shaer and R.J.K. Jacob, "A Specification Paradigm for the Design and Implementation of Tangible User
Interfaces," ACM Transactions on Computer
-
Human Interaction (2009)
[http://www.cs.tufts.edu/~jacob/papers/tochi.shaer.pdf].

B. Shneiderman, C. Plaisant, M. Cohen, and S. Jacobs, Designing the User Interface: Strategies for Effective
Human
-
Computer Interaction, Addison
-
Wesley, Reading, Mass. (2009).

E.R. Tufte, ''Visual Design of the User Interface,'' IBM Corporation, Armonk, N.Y. (1989).


5
-
37

T.S. Tullis, ''A Computer
-
Based Tool for Evaluating Alphanumeric Displays,'' Proc. INTERACT '84 Conference,
B. Shackel, ed. (1984).

J. Zigelbaum, M. Horn, O. Shaer, and R.J.K. Jacob, "The Tangible Video Editor: Collaborative Video Editing
with Active Tokens," Proc. TEI 2007 First International Conference on Tangible and Embedded Interaction
pp. 43
-
46 (2007) [http://www.cs.tufts.edu/~jacob/papers/tei07.zigelbaum.pdf].



5
-
38