the Information Age

fearlessquickΚινητά – Ασύρματες Τεχνολογίες

12 Δεκ 2013 (πριν από 3 χρόνια και 7 μήνες)

74 εμφανίσεις

The “
Assembly Line

for

the Information
Age

Human
-
Computer
Cooperation
for
Large
-
Scale
Product Classification

Jianfu

Chen

Computer
Science Department,

Stony
Brook University


Machines Transform
Human History

People have always been seeking the
optimal way of integrating

machine and human labor.

20
th

Century

Ford Assembly Line

Integrates Machine and Human Labor Efficiently

21
st

Century


Information Age

“Mass Production” of Information

We want to find the optimal ways to
integrate machine and human intelligence.


NOT

all products could be produced fully
automatically by machines


assembly line integrated machine & human labor


NOT

all information can be produced fully
automatically by computers


We want to find optimal ways to integrate
machine and human intelligence


What’s the “
Assembly Line
” for the
Information Age?

A Case Study: Large scale product
classification

Kindle Fire HD
8.9" 4G LTE
Wireless

8.9" HD Display,
Dolby Audio, Dual
-
Band Dual
-
Antenna
Wi
-
Fi, 4G LTE, 32GB
or 64GB

Goal:


optimally integrate computer and human effort


Achieve a lower unit cost for product classification


More precisely, optimize the
accuracy
-
cost

tradeoff

An “Assembly Line” for

Human Computer Cooperation

3Com V.35
cable

V.35
cable ( DTE
)
-

DB
-
50 (M)
-

M/34 (V.35) (M)
-

10 ft

26121609

A list of
K

candidate
classes

System
Accuracy

Machine
Accuracy

Human
Accuracy

X

Cost

is Human labor cost, i.e., the salary paid to
workers, which is proportional to the
working
time
spent.

=

A quick glance at Accuracy
-
Cost
Relation


Assume
K

determines the Accuracy and Cost.


System Accuracy
𝐴
(
𝐾
)
=
𝐴
𝑚
𝐾
×
𝐴

(
𝐾
)


Machine Accuracy
increases

as K increases

𝐴
𝑚
𝐾
=
2
1
+
exp

(

0
.
25

𝐾
)

1


Human
Accuracy
decreases

as K increases

𝐴

𝐾
=
exp

(

0
.
03
𝐾

1
)


Cost increases as K increases


𝐾
=
𝑊𝑇
𝐾
=
𝑊𝐾 𝑜𝑔𝐾


A quick glance at Accuracy
-
Cost Relation

There is an optimal cost that gives the
highest accuracy.

Towards
a more
realistic analysis
of
accuracy
-
cost relationship


With the above “assembly line” model, human
accuracy and working time are influenced by a
set of factors


K


Task difficulty


Expertise


I am familiar with
office supplies
, but not familiar with
nuts
and bolts.


Cognitive characteristics


Careful, smart, quick


Independent of the task


Use a probabilistic
graphical model
to
capture the
cognitive process of human
classification


A
probabilistic graphical
model shows how the
above different factors interact with each
other, and influence the accuracy and cost.


Specifically, we use Bayesian Network, which
characterizes the
causal

relationships of
different factors.

Use a
Bayesian Network to predict

accuracy and cost

𝑨


𝑨
𝒎

𝑨

𝑻𝒎𝒆

𝒐 

𝒐

𝑴

𝑲

𝑻𝒄





Not only visually intuitive, but also formal

𝑨


𝑨
𝒎

𝑨

𝑻𝒎𝒆

𝒐 

𝒐

𝑴

𝑲

𝑻𝒄





𝑃
𝐴

=
1

,
𝐾
,
𝑀
,
𝑜𝑔
=
1
1
+
exp

(

𝑤
1
𝑇

+
𝑤
2
𝐾
+
𝑤
3
𝑀
+
𝑤
4
𝑜𝑔


)

𝑇𝑖𝑒
=
𝑤
1
𝑇

+
𝑤
2
𝐾
+
𝑤
3
𝑀
+
𝑤
4
𝑜𝑔
+


Inference and learning


with this Bayesian Network, we predict
the
accuracy and cost
by

𝑃
𝐴
,
𝑜 

,
𝐾
,

,
𝑜𝑔


Training data


A set of examples with
known class
labels


We let each human worker work on multiple
examples, record the correctness and the working
time


EM
algorithm
learns
the parameters and the
hidden variables


usage of the model


Predict
the accuracy
-
cost
tradeoff


Given certain
budget
, what’s the highest
accuracy

we can achieve?


To achieve certain
accuracy
, what’s the lowest
expected
cost
?


How to charge customers?


Optimally assign the workers to the tasks

Related Works


time and motion study


Scientific management (
Taylorism
)


Crowdsourcing


Amazon Mechanical Turk


learning worker expertise and accuracy


Item Response Theory


Psychometrics


IQ test, GRE, GMAT

Conclusion


In information age, we need a new “
assembly
line
” to integrate human and machine
intelligence.


We try to model human accuracy and working
time by considering the interactions
of a set of
relevant
factors
, using a
probabilistic

graphical
model
.


We use the model to
predict the
accuracy
-
cost

tradeoff
, decide how to charge customers, and
optimally assign tasks to human workers.

Thank you!