# the Information Age

Κινητά – Ασύρματες Τεχνολογίες

12 Δεκ 2013 (πριν από 4 χρόνια και 7 μήνες)

100 εμφανίσεις

The “
Assembly Line

for

the Information
Age

Human
-
Computer
Cooperation
for
Large
-
Scale
Product Classification

Jianfu

Chen

Computer
Science Department,

Stony
Brook University

Machines Transform
Human History

People have always been seeking the
optimal way of integrating

machine and human labor.

20
th

Century

Ford Assembly Line

Integrates Machine and Human Labor Efficiently

21
st

Century

Information Age

“Mass Production” of Information

We want to find the optimal ways to
integrate machine and human intelligence.

NOT

automatically by machines

assembly line integrated machine & human labor

NOT

all information can be produced fully
automatically by computers

We want to find optimal ways to integrate
machine and human intelligence

What’s the “
Assembly Line
” for the
Information Age?

A Case Study: Large scale product
classification

Kindle Fire HD
8.9" 4G LTE
Wireless

8.9" HD Display,
Dolby Audio, Dual
-
Band Dual
-
Antenna
Wi
-
Fi, 4G LTE, 32GB
or 64GB

Goal:

optimally integrate computer and human effort

Achieve a lower unit cost for product classification

More precisely, optimize the
accuracy
-
cost

An “Assembly Line” for

Human Computer Cooperation

3Com V.35
cable

V.35
cable ( DTE
)
-

DB
-
50 (M)
-

M/34 (V.35) (M)
-

10 ft

26121609

A list of
K

candidate
classes

System
Accuracy

Machine
Accuracy

Human
Accuracy

X

Cost

is Human labor cost, i.e., the salary paid to
workers, which is proportional to the
working
time
spent.

=

A quick glance at Accuracy
-
Cost
Relation

Assume
K

determines the Accuracy and Cost.

System Accuracy
𝐴
(
𝐾
)
=
𝐴
𝑚
𝐾
×
𝐴

(
𝐾
)

Machine Accuracy
increases

as K increases

𝐴
𝑚
𝐾
=
2
1
+
exp

(

0
.
25

𝐾
)

1

Human
Accuracy
decreases

as K increases

𝐴

𝐾
=
exp

(

0
.
03
𝐾

1
)

Cost increases as K increases

𝐾
=
𝑊𝑇
𝐾
=
𝑊𝐾 𝑜𝑔𝐾

A quick glance at Accuracy
-
Cost Relation

There is an optimal cost that gives the
highest accuracy.

Towards
a more
realistic analysis
of
accuracy
-
cost relationship

With the above “assembly line” model, human
accuracy and working time are influenced by a
set of factors

K

Expertise

I am familiar with
office supplies
, but not familiar with
nuts
and bolts.

Cognitive characteristics

Careful, smart, quick

Use a probabilistic
graphical model
to
capture the
cognitive process of human
classification

A
probabilistic graphical
model shows how the
above different factors interact with each
other, and influence the accuracy and cost.

Specifically, we use Bayesian Network, which
characterizes the
causal

relationships of
different factors.

Use a
Bayesian Network to predict

accuracy and cost

𝑨


𝑨
𝒎

𝑨

𝑻𝒎𝒆

𝒐 

𝒐

𝑴

𝑲

𝑻𝒄





Not only visually intuitive, but also formal

𝑨


𝑨
𝒎

𝑨

𝑻𝒎𝒆

𝒐 

𝒐

𝑴

𝑲

𝑻𝒄





𝑃
𝐴

=
1

,
𝐾
,
𝑀
,
𝑜𝑔
=
1
1
+
exp

(

𝑤
1
𝑇

+
𝑤
2
𝐾
+
𝑤
3
𝑀
+
𝑤
4
𝑜𝑔

)

𝑇𝑖𝑒
=
𝑤
1
𝑇

+
𝑤
2
𝐾
+
𝑤
3
𝑀
+
𝑤
4
𝑜𝑔
+


Inference and learning

with this Bayesian Network, we predict
the
accuracy and cost
by

𝑃
𝐴
,
𝑜 

,
𝐾
,

,
𝑜𝑔

Training data

A set of examples with
known class
labels

We let each human worker work on multiple
examples, record the correctness and the working
time

EM
algorithm
learns
the parameters and the
hidden variables

usage of the model

Predict
the accuracy
-
cost

Given certain
budget
, what’s the highest
accuracy

we can achieve?

To achieve certain
accuracy
, what’s the lowest
expected
cost
?

How to charge customers?

Optimally assign the workers to the tasks

Related Works

time and motion study

Scientific management (
Taylorism
)

Crowdsourcing

Amazon Mechanical Turk

learning worker expertise and accuracy

Item Response Theory

Psychometrics

IQ test, GRE, GMAT

Conclusion

In information age, we need a new “
assembly
line
” to integrate human and machine
intelligence.

We try to model human accuracy and working
time by considering the interactions
of a set of
relevant
factors
, using a
probabilistic

graphical
model
.

We use the model to
predict the
accuracy
-
cost