Crowdsourcing for Business:

Crowdsourcing for Business:

An Emerging Paradigm

Shourya Roy

Area Manager, Human Computation

Xerox Research Centre India, Bangalore

shourya.roy@xerox.com

Workshop on Social
Computing, IIT
Kharagpur

5
th

Oct, 2012

Crowdsourcing :
What is it?


The act of taking a task traditionally performed by an
employee or contractor, and outsourcing it to an
undefined, generally large group of people, in the form of
an open call


Digitization, image labeling, user studies, machine translation
evaluation, logo design, EDA simulation, innovation contests, . .
.

Services Thrust

–

Xerox Confidential
–

Handwriting
Recognition Problem
*


Many tasks are easy/feasible/
doable

for
humans
, but difficult/challenging/impossible
for
computer programs

1.
Make progress


towards deciphering this
handwriting

2.
Put words which you

are unsure about in parenthesis

Instructions to Crowd

Services Thrust

Examples (1/5)

Services Thrust

Examples (2/5)

Services Thrust

Examples (3/5)

Services Thrust

Examples (4/5)

It has been Existing

Humans were the first “computers,” computers, used for math
computations

9 examples of crowdsourcing, before ‘crowdsourcing’ existed :
http://bit.ly/mXFdRp

–

Xerox Confidential
–

Internet and Mobile Have Made
it More Common and Promising

Services Thrust

Increasing Activities and Popularity

Page
10


“2M contributors who does more

than 4PY of work on an average
day!”
--


Increasing Popularity as Depicted
by Google Trends

Crowdsourcing on Google Scholar
Over the Last Few Years

CEO

Services Thrust

Changing Demographics

What is the Problem


Given a computational problem, design a solution
using
human
computers
and
automated
computers

–

Xerox Confidential
–

Human in the loop
(
and not
Guinea Pigs
)

Main doer is Human
(
and not
Machines as in Assembly Lines
)

H
umans
are actively
computing
(
not merely carrier
of sensors
)

The
outcome is determined by
an algorithm
(
and not the
natural dynamics of the crowd
)

Why is it Different?

Where is Research?

–

Xerox Confidential
–


Quality Estimation and Assurance

(
Redundancy and voting; Gold data; joint estimation of worker
quality and task difficulty; Symbiosis with Machine Learning
)


Complex Tasks
(
No discrete answer; Exploration and exploitation; crowd workflows
;)


Task Design
(
Optimize cost, quality and time; infinite completion time; Real time
)


Incentive and Motivation
(
Payment vs. non
-
payment; Optimal payment; Payment and quality
; )


Market Design
(
Reputation Mechanism; Monitoring and feedback; Task Discovery
;


Behavioral Aspects
(
Noisy
behaviour
; Non
-
reproducible
;)

An Emerging Research Field

An Interdisciplinary Research Field

That’s Alright
–

but Xerox!!?



We have transformed…




… into the world’s
leading enterprise
for
Business
Process
and
Document
Management


Revenue

Market
Opportunity

2011

Services
-
led

~50% Services


Document Outsourcing
Business Process
Outsourcing


Information Tech
Outsourcing

~$23 billion

$500 billion +

Services
Leadership In

$15.2 billion

2009

Technology
-
led

~25% Services

$132 billion

Document
Outsourcing

18

Xerox Revenue by Business Segment*

* http://
www.fastcompany.com
/magazine/161/
ursula
-
burns
-
xerox


Page
20


Is Crowdsourcing a Viable Alternative
to Outsourcing?


Outsourcing is


Focus
on the core business while
partnering
with 3rd
party vendors to tackle the

non
-
core operations


Tasks requiring human intelligence and skills


Data and process migration by smart use of
technology


Heavily human intensive; typically with the help of
computing technologies


Large distributed workforce enabled by
technology
executing tiny pieces of work
requiring
human intelligence


Page
21


Data Entry by Crowd


We started by
considering a typical
outsourced process
(Data Entry)


Objective is to
understand a process in
detail and identify
implications for
crowdsourcing


Digitisation of
insurance forms and
medical records for
US based insurance
companies



Typing in,
validation/
correction of
information from
scanned forms


Outsourced,
distributed
process


Features that make
Form Digitization
process
amenable to crowd sourcing

Page
22



Relatively low skill data entry work, known
as ‘key what you see’


Already an outsourced process requiring a
low level of interactivity between
sequential steps


Strong workflow tool to manage work, which
flows through a series of system and human
steps


Between sites


Between sequential tasks


Between agents (given their known skill
set
)

Findings from Work
-
Practice Study (1/2)

Findings from Work
-
Practice Study
(2/2
)

Page
24




Workplace Ecology

: Data security is physical, technical
&
social


Crowdsourcing
: lose physical and social enforcement,
reduced control of workforce. Need technical solutions.


Skills and
Knowledge

1)‘key what you see’ data entry
actually involves extensive rule set. 2) Form difficulty
is situational. 3) Non
-
standard means non
-
standard
.



Crowdsourcing
: Situational
-
based incentives and
supporting learning



Being
a Corporate Employee
Pay
alone not enough to
achieve SLA. Agents made accountable.


Crowdsourcing
: reduced accountability could increase
rejections of difficult work.


Making the Workflow
Work:

Push model of
work


Crowdsourcing
: Pull model of work raises coordination and
completion issues.



Collaborative Working
: Work is not collaborative at
workflow level; but it is at claim level (floorwalkers &
colleagues).


Crowdsourcing
: building collaboration in? Pull models of
supervision?

Conclusion

Page
25



Crowdsourcing is an emerging Research
area


It requires expertise and research
competencies from a number
disciplines


Crowdsourcing can be applied in
various domains to solve problems in
a more effective manner


Finally, a large fraction of the
crowd comes from India



Focused research and technologies
will be highly relevant

Services Thrust

References

•
TurKit
: Tools for Iterative Tasks on Mechanical Turk; Greg Little,
Lydia B. Chilton, Robert C. Miller, and Max
Goldman

•
Matt Lease Tutorial

•
Soylent

–

A
cr

•
Fold.it
–

S. Cooper et. al