Crowdsourcing for Business:

wonderfuldistinctAI and Robotics

Oct 16, 2013 (4 years and 23 days ago)

104 views

Crowdsourcing for Business:

An Emerging Paradigm

Shourya Roy

Area Manager, Human Computation

Xerox Research Centre India, Bangalore

shourya.roy@xerox.com

Workshop on Social
Computing, IIT
Kharagpur

5
th

Oct, 2012

Crowdsourcing :
What is it?


The act of taking a task traditionally performed by an
employee or contractor, and outsourcing it to an
undefined, generally large group of people, in the form of
an open call


Digitization, image labeling, user studies, machine translation
evaluation, logo design, EDA simulation, innovation contests, . .
.

Services Thrust



Xerox Confidential


Handwriting
Recognition Problem
*


Many tasks are easy/feasible/
doable

for
humans
, but difficult/challenging/impossible
for
computer programs

1.
Make progress


towards deciphering this
handwriting

2.
Put words which you

are unsure about in parenthesis

Instructions to Crowd

Services Thrust

Examples (1/5)

Services Thrust

Examples (2/5)

Services Thrust

Examples (3/5)

Services Thrust

Examples (4/5)

It has been Existing

Humans were the first “computers,” computers, used for math
computations

9 examples of crowdsourcing, before ‘crowdsourcing’ existed :
http://bit.ly/mXFdRp



Xerox Confidential


Internet and Mobile Have Made
it More Common and Promising

Services Thrust

Increasing Activities and Popularity

Page
10


“2M contributors who does more

than 4PY of work on an average
day!”
--


Increasing Popularity as Depicted
by Google Trends

Crowdsourcing on Google Scholar
Over the Last Few Years

CEO

Services Thrust

Changing Demographics

What is the Problem


Given a computational problem, design a solution
using
human
computers
and
automated
computers



Xerox Confidential


Human in the loop
(
and not
Guinea Pigs
)

Main doer is Human
(
and not
Machines as in Assembly Lines
)

H
umans
are actively
computing
(
not merely carrier
of sensors
)

The
outcome is determined by
an algorithm
(
and not the
natural dynamics of the crowd
)

Why is it Different?

Where is Research?



Xerox Confidential



Quality Estimation and Assurance

(
Redundancy and voting; Gold data; joint estimation of worker
quality and task difficulty; Symbiosis with Machine Learning
)


Complex Tasks
(
No discrete answer; Exploration and exploitation; crowd workflows
;)


Task Design
(
Optimize cost, quality and time; infinite completion time; Real time
)


Incentive and Motivation
(
Payment vs. non
-
payment; Optimal payment; Payment and quality
; )


Market Design
(
Reputation Mechanism; Monitoring and feedback; Task Discovery
;


Behavioral Aspects
(
Noisy
behaviour
; Non
-
reproducible
;)

An Emerging Research Field

An Interdisciplinary Research Field

That’s Alright


but Xerox!!?



We have transformed…




… into the world’s
leading enterprise
for
Business
Process
and
Document
Management


Revenue

Market
Opportunity

2011

Services
-
led

~50% Services


Document Outsourcing
Business Process
Outsourcing


Information Tech
Outsourcing

~$23 billion

$500 billion +

Services
Leadership In

$15.2 billion

2009

Technology
-
led

~25% Services

$132 billion

Document
Outsourcing

18

Xerox Revenue by Business Segment*

* http://
www.fastcompany.com
/magazine/161/
ursula
-
burns
-
xerox


Page
20


Is Crowdsourcing a Viable Alternative
to Outsourcing?


Outsourcing is


Focus
on the core business while
partnering
with 3rd
party vendors to tackle the

non
-
core operations


Tasks requiring human intelligence and skills


Data and process migration by smart use of
technology


Heavily human intensive; typically with the help of
computing technologies


Large distributed workforce enabled by
technology
executing tiny pieces of work
requiring
human intelligence


Page
21


Data Entry by Crowd


We started by
considering a typical
outsourced process
(Data Entry)


Objective is to
understand a process in
detail and identify
implications for
crowdsourcing


Digitisation of
insurance forms and
medical records for
US based insurance
companies



Typing in,
validation/
correction of
information from
scanned forms


Outsourced,
distributed
process


Features that make
Form Digitization
process
amenable to crowd sourcing

Page
22



Relatively low skill data entry work, known
as ‘key what you see’


Already an outsourced process requiring a
low level of interactivity between
sequential steps


Strong workflow tool to manage work, which
flows through a series of system and human
steps


Between sites


Between sequential tasks


Between agents (given their known skill
set
)

Findings from Work
-
Practice Study (1/2)

Findings from Work
-
Practice Study
(2/2
)

Page
24




Workplace Ecology

: Data security is physical, technical
&
social


Crowdsourcing
: lose physical and social enforcement,
reduced control of workforce. Need technical solutions.


Skills and
Knowledge

1)‘key what you see’ data entry
actually involves extensive rule set. 2) Form difficulty
is situational. 3) Non
-
standard means non
-
standard
.



Crowdsourcing
: Situational
-
based incentives and
supporting learning



Being
a Corporate Employee
Pay
alone not enough to
achieve SLA. Agents made accountable.


Crowdsourcing
: reduced accountability could increase
rejections of difficult work.


Making the Workflow
Work:

Push model of
work


Crowdsourcing
: Pull model of work raises coordination and
completion issues.



Collaborative Working
: Work is not collaborative at
workflow level; but it is at claim level (floorwalkers &
colleagues).


Crowdsourcing
: building collaboration in? Pull models of
supervision?

Conclusion

Page
25



Crowdsourcing is an emerging Research
area


It requires expertise and research
competencies from a number
disciplines


Crowdsourcing can be applied in
various domains to solve problems in
a more effective manner


Finally, a large fraction of the
crowd comes from India



Focused research and technologies
will be highly relevant

Services Thrust

References


TurKit
: Tools for Iterative Tasks on Mechanical Turk; Greg Little,
Lydia B. Chilton, Robert C. Miller, and Max
Goldman


Matt Lease Tutorial


Soylent



A
cr


Fold.it


S. Cooper et. al