Presto:Distributed Machine Learning and
Graph Processing with Sparse Matrices
ShivaramVenkataraman
1
Erik Bodzsar
2
Indrajit Roy Alvin AuYoung Robert S.Schreiber
1
UC Berkeley,
2
University of Chicago,HP Labs
shivaram@cs.berkeley.edu,{erik.bodzsar,indrajitr,alvina,rob.schreiber}@hp.com
Abstract
It is cumbersome to write machine learning and graph al
gorithms in dataparallel models such as MapReduce and
Dryad.We observe that these algorithms are based on matrix
computations and,hence,are inefﬁcient to implement with
the restrictive programming and communication interface of
such frameworks.
In this paper we show that arraybased languages such
as R [3] are suitable for implementing complex algorithms
and can outperform current data parallel solutions.Since R
is singlethreaded and does not scale to large datasets,we
have built Presto,a distributed system that extends R and
addresses many of its limitations.Presto efﬁciently shares
sparse structured data,can leverage multicores,and dynam
ically partitions data to mitigate load imbalance.Our results
showthe promise of this approach:many important machine
learning and graph algorithms can be expressed in a single
framework and are substantially faster than those in Hadoop
and Spark.
1.A matrixbased approach
Many realworld applications require sophisticated analysis
on massive datasets.Most of these applications use machine
learning,graph algorithms,and statistical analyses that are
easily expressed as matrix operations.
For example,PageRank corresponds to the dominant
eigenvector of a matrix G that represents the Web graph.
It can be calculated by starting with an initial vector x
and repeatedly performing x=G∗x until convergence [8].
Similarly,recommendation systems in companies like Net
ﬂix are implemented using matrix decomposition [37].Even
graph algorithms,such as shortest path,centrality measures,
strongly connected components,etc.,can be expressed using
operations on the matrix representation of a graph [19].
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page.To copy otherwise,to republish,to post on servers or to redistribute
to lists,requires prior speciﬁc permission and/or a fee.
Eurosys’13 April 1517,2013,Prague,Czech Republic
Copyright c2013 ACM9781450319942/13/04...$15.00
Arraybased languages such as R and MATLAB provide
an appropriate programming model to express such machine
learning and graph algorithms.The core construct of arrays
makes these languages suitable to represent vectors and ma
trices,and perform matrix computations.R has thousands
of freely available packages and is widely used by data min
ers and statisticians,albeit for problems with relatively small
amounts of data.It has serious limitations when applied to
very large datasets:limited support for distributed process
ing,no strategy for load balancing,no fault tolerance,and is
constrained by a server’s DRAMcapacity.
1.1 Towards an efﬁcient distributed R
We validate our hypothesis that R can be used to efﬁciently
execute machine learning and graph algorithms on large
scale datasets.Speciﬁcally,we tackle the following chal
lenges:
Effective use of multicores.R is singlethreaded.The
easiest way to incorporate parallelism is to execute pro
grams across multiple R processes.Existing solutions for
parallelizing R use message passing techniques,includ
ing network communication,to communicate among pro
cesses [25].This multiprocess approach,also used in com
mercial parallel MATLAB,has two limitations.First,it
makes local copies of many data objects,boosting memory
requirements.Figure 1 shows that two R instances on a sin
gle physical server would have two copies of the same data,
hindering scalability to larger datasets.Second,the network
communication overhead becomes proportional to the num
ber of cores utilized instead of the number of distinct servers,
again limiting scalability.
Existing efforts for parallelizing R have another limita
tion.They do not support pointtopoint communication.In
stead data has to be moved fromworker processes to a desig
nated master process after each phase.Thus,it is inefﬁcient
to execute anything that is not embarrassingly parallel [25].
Even simple iterative algorithms are costly due to the com
munication overhead via the master.
Imbalance in sparse computations.Most realworld
datasets are sparse.For example,the Netﬂix prize dataset
is a matrix with 480K users (rows) and 17K movies (cols)
but only 100 million of the total possible 8 billion ratings
are available.Similarly,very few of the possible edges are
Server1
Server2
Rprocess
copyof
data
Rprocess
data
Rprocess
copyof
data
Rprocess
copyof
data
local copy
network copy
network copy
Figure 1.
R’s poor multicore support:multiple copies of data on
the same server and high communication overhead across servers.
0 20 40 60 80 100
1
5
10
50
100
500
1000
5000
Block id
Block density (normalized)
Netflix
LiveJournal
ClueWeb−1B
Twitter
Figure 2.
Variance in block density.Yaxis shows density of a
block normalized by that of the sparsest block.Lower is better.
present in Web graphs.It is important to store and manipu
late such data as sparse matrices and retain only nonzero en
tries.These datasets also exhibit skew due to the powerlaw
distribution [14],resulting in severe computation and com
munication imbalance when data is partitioned for parallel
execution.Figure 2 illustrates the result of na¨ıve partition
ing of various sparse data sets:LiveJournal (68Medges) [4],
Twitter (2B edges),preprocessed ClueWeb sample
1
(1.2B
edges),and the ratings from Netﬂix prize (100M ratings).
The yaxis represents the block density relative to the spars
est block,when each matrix is partitioned into 100 blocks
having the same number of rows/columns.The plot shows
that a dense block may have 1000× more elements than
a sparse block.Depending upon the algorithm,variance in
block density can have a substantial impact on performance
(Section 7).
1.2 Limitations of current dataparallel approaches
Existing distributed data processing frameworks,such as
MapReduce and DryadLINQ,simplify largescale data pro
cessing [12,17].Unfortunately,the simplicity of the pro
gramming model (as in MapReduce) or reliance on rela
tional algebra (as in DryadLINQ) makes these systems un
suitable for implementing complex algorithms based on ma
trix operations.Current systems either do not support state
ful computations,or do not retain the structure of global
shared data (e.g.,mapping of data to matrices),or do not
allowpoint to point communication (e.g.,restrictive MapRe
duce communication pattern).Such shortcomings in the pro
1
http://lemurproject.org/clueweb09.php
gramming model have led to inefﬁcient implementations of
algorithms or the development of domain speciﬁc systems.
For example,Pregel was created for graph algorithms be
cause MapReduce passes the entire state of the graph be
tween steps [24].
There have been recent efforts to better support large
scale matrix operations.Ricardo [11] and HAMA [30] con
vert matrix operations to MapReduce functions but end up
inheriting the inefﬁciencies of the MapReduce interface.
PowerGraph [14] uses a vertexcentric programming model
(non matrix approach) to implement data mining and graph
algorithms.MadLINQ provides a linear algebra platformon
Dryad but does not efﬁciently handle sparse matrix compu
tations [29].Unlike MadLINQ and PowerGraph,our aim is
to address the issues in scaling R,a system which already
has a large user community.Additionally,our techniques for
handling load imbalance in sparse matrices can be applicable
to existing systems like MadLINQ.
1.3 Our Contribution
We present Presto,an R prototype to efﬁciently process
large,sparse datasets.Presto introduces the distributed array,
darray,as the abstraction to process both dense and sparse
datasets in parallel.Distributed arrays store data across mul
tiple machines.Programmers can execute parallel functions
that communicate with each other and share state using ar
rays,thus making it efﬁcient to express complex algorithms.
Presto programs are executed by a set of worker processes
which are controlled by a master.For efﬁcient multicore
support each worker on a server encapsulates multiple R in
stances that read shared data.To achieve zero copying over
head,we modify R’s memory allocator to directly map data
from the worker into the R objects.This mapping preserves
the metadata in the object headers and ensures that the allo
cation is garbage collection safe.
To mitigate load imbalance,the runtime tracks the exe
cution time and the number of elements in each array parti
tion.In case of imbalance,the runtime dynamically merges
or subdivides array partitions between iterations and assigns
them to a new task,thus varying the parallelism and load in
the system.Dynamic repartitioning is especially helpful for
iterative algorithms where computations are repeated across
iterations.
We have implemented seven different applications in
Presto,ranging from a recommendation system to a graph
centrality measure.Our experience shows that Presto pro
grams are easy to write and can be used to express a wide va
riety of complex algorithms.Compared to published results
of Hadoop and Spark [36],Presto achieves equally good
execution times with only a handful of multicore servers.
For the PageRank algorithm,Presto is more than 40×faster
than Hadoop and 15× faster than Spark.We also show how
Presto’s multicore support reduces communication over
head and ﬁnally measure the impact of dynamic repartition
ing using two realworld applications.
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
A B C D E
1 0 0 0 0
* * * 0 0
A B C D E
* * * * 0
A B C D E
* * * * *
A B C D E
C
E
C
E
C
E
C
E
X
A
1 1 1 0 0
B
0 1 0 1 0
C
0 1 1 0 0
D
0
0
0
1
1
Y= X * G Y
1
= Y * G Y
2
= Y
1
* G
G
D
0
0
0
1
1
E
0 0 0 0 1
Figure 3.
Breadthﬁrst search using matrix operations.The k
th
multiplication uncovers vertices up to k hop distance away.
2.Background
Matrix computation is heavily used in data mining,image
processing,graph analysis,and elsewhere [32].Our focus is
to analyze sparse datasets that are found as web graphs,so
cial networks,product ratings in Amazon,and so on.Many
of these analyses can be expressed using matrix formula
tions that are difﬁcult to write in dataparallel models such
as MapReduce.
Example:graph algorithms.Many common graph algo
rithms can be implemented by operating on the adjacency
matrix [19].To perform breadthﬁrst search (BFS) from a
vertex i we start with a 1 ×N vector x which has all ze
roes except the i
th
element.Using the multiplication y=x∗G
we extract the i
th
row in G,and hence the neighbors of ver
tex i.Multiplying y with G gives vertices two steps away
and so on.Figure 3 illustrates BFS from source vertex A in
a ﬁve vertex graph.After each multiplication step the non
zero entries in Y
i
(starred) correspond to visited vertices.If
we use a sparse matrix representation for G and x,then the
performance of this algorithm is similar to traditional BFS
implementations on sparse graphs.
The BellmanFord singlesource shortest path algorithm
(SSSP) ﬁnds the shortest distance to all vertices from a
source vertex.SSSP can be implemented by starting with
a distance vector d and repeatedly performing a modiﬁed
matrix multiplication,d=d⊗G.In the modiﬁed multipli
cation d(j)=min
k
{d(k)+G(k,j)} instead of the usual
d(j)=
∑
k
{d(k)∗G(k,j)}.In essence,each multiplica
tion step updates the vertex distances by choosing the mini
mum of the current distance,and that of reaching the vertex
using one more edge.
2.1 R:An arraybased environment
R provides an interactive environment to analyze data.It
has interpreted conditional execution (if),loops (for,
while,repeat),and uses array operators written in C,
C++ and FORTRAN for better performance.Line 1 in Fig
ure 4 shows how a 3 ×3 matrix can be created.The argu
ment dim speciﬁes the shape of the matrix and the sequence
10:18 is used to ﬁll the matrix.One can refer to entire subar
rays by omitting an index along a dimension.For example,in
1:> A<array(10:18,dim=c(3,3))#3x3 matrix
2:> A
[,1] [,2] [,3]
[1,] 10 13 16
[2,] 11 14 17
[3,] 12 15 18
3:> A[1,]#First row
[1] 10 13 16
4:> idx<array(1:3,dim=c(3,2))#Index vector
5:> idx
[,1] [,2]
[1,] 1 1
[2,] 2 2
[3,] 3 3
6:> A[idx]#Diagonal of A
[1] 10 14 18
7:> A%
*
%idx#Matrix multiply
[,1] [,2]
[1,] 84 84
[2,] 90 90
[3,] 96 96
Figure 4.
Example array use in R.
line 3 the ﬁrst row of the matrix is obtained by A[1,],where
the column index is left blank to fetch the entire ﬁrst row.
Subsections of a matrix can be easily extracted using index
vectors.Index vectors are an ordered vector of integers.To
extract the diagonal of A we create an index matrix idx in
line 4 whose elements are (1,1),(2,2) and (3,3).In line 6,
A[idx] returns the diagonal elements of A.In a single ma
chine environment,R has native support for matrix multi
plication,linear equation solvers,matrix decomposition and
other operations.For example,%∗ % is an R operator for
matrix multiplication (line 7).
3.Programming model
Presto is R with new language extensions and a runtime to
manage distributed execution.The extensions add data dis
tribution and parallel execution.The runtime takes care of
memory management,scheduling,dynamic data partition
ing,and fault tolerance.As shown in Figure 5,programmers
write a Presto program and submit it to a master process.
The runtime at the master is in charge of the overall execu
tion.It executes the programas distributed tasks across mul
tiple worker processes.Table 1 depicts the Presto language
constructs which we discuss in this section.
3.1 Distributed arrays
Presto solves the problem of structure and scalability by
introducing distributed arrays.Distributed array (darray)
provides a shared,inmemory view of multidimensional
data stored across multiple servers.Distributed arrays have
the following characteristics:
Partitioned.Distributed arrays can be partitioned into con
tiguous ranges of rows,columns or blocks.Users specify
the size of the initial partitions.Presto workers store parti
tions of the distributed array in the compressed sparse col
umn format unless the array is deﬁned as dense.Program
mers use partitions to specify coarsegrained parallelism by
writing functions that execute in parallel and operate on par
Figure 5.
Presto architecture
titions.Partitions can be referred to by the splits func
tion.The splits function automatically fetches remote
partitions and combines them to form a local array.For ex
ample,if splits(A) is an argument to a function exe
cuting on a worker then the whole array A would be re
constructed by the runtime,fromlocal and remote partitions,
and passed to that worker.The i
th
partition can be referenced
by splits(A,i).
Shared.Distributed arrays can be readshared by multiple
concurrent tasks.The user simply passes the array partitions
as arguments to many concurrent tasks.Arrays can be mod
iﬁed inside tasks and the changes are visible globally when
update is called.Presto supports only a single writer per
partition.
Dynamic.Partitions of a distributed array can be loaded
in parallel from data stores such as HBase,Vertica,or from
ﬁles.Once loaded,arrays can be dynamically repartitioned
to reduce load imbalance and prevent straggling.
3.2 Distributed parallelism
Presto provides programmers with a foreach construct to
execute deterministic functions in parallel.The functions do
not return data.Instead,programmers call update inside
the function to publish changes.The Presto runtime starts
tasks on worker nodes for parallel execution of the loop
body.By default,there is a barrier at the end of the loop
to ensure all tasks ﬁnish before statements after the loop are
executed.
3.3 Repartition and invariants
At runtime,programmers can use the repartition com
mand to trigger Presto’s dynamic repartitioning method.
Repartitioning can be used to subdivide an array into a spec
iﬁed number of parts.Repartitioning is an optional perfor
mance optimization which helps when there is load imbal
ance in the system.
One needs be careful while repartitioning structured data,
otherwise program correctness may be affected.For exam
Functionality Description
darray(dim=,blocks=,
sparse=)
Create a distributed array with dimensions
speciﬁed by dim,and partitioned by blocks
of size blocks.
splits(A,i) Return i
th
partition of the distributed array A
or the whole array if i is not speciﬁed.
foreach(v,A,f()) Execute function f as distributed tasks for
each element vof A.Implicit barrier at the end
of the loop.
update(A) Publish the changes to A.
repartition(A,n=,
merge=)
Repartition A into n parts.
invariant(A,B,type=) Declare compatibility between arrays A and
B by rows or columns or both.Used by
the runtime to maintain invariants while re
partitioning.
Table 1.
Main programming language constructs in Presto
ple,when multiplying two matrices,the number of rows
and columns in partitions of both the matrices should con
form.If we repartition only one of the matrices then this in
variant may be violated.Therefore,Presto allows program
mers to optionally specify the array invariants in the pro
gram.We show in Section 5.3 how the runtime can use the
invariant and repartition functions to automati
cally detect and reduce imbalance without any user assis
tance.
Note that for programs with general data structures (e.g.,
trees) writing invariants is difﬁcult.However,for matrix
computation,arrays are the only data structure and the
relevant invariant is the compatibility in array sizes.The
invariantin Presto is similar in spirit to the alignment di
rectives used in High Performance Fortran (HPF [22]).The
HPF directives align elements of multiple arrays to ensure
the arrays are distributed in the same manner.Unlike HPF,
in Presto the invariants are used to maintain correctness dur
ing repartitioning.
4.Applications
We illustrate Presto’s programming model by discussing the
implementation of two algorithms:PageRank and Alternat
ing Least Squares.
PageRank.Figure 6 shows the Presto code for PageRank.
M is the modiﬁed adjacency matrix of the Web graph.PageR
ank is calculated in parallel (lines 7–13) using the power
method [8].In line 1,M is declared as an NxN array.M is
loaded in parallel from the underlying ﬁlesystem using the
Presto driver,and is partitioned by rows.In line 3 the num
ber of columns of M is used to deﬁne the size of a dense
vector pgr which acts as the initial PageRank vector.This
vector is partitioned such that each partition of pgr has the
same number of rows as the corresponding partition of M.
The accompanying illustration points out that each partition
of the vector pgr requires the corresponding (shaded) parti
tions of M,Z,and the whole array xold.The Presto runtime
passes these partitions and reconstructs xold from its par
titions before executing prFunc at each worker.In line 12
#Load data in parallel into adjacency matrix
1:M< darray(dim=c(N,N),blocks=c(s,N),sparse=T)
2:load(M,file="...")
3:pgr< darray(dim=c(ncol(M),1),blocks=c(s,1),sparse=F)
4:xold< darray(dim=c(ncol(M),1),blocks=c(s,1),sparse=F)
5:...
6:invariant(pgr,M,xold,Z,type=ROW)
#Calculate PageRank (pgr)
7:repeat{
#Distributed matrix operations
8:foreach(i,1:numsplits(M),
prFunc(p= splits(pgr,i),m= splits(M,i),
x= splits(xold),z= splits(Z,i)) {
9:p<(m%
*
%x)+ z
10:update(p)
11:})
12:if(norm(pgrxold)<1e9) break
13:xold<pgr
14:}
N
P
1
P
1
s
P
1
s
P
1
P
2
P
1
P
N/s
P
1
P
2
P
N/s
…
s
P
1
P
2
P
N/s
…
s
P
1
P
2
P
N/s
…
N
N/s
M
N/s
xold
Z
N/s
pgr
N/s
Figure 6.
PageRank on a Web graph.
the norm of the two distributed arrays,pgr and xold,are
calculated in parallel.Internally,normis implemented using
foreach.
Line 6 is an example invariant for the PageRank code.
Each of pgr,M,xold,and Z should have the same number
of rows in each partition.By specifying this invariant,the
programmer constrains the runtime to adhere to this compat
ibility between arrays even during automatic repartitioning.
Alternating Least Squares.The Alternating Least Squares
(ALS) algorithm with weighted regularization is known to
perform well for matrix problems that arise in recommen
dation systems,as exempliﬁed by the Netﬂix Prize compe
tition [2].Figure 7 shows the implementation of a parallel
version of the algorithmin Presto [37].In Line 1 we declare
R,a sparse distributed array partitioned by columns.R is a
nu×nm array where nu and nm are the number of users and
number of movies respectively.The ratings are loaded from
the ﬁlesystemin line 2.In line 3 and 4,we create distributed
arrays to hold the feature matrices for users and movies,and
partition the arrays by columns.Each iteration of the ALS al
gorithmconsists of two steps:the ﬁrst step updates the movie
feature matrix (M) based on the existing user feature matrix
(U),while the second step uses M to compute updates for U.
Lines 9–29 show how the ﬁrst step of the algorithm is ex
pressed in Presto.
The foreach loop updates each partition of M in paral
lel.To update a particular partition,we pass the correspond
ing partition of the ratings matrix R and the entire user fea
ture matrix U (lines 9−12).One of the advantages of Presto
is that existing R functions can be used directly.For exam
ple,in line 23 R’s linear solver minimizes the sum of the
squares of the differences between the known and the pre
dicted ratings and computes the new values of M.Similarly,
#Load ratings into a sparse matrix
1:R < darray(dim=c(nu,nm),blocks=c(nu,ms),sparse=T)
2:load(R,file="...")
#Create feature matrices for users and movies
3:M < darray(dim=c(nf,nm),blocks=c(nf,ms),sparse=F)
4:U < darray(dim=c(nf,nu),blocks=c(nf,us),sparse=F)
5:...
6:#Initialize the features
7:...
8:repeat {
#Update movie features based on user ratings
9:foreach(i,1:numsplits(M),
10:function(ms = splits(M,i),
11:us = splits(U),
12:rs = splits(R,i)) {
13:lamI < diag(lambda,nrow=nf,ncol=nf)
#Function to update single movie’s features
15:update
movie < function(movieRatings) {
16:#Get users who have rated this movie
17:users < which(movieRatings!= 0)
18:Um < matrix(nrow=nf,data=us[,users])
19:
20:#Update by minimizing least squares
21:vec < Um %
*
% movieRatings[users]
22:mat < Um %
*
% t(Um) + (length(users)
*
lamI)
23:m
new < solve(mat,vec)
24:return(m
new)
25:}
26:#Calculate updates for movies in this split
27:ms < apply(rs,2,update
movie)
28:update(ms)
29:})
30:#Similarly update user features using movies
31:...
32:#Check for convergence by computing RMSE
33:rmse < compute
rmse(U,M)
34:if (rmse < threshold) { break }
35:}
Figure 7.
ALS algorithmon a Netﬂix ratings dataset
in line 27 the familiar R construct apply is used to invoke
the function update
movie on all the movies present in
the partition.Finally,in line 28 the newvalues of movie fea
tures are made globally visible by invoking update.
5.Systemdesign
The Presto master acts as the control thread.The workers
execute the loop body in parallel whenever foreach loops
are encountered.The master keeps a symbol table which
maps variables to their physical location.This map is used
by workers to exchange information using pairwise commu
nication.In this paper we describe only the main mecha
nisms related to multicore support,and optimizations for
sparsity and caching.
5.1 Versioning arrays
Presto uses versioning to ensure correctness when arrays are
shared across different machines.Each partition of a dis
tributed array has a version.The version of a distributed ar
ray is a concatenation of the versions of its partitions,similar
in spirit to vector clocks.Updates to array partitions create
a new version of the partition.By updating the version,con
current readers of previous versions will not be affected by
the update.For example,the PageRank vector pgr in Fig
ure 6 starts with version 0,0,..,0.In the ﬁrst iteration,ev
ery task reads in one partition pgr with version 0.At the
Worker
DRAM
Shareddata
R
instance
O h
Connections
R
instance
R
instance
O
t
h
er
workers
Connections
Networklayer
Figure 8.
Multiple R instances share data hosted in a worker.
Robject
1
Headercorruption
Data
Header
R instance
R instance
write writecorruption!
R
instance
R
instance
Robject
Data
Header
R instance
R instance
gc()
accesserror!
2
Danglingpointer
R
instance
R
instance
Figure 9.
Simply sharing R objects can lead to errors.
end of the task when update is called,a newpartition with
version 1 is created.Hence,after the ﬁrst iteration the dis
tributed array’s newversion becomes 1,1,..,1.Presto uses
reference counting to garbage collect older versions of ar
rays not used by any task.
5.2 Efﬁcient multicore support
Since R is not thread safe,a simple approach to utilize
multicores is to start multiple worker processes on the same
server.There are three major drawbacks:(1) on the server
multiple copies of the same array will be created,thus in
hibiting scalability,(2) copying the data across processes,
using pipes or network,takes time,and (3) the network com
munication increases as we increase the number of cores be
ing utilized.
Instead,Presto allows a worker to encapsulate multiple
R processes that can communicate through shared mem
ory with zero copying overhead (Figure 8).The key idea
in Presto is to efﬁciently initialize R objects by mapping
data using mmap or shared memory constructs.However,
there are some important safety challenges that need to be
addressed.
Issues with data sharing.Each R object consists of a
ﬁxedsize header,and an array of data immediately follow
ing the header.The header (among other things) has infor
mation about the type and size of the corresponding data
part.Simply pointing an Rvariable to an external data source
leads to data corruption.As shown in Figure 9,if we were to
share an R object across different R instances two problems
can arise.First,both the instances may try to write instance
speciﬁc values to the object header.This conﬂict will lead
to header corruption.Second,R is a garbagecollected lan
guage.If one of the instances garbage collects the object then
the other instance will be left with a dangling pointer.
page
Local R object data part
Local R
object
header
page boundary
page boundary
R object
allocator
glibc malloc
Presto malloc
Local objects
Shared objects
Shared data
R’s virtual memory space
1
2
Allocate object
Map shared data
obj. start address
Figure 10.
Shared object allocation in an R instance.
Safe data sharing.We solve the data sharing challenge by
entrusting each worker with management of data shared by
multiple R processes.We only share readonly data since
only one process may write to a partition during a loop iter
ation and writes always create a new version of a partition.
Presto ﬁrst allocates process local objects in each R instance
and then maps the shared data on the data part of the object.
Since the headers are local to each Rinstance,write conﬂicts
do not occur on the header.
There is another issue that has to be solved:the mmap call
locates data only to an address at a page boundary.However,
R’s internal allocator does not guarantee that the data part
of an object will start at a page boundary.To solve this
issue,Presto overrides the behavior of the internal allocator
of R.We use malloc
hook to intercept R’s malloc() calls.
Whenever we want to allocate a shared R object we use our
custommalloc to return a set of pages rounded to the nearest
multiple of the page size.Once the object has been allocated
the shared data can be mapped using mmap.
Figure 10 shows that R objects are allocated through the
default malloc for local objects and through Presto’s malloc
function for shared objects.The shared objects consist of a
set of pages with the data part aligned to the page bound
ary.The ﬁrst page starts with an unused region because the
header is smaller than a full page.
When the objects are no longer needed,these spe
cially allocated regions need to be unmapped.Presto uses
free
hook to intercept the calls to the glibc free() func
tion.Presto also maintains a list of objects that were spe
cially allocated.The list contains the starting address and al
location size of the shared objects.Whenever free is called,
the runtime checks if the object to be freed is present in the
list.If it is then munmap is called.Otherwise,the glibc free
function is called.Note that while the malloc hook is used
only when allocating shared R objects,the free hook is ac
tive throughout the lifetime of the program,because we do
not know when R may garbage collect objects.
5.3 Dynamic partitioning for sparse data
While shared memory constructs help in reducing the net
work overhead,the overall time taken for a distributed com
putation also depends on the execution time.Partitioning a
sparse matrix into contiguous ranges of rows or columns
may lead to uneven distribution of nonzero elements and
cause a skew in task execution times.Moreover,the num
ber of tasks in the system is tied to the number of partitions
which makes it difﬁcult to effectively use additional work
ers at runtime.Presto uses dynamic partitioning to mitigate
load imbalance,and to increase or decrease the amount of
parallelism in the program at runtime.One can determine
optimal partitions statically to solve load imbalance but it is
an expensive solution.Such partitions may not remain opti
mal as data is updated and static partitioning does not adjust
to changes in the number of workers.
Presto uses two observations to dynamically adjust parti
tions.First,since our target algorithms are iterative,we re
ﬁne the partitions based on the execution of the ﬁrst few it
erations.Second,by knowing the invariants for the program
we can repartition data without affecting correctness.
The Presto runtime tracks both the number of elements in
a partition (e
i
) and the execution time of the tasks (t
i
).It uses
these metrics to decide when to repartition data to reduce
load imbalance.The runtime starts with an initial partition
ing (generally userspeciﬁed),and in subsequent iterations
may either merge or subdivide partitions to create newones.
The aimof dynamic partitioning is to keep the partition sizes
and the execution time of each task close to the median [5].
The runtime tracks the median partition size (e
m
) and task
execution time (t
m
).After each iteration,the runtime checks
if a partition has more (fewer) elements than the median
by a given constant (partition threshold e.g.e
i
/e
m
≥δ) and
subdivides (merges) them.In the PageRank program (Fig
ure 6),after repartitioning the runtime simply invokes the
loop function (pgFunc) for a different number of partitions
and passes the corresponding data.No other changes are re
quired.While our current implementation only uses parti
tion sizes for repartitioning,we plan to explore other metrics
which combine partition sizes and execution times.
For dynamic partitioning,the programmer needs to spec
ify the invariants and annotate functions as safe under repar
titioning.For example,a function that assigns the ﬁrst ele
ment of each partition is unsafe.Such a function is closely
tied to each partition,and if we subdivide an existing par
tition then two cells will be updated instead of one.In our
applications,the only unsafe functions are related to initial
ization such as setting A[i]=1 in breadthﬁrst search.
5.4 Colocation,scheduling,and caching
Presto workers execute functions which generally require
multiple array partitions,including remote ones.Presto uses
three mechanisms to reduce communication:locality based
scheduling,partition colocation,and caching.
The Presto master schedules tasks on workers.The mas
ter uses the symbol table to calculate the amount of re
mote data copy required when assigning a task to a worker.
It then schedules tasks to minimize data movement.Parti
tions that are accessed and modiﬁed in the same function
can be colocated on the same worker.As matrix compu
tations are structured,in most cases colocating different
array partitions simply requires placing the i
th
partition of
the corresponding arrays together.For example,in PageR
ank,the i
th
partition of vectors pgr,M,and Z should be
colocated.Instead of another explicit placement directive,
Presto reuses information provided by the programmer in the
invariant function to determine which arrays are related
and attempts to put the corresponding partitions on same
workers.This strategy of colocation works well for our ap
plications.In the future,we plan to consider workstealing
schedulers [6,28].
Presto automatically caches and reuses arrays whose ver
sions have not changed.For example,in the PageRank code
Z is never modiﬁed.After the ﬁrst iteration,workers always
reuse Z as its version never changes.The runtime simply
keeps the reference to partitions of Z alive and is informed
by the master when a new version is available.Due to auto
matic caching,Presto does not need to provide explicit di
rectives such as broadcast variables [36].
5.5 Fault tolerance
Presto uses primarybackup replication to withstand failures
of the master node.Only the metadata information like the
symbol table,programexecution state,and worker informa
tion is replicated at the backup.The state of the master is
reliably updated at the backup before a statement of the pro
gram is considered complete.R programs are generally a
couple of hundred lines of code,but most lines perform a
compute intensive task.The overhead of checkpointing the
master state after each statement is lowcompared to the time
spent to execute the statement.
We use existing techniques in literature for worker fault
tolerance.The master sends periodic heartbeat messages to
determine the progress of worker nodes.When workers fail
they are restarted and the corresponding functions are re
executed.Like MapReduce and Dryad we assume that tasks
are deterministic,which removes checkpointing as data can
be recreated using task reexecution.The matrix computa
tion focus of Presto simpliﬁes worker faulttolerance.Arrays
undergo coarsegrained transformations and hence it is suf
ﬁcient to just store the transformations reliably instead of
the actual content of the arrays.Therefore,Presto recursively
recreates the corresponding versions of the data after a fail
ure.The information on how to recreate the input is stored
in a table which keeps track of what input data versions and
functions result in speciﬁc output versions.In practice,ar
rays should periodically be made durable for faster recovery.
6.Implementation
Presto is implemented as an R addon package and provides
support for the newlanguage features described in Section 3.
Dense and sparse matrices are stored using R’s Matrix li
Application
Algorithm
R
Presto
Characteristic
LOC
LOC
PageRank
Eigenvector calculation
20
41
Vertex centrality
Graph Algorithm
40
128
Edge centrality
Graph Algorithm
48
132
SSSP
Graph Algorithm
30
62
Netﬂix recom
mender [37]
Matrix decomposition
78
130
Triangle count [18]
Topk eigenvalues
65
121
kMeans clustering
Dense linear algebra
35
71
Input data Size Application
Twitter V=54M,E=2B Triangle counting
TwitterS V=41M,E=1.4B PageRank,Centrality,SSSP
ClueWebS V=100M,E=1.2B PageRank
ClueWeb V=2B,E=6B PageRank
Netﬂix V=480K,E=100M Collaborative Filtering
Table 2.
Presto applications and their input data.
brary.Our current prototype has native support for a limited
set of distributed array operators such as load,save,matrix
multiplication,addition,and so on.Other operators and al
gorithms can be written by programmers using functions in
side foreach.The implementation of both Presto master
and workers use ZeroMQ servers [16].Control messages,
like starting the loop body in a worker or calls to garbage
collect arrays,are serialized and sent using Google’s pro
tocol buffers.Transfers of arrays between workers are im
plemented directly using BSD sockets.The Presto package
contains 800 lines of R code and 10,000 lines of C++ code.
7.Evaluation
Programmers can express various algorithms in Presto that
are difﬁcult or inefﬁcient to implement in current systems.
Table 2 lists seven applications that we implement in Presto.
These applications span graph algorithms,matrix decompo
sition,and dense linear algebra.The sequential version of
each of these algorithms can be written in fewer than 80 lines
in R.In Presto,the distributed versions of the same applica
tions take at most 135 lines.Therefore,only a modest effort
is required to convert these sequential algorithms to run in
Presto.
In this paper we focus on PageRank,vertex central
ity,singlesource shortest path (SSSP),triangle counting,
and collaborative ﬁltering.We compare the performance of
Presto to Spark [36],which is a recent inmemory systemfor
cluster computing,and Hadoopmem,which is Hadoop0.20
but run entirely on ramfs to avoid disk latencies.Spark per
forms inmemory computations,caches data,and is known
to be 20× faster than Hadoop on certain applications.In all
the experiments we disregard the initial time spent in load
ing data from disk.Subsequent references to Hadoop in our
experiments refer to Hadoopmem.
Our evaluation shows that:
•
Presto is the ﬁrst R extension to efﬁciently leverage
multicores by reducing memory and network overheads.
•
Presto can handle load imbalance due to sparsity by dy
namic partitioning.
•
Presto is much faster than current systems.On PageRank
Presto is 40×faster than Hadoop,15×faster than Spark,
and comparable to MPI implementations.
Our experiments use a cluster of 50 HP SL390 servers with
Ubuntu 11.04.Each server has two 2.67GHz (12core) Intel
Xeon X5650 processors,96GB of RAM,120GB SSD,and
the servers are connected with full bisection bandwidth on
a 10Gbps network.Presto,Hadoop,and Spark are run with
the same number of workers or mappers.Hadoop algorithms
are part of Apache Mahout [1].
7.1 Application description
Since we have discussed PageRank and SSSP in Section 2,
we brieﬂy describe centrality measure and triangle counting
algorithms.
Centrality.Vertex or edge betweenness centrality deter
mines the importance of a vertex or edge in a network (e.g.,
social graph) based on the number of shortest paths that in
clude the vertex or edge.We implement Brandes’ algorithm
for unweighted graphs [7].Each betweenness algorithmcon
sists of two phases:ﬁrst the shortest paths from each vertex
to all other vertices are determined (using BFS) and then
these paths are used to update the centrality measure using
scalar transformations.In our experiments we show the re
sults of starting froma vertex whose BFS has 13 levels.
Triangle counting.In large social network graphs,anoma
lous behavior can be detected by counting the number of tri
angles that every vertex belongs to [18].Since a direct count
is expensive for large graphs,the number of triangles is ap
proximated using the top eigenvalues [33].We implement
the iterative Lanzcos algorithm with selective reorthogonal
ization to ﬁnd the topk eigenvalues of a matrix.In each
iteration of the algorithm,the sparse input matrix is mul
tiplied by a dense vector representing the Lanczos vector
from the previous iteration.The result is then orthogonal
ized to form a new basis vector and the eigenvalues for the
input matrix can be computed using the last k orthogonalized
Lanczos vectors.To handle numerical inaccuracies,the al
gorithmuses selectivereorthogonalization,where basis vec
tors are selectively chosen for reorthogonalization.Not every
step in this algorithmneeds to be distributed across the clus
ter,and using Presto,we can effectively mix parallel com
putation with computation on the master.For example,the
matrixvector multiplication is distributed across machines,
but ﬁnding the eigenvalues from the basis vectors is per
formed on the master using existing R functions.
7.2 Advantages of multicore support
With Presto’s multicore support,the memory footprint and
communication overhead are lesser than when using a sin
gle Rinstance per core.In this section we vary the number
of cores and show the time spent during computation,com
posite creation (constructing a distributed array fromits par
titions),and data transfer.We use PrestoNoMC to denote
Pagerank
0
5
10
15
20
25
# cores
Time per iteration (sec)
Composite creation
Transfers
Compute
2 4 6 8
With mc
support
No mc
support
With mc
support
No mc
support
With mc
support
No mc
support
With mc
support
No mc
support
Input data
Vertices
#Cores
Additional mem
ory used (no MC)
TwitterS
41M
8
2.1G
ClueWebS
100M
8
5.3G
Figure 11.
Multicore (MC) support lowers total execution time
and memory usage on a single server.Lower is better.
the system which does not have multicore support and has
single core workers.
Single server:low memory overhead.The ﬁrst advantage
of multicore support is that there is no need to copy data be
tween two R instances that are running on the same server.
Unlike other R packages,Presto can safely share data across
processes through shared memory.Figure 11 shows the av
erage iteration time of PageRank on the 1.5B edge Twitter
graph when executed on a single server.The data transferred
in this algorithm is the PageRank vector.In Presto there is
no transfer overhead as all the R instances are on the same
server and can share data.At 8 cores PrestoNoMC spends
7% of the time in data transfers and takes 5% longer to
complete than Presto.The difference in execution time is
not much as communication over localhost is very ef
ﬁcient even with multiple workers per server.However,the
real win for multicore support in a single server is the re
duction in memory footprint.The table in Figure 11 shows
that at 8 cores the redundant copies of the PageRank vec
tor in PrestoNoMC increase the memory footprint by 2 GB,
which is 10%of the total memory usage.For the CluewebS
dataset PrestoNoMC uses up to 5.3 GB of extra memory.
Multiple servers:low communication overhead.The sec
ond advantage of Presto is that in algorithms with alltoall
communication (broadcast),the amount of data transferred
is proportional only to the number of servers,not the number
of R instances.Figure 12 shows the signiﬁcance of this im
provement for experiments on the TwitterS graph.In these
experiments we ﬁx the number of servers to 5 and vary the
total number of cores.Figure 12(a) shows that the network
transfer overhead for PrestoNoMC is 2.1× to 9.7× higher
than Presto as we vary the total cores from 10 to 40.Worse
still,at 40 cores the PageRank code on PrestoNoMC not
only stops scaling rather it takes more time to complete than
Pagerank
0
2
4
6
8
# cores
Time per iteration (sec)
Composite
creation
Transfers Compute
10 20 40
With mc
support
No mc
support
With mc
support
No mc
support
With mc
support
No mc
support
Vertex centrality
0
100
200
300
# cores
Time (sec)
Composite creation
Transfers
Compute
10 20 40
With mc
support
No mc
support
With mc
support
No mc
support
With mc
support
No mc
support
Figure 12.
Multicore support reduces communication overhead
in (a) PageRank (b) Centrality.Lower is better.
with 20 cores due to higher transfer overhead.In compari
son,Presto can complete an iteration of PageRank in about
3 seconds,though there is only marginal beneﬁt of adding
more than 20 cores for this dataset.Figure 12(b) shows
similar behavior for the centrality measure algorithm.Us
ing Presto the execution time for a single vertex decreases
from 244 seconds at 10 cores to 116 seconds at 40 cores.
In comparison,with no multicore support PrestoNoMC in
curs very high transfer overhead at 40 cores and the execu
tion time is worse by 43%and takes 168 seconds.
7.3 Advantages of dynamic partitioning
While multicore support lowers the memory and communi
cation overhead,dynamic repartitioning of matrices reduces
imbalance due to data sparsity.We evaluate the effective
ness of dynamic partitioning using two algorithms:ﬁrst by
running PageRank on the ClueWeb graph with 2B vertices
and 6B edges and secondly by using the Lanczos method to
ﬁnd topk eigenvalues on the Twitter graph with 54M ver
tices and 2B edges.
7.3.1 PageRank
We ﬁrst look at the PageRank experiments which were run
using 25 servers each with 8 R instances.Even though we
use 200 cores in this experiment,we initially partition the
graph into 1000 parts.This allows the scheduler to intel
ligently overlap computations and attempts to improve the
balance.In this section we show that dynamic repartitioning
improves performance even in such a case.
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
22
Split size (GB)
Iteration count
Figure 13.
We trace the repartitioning seen in the initial four
matrix blocks.Black boxes represents heavy blocks chosen for
repartitioning and gray boxes indicate newly created blocks.
10
15
20
25
30
35
artition Size GB
0
50
100
150
200
250
300
350
400
2000
3000
4000
5000
6000
7000
8000
0 5 10 15 20
Cumulative partitioning time (s)
Time to convergence (s)
Number of Repartitions
Convergence Time
Time spent partitioning
Figure 14.
Convergence time decreases with repartitioning.The
cumulative partitioning time is the time spent in repartitioning.
Effects of repeated partitioning.Figure 13 shows howthe
repartitioning algorithmproceeds on the ClueWeb dataset.A
black colored partition indicates that the particular block was
heavy and chosen for repartitioning.The newly created array
partitions are shown in gray.In Figure 13 the ﬁrst block (also
the densest) is continuously repartitioned fromiteration 1 to
iteration 7 and then again at iterations 11,13,15,and 20.
Overall,repartitioning reduces the size of this partition from
23GB to 2.2GB.
However there is a cost associated with repartitioning.In
our current implementation,PageRank iterations are paused
while the graph is being repartitioned.To quantify the cost
beneﬁt tradeoff,we estimate the total running time of
PageRank as we increase the number of repartitions.Assum
ing that we need to perform 50 iterations for convergence,
Figure 14 shows the estimated time to converge as we vary
the number of repartitions.We calculate the total execution
time after a certain number of repartitions by assuming no
more repartitions will occur.For example,at xaxis value
of 5,Presto has performed ﬁve repartitions and the conver
gence time is 5,126 seconds if no further repartitions occur.
The convergence time reduces by 32%(40 minutes) after the
ﬁrst four repartitions,but the beneﬁts diminish beyond that.
Note that the cumulative time spent in partitioning is a small
fraction of the total execution time (between 0.3%and 3%).
Beneﬁts of reducing imbalance.Reducing the imbalance
among partitions helps decrease the PageRank iteration
time.Figure 15 shows the time taken by each worker dur
0 50 100 150
1
4
7
10
13
16
19
22
25
Time (seconds)
Workers
Fetch
Execute
0 50 100
150
1
4
7
10
13
16
19
22
25
Time (seconds)
Workers
Fetch
Execute
Figure 15.
Per worker execution time for PageRank (a) before
repartitioning (b) after four repartitions.Shorter bar is better.
0
500
1000
1500
2000
2500
3000
3500
0 5 10 15
20
Time (seconds)
Iteration Count
No Repart
i
t
i
on
i
ng
With Repartitioning
Figure 16.
Comparison of overall execution time with and with
out repartitioning.Lower is better.
ing one iteration of PageRank.The horizontal bars depict
what part of the total time was spent in transferring data ver
sus the time taken to perform the computation.Since there
is a barrier at the end of an iteration,the iteration time is de
termined by the maximum execution time among the work
ers.Figure 15(a) shows that the slowest worker takes 147
seconds initially but after four repartitions (Figure 15(b)) it
ﬁnishes in 95 seconds thus reducing the periteration time.
Reducing imbalance is especially important for iterative
algorithms as the overall execution time can be signiﬁcantly
high due to the skew among workers.As seen in Figure 16,
repartitioning reduces the completion time by around 822
seconds (13.7 minutes) when the PageRank algorithmis run
for 20 iterations.
7.3.2 Lanczos calculation
We ran the Lanczos algorithm on the Twitter graph using
20 servers.As the dataset was relatively smaller (2B edges),
we divide the graph into 20 partitions.Each server uses as
many R instances as the number of partitions present on the
server.Similar to the PageRank experiments,we study the
beneﬁts of dynamic partitioning by looking at the imbalance
among different workers while executing a single iteration of
0 5 10 15
20
1
4
7
10
13
16
19
Time (seconds)
Workers
Fetch
Execute
0 5 10 15
20
1
4
7
10
13
16
19
Time (seconds)
Workers
Fetch
Execute
Figure 17.
Per worker execution time for Lanczos algorithm on
Twitter dataset (a) before repartitioning (b) after eight repartitions.
Shorter bar is better.
the Lanczos algorithm.The time taken in a single iteration
of the Lanczos algorithmis dominated by the sparse matrix
dense vector multiplication.The execution time for this step
grows linearly as the number of nonzero entries in the
matrix increase.Figure 17 shows the time taken by each
worker during one iteration with and without repartitioning.
We observe that the Twitter dataset contains one partition
which is much larger than the others,and that repartitioning
reduces the per iteration execution time from19s to 7s.
7.4 Scalability
We evaluate the scalability of Presto using two algorithms:
collaborative ﬁltering [29] (CF) and singlesource shortest
path (SSSP).In the following set of experiments we use 8
cores per server and measure the time taken as we increase
the number of cores.In our ﬁrst experiment,we load the Net
ﬂix ratings dataset [2] as a matrix (R) and run two steps of
a collaborative ﬁltering algorithm.The ﬁrst step computes
R
t
×R,and the second step multiplies R with the output of
the ﬁrst step computing (R×R
t
) ×R.Figure 18 shows the
time taken by each step of the algorithm as we increase the
number of cores.Presto scales quite well in this case with a
speedup of 4.89×(755s to 154s) while using 6×more cores
(8 to 48).These numbers also indicate that Presto’s perfor
mance is competitive to published results for MadLINQ[29]
which takes 840s on 48 machines for the same algorithm.As
the performance numbers for MadLINQare froma different
hardware conﬁguration,instead of direct comparison,our re
sults only indicate that Presto can match the performance
of existing matrixbased systems.We also tried to compare
Presto’s performance to a vanillaR implementation of col
laborative ﬁltering.While vanillaR took 385s to perform
R
t
×R,the second multiplication (R×R
t
×R) failed to com
plete as the intermediate data did not ﬁt in the server’s 96GB
Netflix Collaborative Filtering
0
200
400
600
800
1000
# cores
Total time (sec)
Load t(R)xR Rxt(R)xR
8 16 24 32 40 48
Figure 18.
Running time for collaborative ﬁltering on the Netﬂix
dataset as we increase the number of cores.
Twitter
0
20
40
60
80
100
120
140
# cores
Total time (sec)
Composite creation
Transfers
Compute
16 32 48 64 80 96 112 128
Figure 19.
SSSP scalability on the TwitterS dataset.
memory.This example highlights the need for distributing
the computation across more servers.
Figure 19 uses SSSP on the 1.5B edge TwitterS dataset
to showthe performance scaling of Presto.While Presto can
scale to hundreds of cores,and the execution time continues
to decrease,in this case the scaling factor is less than the
ideal.For example,when increasing the cores from 16 to
128 (8×),the execution time drops from 125 seconds to
41 seconds (3×).The less than ideal scaling is a result of
the communication overhead involved in SSSP,which is
proportional to the number of vertices in the graph.In future
we plan to rewrite the SSSP algorithmto use block partitions
of the matrix (instead of row partitions) so that no single R
instance requires the full shortest path vector.
7.5 Comparison with MPI,Spark,and Hadoop
PageRank experiments on the 1.2B edge ClueWebS graph
shows that Presto is more than 40×faster than Hadoop,more
than 15×faster than Spark,and can outperformsimple MPI
implementations.
MPI.We implemented PageRank using sparse matrix and
vector multiplication in MPI.The communication phase in
the code uses MPI
Allgather to gather the partitions of
the PageRank vector from processes and distribute it to all.
Figure 20(a) shows that Presto outperforms the MPI code
MPI Pagerank
0
2
4
6
8
10
12
14
# cores
Time per iteration (sec)
Transfers Compute
8 16 32 64
Presto
MPI
Presto
MPI
Presto
MPI
Presto
MPI
Spark PageRank
1
2
5
10
20
50
100
200
500
# cores
Time per iteration (sec)
Transfers Compute
8 16 32 64
Presto
Presto
Presto
Presto
Spark
Spark
Spark
Spark
Hadoop PageRank
1
2
5
10
20
50
100
200
500
# cores
Time per iteration (sec)
8 16 32 64
Presto
Presto
Presto
Presto
Hadoop−mem
Hadoop−mem
Hadoop−mem
Hadoop−mem
Figure 20.
Performance advantage over (a) MPI (b) Spark and (c) Hadoop.Lower is better.
sometimes by 2×.There are two reasons for this perfor
mance difference.First,the MPI code does not handle com
pute imbalance.For example,at 64 cores one MPI process
ﬁnishes in just 0.6 seconds while another process takes 4.4
seconds.Since processes wait for each other before the next
iteration,the compute time is determined by the slowest pro
cess.Second,while MPI’s network overhead is very low at
8 processes,it increases with the increase in the number of
cores.However,for Presto the network overhead is propor
tional to the number of multicore servers used,and hence
does not increase at the same rate.With more effort one can
implement multithreaded programs executing at each MPI
process.Such an implementation will reduce the network
overhead but not the compute imbalance.
Spark.We use Spark’s PageRank implementation [36] to
compare its performance with Presto.Spark takes about
64.185 seconds periteration with 64 cores.The periteration
time includes a map phase which computes the rank of ver
tices and then propagates themto reducers that sumthe val
ues.We found that the ﬁrst phase was mostly compute inten
sive and took around 44.3 seconds while the second phase
involved shufﬂing data across the network and took 19.77
seconds.At fewer cores,the compute time is as high as
267.26 seconds with 8 cores.The main reason why Spark is
at least 15×slower than Presto is because it generates a large
amount of intermediate data and hence spends more time
than Presto during execution and network transfers.Note
that the Yaxis in the plot is log scale.
Hadoop.Figure 20(c) compares the performance of Ma
hout’s PageRank implementation to that of Presto.Since
mappers and reducers overlap during the Hadoop compu
tation,we depict only the overall execution time.Each it
eration of Mahout’s PageRank takes 161 seconds with 64
mappers.In comparison each iteration of PageRank in Presto
takes less than 4 seconds.Aportion of the 40×performance
difference is due to the use of Java.However unlike Presto,
MapReduce has the additional overhead of the sort phase
and the time spent in deserialization.Presto preserves the
matrix structure in between operations,and also eliminates
the need to sort data between iterations.
Existing R packages.To obtain a baseline for R
implementations,we measured the time taken for a single
PageRank iteration using vanillaR.R takes 30 seconds per
iteration in our setup and was faster than Presto which takes
58 seconds when using a singlecore.We found that Presto
was slower due to the overheads associated with mapping
and processing 128 partitions.When the dataset was merged
to forma single partition (similar to vanillaR case) Presto’s
performance matches that of vanillaR.However,partition
ing the dataset is helpful when using multiple cores.Presto
running on 8 cores takes less than 10 seconds for each
PageRank iteration.
Unfortunately,existing parallel R packages only allow
sideeffect free functions to be executed in parallel.It means
that R objects in workers are deleted across iterations.Thus,
to run more than one iteration of parallel PageRank the
whole graph needs to be reloaded in the next iteration mak
ing the measurements ﬂawed.Instead,we ran a microbench
mark with 8 cores where the sparse matrices were not ex
changed and only a dense vector of 100M entries was ex
changed after each round (similar to the PageRank vector).
By efﬁciently using multicores and workerworker commu
nication Presto is more than 4×faster than doMC,a parallel
R package.
8.Discussion
Presto makes it easy for users to algorithmically explore
large datasets.It is a step towards a platform on which
high level libraries can be implemented.We believe that
Presto packages that implement scalable machine learning
and graph algorithms will help the large R user base reap the
beneﬁts of distributed computing.
However,certain challenges remain both in the current
prototype and in the applicability of R to all problems.First,
the current prototype is limited by main memory:datasets
need to ﬁt the aggregate memory of the cluster.While most
preprocessed graphs are in the low terabyte size range,for
larger datasets it may be economical to use an outofcore
system.We are working on adding outofcore support for
distributed arrays in future versions of Presto.
Second,Presto assumes that there is one writer per parti
tion during a single foreach execution.Instead of using locks
to synchronize concurrent accesses,in Presto multiple tasks
explicitly write to their partitions and then combine or re
duce the data in another foreach loop.For example,in
kmeans the centers are calculated and stored in separate ar
rays by each task and then summed up in another loop.This
programming model retains the simplicity of R and we have
found it sufﬁcient for all the algorithms implemented so far.
This model may not be appropriate for implementing irreg
ular applications like Delaunay mesh reﬁnement that require
ﬁne grained synchronization [20].
When applied to different datasets,arraybased program
ming may require additional preprocessing.For example,
Presto is based on R and is very efﬁcient at processing ar
rays.However,graphs may have attributes attached to each
vertex.An algorithmwhich uses these attributes (e.g.,search
shortest path with attribute pattern) may incur the additional
overhead of referencing attributes stored in R vectors sepa
rate from the adjacency matrix.In general,real world data
is semistructured and preprocessing may be required to
extract relevant ﬁelds and convert them into arrays.Unlike
the Hadoop ecosystem which has both storage (HDFS) and
computation (MapReduce),Presto only has a efﬁcient com
putation layer.In our experience,it’s easier to load data into
Presto if the underlying store has tables (databases,HBase,
etc.) and supports extraction mechanisms (e.g.,SQL).
9.Related Work
Dataﬂow models.MapReduce and Dryad are popular
dataﬂow systems for parallel data processing [12,17].To
increase programmer productivity highlevel programming
models–DryadLINQ [35] and Pig [27]—are used on top of
MapReduce and Dryad.These systems scale to hundreds
of machines.However,they are best suited for batch pro
cessing,and because of their restrictive programming and
communication interface make it difﬁcult to implement ma
trix operations.Recent improvements,such as HaLoop [9],
Twister [13],and Spark [36],do not change the program
ming model but improve iterative performance by caching
data or using lineage for efﬁcient fault tolerance.CIEL in
creases the expressibility of programs by allowing newdata
dependent tasks during job execution [26].However,none
of these systems can efﬁciently express matrix operations.
Piccolo runs parallel applications that can share state us
ing distributed,inmemory,keyvalue tables [28].Compared
to MapReduce,Piccolo is better suited for expressing ma
trix operations.However,Piccolo’s keyvalue interface opti
mizes for lowlevel reads and writes to keys instead of struc
tured vector processing.Unlike Presto,Piccolo does not han
dle sparse datasets and the resulting load imbalance.
Pregel and GraphLab support bulk synchronous process
ing (BSP [34]) to execute parallel programs [23,24].With
BSP,each vertex processes its local data and communicates
with other vertices using messages.Both systems require an
application to be (re)written in the BSP model.Presto shows
that the widely used R system can be extended to give sim
ilar performance without requiring any programming model
changes.Presto’s execution time of PageRank on the Twit
ter graph (Figure 11,8 cores,7.3s) compares favorably to
published results of PowerGraph (512 cores,3.6s) [14].
Matrix computations.Ricardo [11] and HAMA [30] use
MapReduce to implement matrix operations.While they
solve the problem of scaling to large datasets,the imple
mentation is inefﬁcient due to the restrictive MapReduce in
terface.In light of this observation,MadLINQ provides a
platformon Dryad speciﬁcally for matrix computations [29].
Similar to Presto,MadLINQ reuses existing matrix libraries
on local partitions,is fault tolerant and distributed.While
MadLINQ’s techniques are efﬁcient for dense matrices,their
systemdoes not efﬁciently handle sparse datasets,or support
dynamic partitioning to overcome load imbalance.
Popular highperformance computing (HPC) systems like
ScaLAPACK do not support general sparse matrices.The
few systems that do support sparse matrices (SLEPc [15],
ARPACK[21]) typically provide only eigensolvers.To write
a new algorithm,such as the betweenness centrality,one
would have to implement it with their low level interfaces
including FORTRANcode.None of these systems have load
balancing techniques or fault tolerance.MATLAB’s parallel
computing toolbox and existing efforts in parallelizing Rcan
run single programs on multiple data.Unlike these systems,
Presto can safely share data across multiple processes,has
fewer redundant copies of data,and can mitigate load imbal
ance due to sparse datasets.
Parallel languages.HPC applications use explicit mes
sage passing models like MPI.MPI programmers have the
ﬂexibility to optimize the messaging layer but are difﬁcult
to write and maintain.Newparallel programming languages
like X10 [10] and Fortress [31] use the partitioned global
address space model (PGAS).These languages are not op
timized for matrix operations and the programmer has to
deal with low level primitives like synchronization and ex
plicit locations.For example,in X10 programmers specify
on what processors computations should occur using Place.
None of these languages are as popular as R,and users will
have to rewrite hundreds of statistical algorithms that are al
ready present in R.
10.Conclusion
Presto advocates the use of sparse matrix operations to sim
plify the implementation of machine learning and graph al
gorithms in a cluster.Presto uses distributed arrays for struc
tured processing,efﬁciently uses multicores,and dynami
cally partitions data to reduce load imbalance.Our experi
ence shows that Presto is a ﬂexible computation model that
can be used to implement a variety of complex algorithms.
Acknowledgments:We thank the anonymous reviewers
and our shepherd,JeanPhilippe Martin,for their valuable
feedback.Aurojit Panda and Evan Sparks suggested im
provements to earlier drafts of this paper.Finally,we thank
John Byrne,Kyungyong Lee,Partha Ranganathan,and Van
ish Talwar for assisting us in developing Presto.
References
[1] Apache mahout.http://mahout.apache.org.
[2] Netﬂix prize.http://www.netflixprize.com/.
[3] The R project for statistical computing.http://www.r
project.org.
[4] Stanford network analysis package.http://snap.
stanford.edu/snap.
[5] G.Ananthanarayanan,S.Kandula,A.Greenberg,I.Stoica,
Y.Lu,B.Saha,and E.Harris.Reining in the outliers in map
reduce clusters using Mantri.In In OSDI’10,Vancouver,BC,
Canada,2010.
[6] R.D.Blumofe and C.E.Leiserson.Scheduling multithreaded
computations by work stealing.In SFCS ’94,pages 356–368,
Washington,DC,USA,1994.
[7] U.Brandes.A faster algorithm for betweenness centrality.
Journal of Mathematical Sociology,25:163–177,2001.
[8] S.Brin and L.Page.The anatomy of a largescale hypertextual
Web search engine.In WWW7,pages 107–117,1998.
[9] Y.Bu,B.Howe,M.Balazinska,and M.D.Ernst.HaLoop:
Efﬁcient iterative data processing on large clusters.Proc.
VLDB Endow.,3:285–296,September 2010.
[10] P.Charles,C.Grothoff,V.Saraswat,C.Donawa,A.Kielstra,
K.Ebcioglu,C.von Praun,and V.Sarkar.X10:An object
oriented approach to nonuniformcluster computing.In OOP
SLA’05,pages 519–538,2005.
[11] S.Das,Y.Sismanis,K.S.Beyer,R.Gemulla,P.J.Haas,
and J.McPherson.Ricardo:Integrating R and Hadoop.In
SIGMOD Conference’10,pages 987–998,2010.
[12] J.Dean and S.Ghemawat.MapReduce:Simpliﬁed data pro
cessing on large clusters.Commun.ACM,51(1),2008.
[13] J.Ekanayake,H.Li,B.Zhang,T.Gunarathne,S.H.Bae,
J.Qiu,and G.Fox.Twister:A runtime for iterative MapRe
duce.In HPDC ’10,pages 810–818,2010.
[14] J.E.Gonzalez,Y.Low,H.Gu,D.Bickson,and C.Guestrin.
PowerGraph:Distributed GraphParallel Computation on Nat
ural Graphs.In OSDI’12,Hollywood,CA,October 2012.
[15] V.Hernandez,J.E.Roman,and V.Vidal.Slepc:A scalable
and ﬂexible toolkit for the solution of eigenvalue problems.
ACMTrans.Math.Softw.,31(3):351–362,Sept.2005.
[16] P.Hintjens.ZeroMQ:The Guide,2010.
[17] M.Isard,M.Budiu,Y.Yu,A.Birrell,and D.Fetterly.Dryad:
Distributed dataparallel programs from sequential building
blocks.In EuroSys ’07,pages 59–72,2007.
[18] U.Kang,B.Meeder,and C.Faloutsos.Spectral Analysis
for BillionScale Graphs:Discoveries and Implementation.In
PAKDD (2),pages 13–25,2011.
[19] J.Kepner and J.Gilbert.Graph Algorithms in the Language of
Linear Algebra.Fundamentals of Algorithms.SIAM,2011.
[20] M.Kulkarni,K.Pingali,B.Walter,G.Ramanarayanan,
K.Bala,and L.P.Chew.Optimistic parallelism requires ab
stractions.In PLDI ’07,pages 211–222.
[21] R.B.Lehoucq,D.C.Sorensen,and C.Yang.ARPACK users’
guide  solution of largescale eigenvalue problems with im
plicitly restarted Arnoldi methods.Software,environments,
tools.SIAM,1998.
[22] D.Loveman.High performance Fortran.IEEE Parallel &
Distributed Technology:Systems &Applications,1(1):25–42,
1993.
[23] Y.Low,J.Gonzalez,A.Kyrola,D.Bickson,C.Guestrin,and
J.M.Hellerstein.GraphLab:A New Framework for Parallel
Machine Learning.CoRR,pages 1–1,2010.
[24] G.Malewicz,M.H.Austern,A.J.Bik,J.C.Dehnert,I.Horn,
N.Leiser,and G.Czajkowski.Pregel:Asystemfor largescale
graph processing.In SIGMOD ’10,pages 135–146,2010.
[25] Q.E.McCallum and S.Weston.Parallel R.O’Reilly Media,
Oct.2011.
[26] D.G.Murray and S.Hand.C
IEL
:A universal execution
engine for distributed dataﬂow computing.In NSDI ’11,
Boston,MA,USA,2011.
[27] C.Olston,B.Reed,U.Srivastava,R.Kumar,and A.Tomkins.
Pig latin:A notsoforeign language for data processing.In
SIGMOD’08,pages 1099–1110,2008.
[28] R.Power and J.Li.Piccolo:Building fast,distributed pro
grams with partitioned tables.In OSDI ’10,Vancouver,BC,
Canada,2010.USENIX Association.
[29] Z.Qian,X.Chen,N.Kang,M.Chen,Y.Yu,T.Moscibroda,
and Z.Zhang.MadLINQ:largescale distributed matrix com
putation for the cloud.In EuroSys ’12,pages 197–210,2012.
[30] S.Seo,E.J.Yoon,J.Kim,S.Jin,J.S.Kim,and S.Maeng.
Hama:An efﬁcient matrix computation with the mapreduce
framework.In In CLOUDCOM’10,pages 721–726.
[31] G.L.Steele,Jr.Parallel programming and code selection in
fortress.In PPoPP ’06,pages 1–1,2006.
[32] G.Strang.Introduction to Linear Algebra,Third Edition.
Wellesley Cambridge Pr,Mar.2003.
[33] C.E.Tsourakakis.Fast counting of triangles in large real net
works without counting:Algorithms and laws.In ICDM’08,
pages 608–617.IEEE,2008.
[34] L.G.Valiant.A bridging model for parallel computation.
Commun.ACM,33:103–111,August 1990.
[35] Y.Yu,M.Isard,D.Fetterly,M.Budiu,U.Erlingsson,P.K.
Gunda,and J.Currey.DryadLINQ:A system for general
purpose distributed dataparallel computing using a highlevel
language.In OSDI ’08,pages 1–14,2008.
[36] M.Zaharia,M.Chowdhury,T.Das,A.Dave,J.Ma,M.Mc
Cauley,M.J.Franklin,S.Shenker,and I.Stoica.Resilient
distributed datasets:a faulttolerant abstraction for inmemory
cluster computing.In NSDI’12,San Jose,CA,2012.
[37] Y.Zhou,D.Wilkinson,R.Schreiber,and R.Pan.LargeScale
Parallel Collaborative Filtering for the Netﬂix Prize.In AAIM
’08,pages 337–348,Shanghai,China,2008.
Enter the password to open this PDF file:
File name:

File size:

Title:

Author:

Subject:

Keywords:

Creation Date:

Modification Date:

Creator:

PDF Producer:

PDF Version:

Page Count:

Preparing document for printing…
0%
Σχόλια 0
Συνδεθείτε για να κοινοποιήσετε σχόλιο