Presto:Distributed Machine Learning and

Graph Processing with Sparse Matrices

ShivaramVenkataraman

1

Erik Bodzsar

2

Indrajit Roy Alvin AuYoung Robert S.Schreiber

1

UC Berkeley,

2

University of Chicago,HP Labs

shivaram@cs.berkeley.edu,{erik.bodzsar,indrajitr,alvina,rob.schreiber}@hp.com

Abstract

It is cumbersome to write machine learning and graph al-

gorithms in data-parallel models such as MapReduce and

Dryad.We observe that these algorithms are based on matrix

computations and,hence,are inefﬁcient to implement with

the restrictive programming and communication interface of

such frameworks.

In this paper we show that array-based languages such

as R [3] are suitable for implementing complex algorithms

and can outperform current data parallel solutions.Since R

is single-threaded and does not scale to large datasets,we

have built Presto,a distributed system that extends R and

addresses many of its limitations.Presto efﬁciently shares

sparse structured data,can leverage multi-cores,and dynam-

ically partitions data to mitigate load imbalance.Our results

showthe promise of this approach:many important machine

learning and graph algorithms can be expressed in a single

framework and are substantially faster than those in Hadoop

and Spark.

1.A matrix-based approach

Many real-world applications require sophisticated analysis

on massive datasets.Most of these applications use machine

learning,graph algorithms,and statistical analyses that are

easily expressed as matrix operations.

For example,PageRank corresponds to the dominant

eigenvector of a matrix G that represents the Web graph.

It can be calculated by starting with an initial vector x

and repeatedly performing x=G∗x until convergence [8].

Similarly,recommendation systems in companies like Net-

ﬂix are implemented using matrix decomposition [37].Even

graph algorithms,such as shortest path,centrality measures,

strongly connected components,etc.,can be expressed using

operations on the matrix representation of a graph [19].

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for proﬁt or commercial advantage and that copies bear this notice and the full citation

on the ﬁrst page.To copy otherwise,to republish,to post on servers or to redistribute

to lists,requires prior speciﬁc permission and/or a fee.

Eurosys’13 April 15-17,2013,Prague,Czech Republic

Copyright c2013 ACM978-1-4503-1994-2/13/04...$15.00

Array-based languages such as R and MATLAB provide

an appropriate programming model to express such machine

learning and graph algorithms.The core construct of arrays

makes these languages suitable to represent vectors and ma-

trices,and perform matrix computations.R has thousands

of freely available packages and is widely used by data min-

ers and statisticians,albeit for problems with relatively small

amounts of data.It has serious limitations when applied to

very large datasets:limited support for distributed process-

ing,no strategy for load balancing,no fault tolerance,and is

constrained by a server’s DRAMcapacity.

1.1 Towards an efﬁcient distributed R

We validate our hypothesis that R can be used to efﬁciently

execute machine learning and graph algorithms on large

scale datasets.Speciﬁcally,we tackle the following chal-

lenges:

Effective use of multi-cores.R is single-threaded.The

easiest way to incorporate parallelism is to execute pro-

grams across multiple R processes.Existing solutions for

parallelizing R use message passing techniques,includ-

ing network communication,to communicate among pro-

cesses [25].This multi-process approach,also used in com-

mercial parallel MATLAB,has two limitations.First,it

makes local copies of many data objects,boosting memory

requirements.Figure 1 shows that two R instances on a sin-

gle physical server would have two copies of the same data,

hindering scalability to larger datasets.Second,the network

communication overhead becomes proportional to the num-

ber of cores utilized instead of the number of distinct servers,

again limiting scalability.

Existing efforts for parallelizing R have another limita-

tion.They do not support point-to-point communication.In-

stead data has to be moved fromworker processes to a desig-

nated master process after each phase.Thus,it is inefﬁcient

to execute anything that is not embarrassingly parallel [25].

Even simple iterative algorithms are costly due to the com-

munication overhead via the master.

Imbalance in sparse computations.Most real-world

datasets are sparse.For example,the Netﬂix prize dataset

is a matrix with 480K users (rows) and 17K movies (cols)

but only 100 million of the total possible 8 billion ratings

are available.Similarly,very few of the possible edges are

Server1

Server2

Rprocess

copyof

data

Rprocess

data

Rprocess

copyof

data

Rprocess

copyof

data

local copy

network copy

network copy

Figure 1.

R’s poor multi-core support:multiple copies of data on

the same server and high communication overhead across servers.

0 20 40 60 80 100

1

5

10

50

100

500

1000

5000

Block id

Block density (normalized)

Netflix

LiveJournal

ClueWeb−1B

Twitter

Figure 2.

Variance in block density.Y-axis shows density of a

block normalized by that of the sparsest block.Lower is better.

present in Web graphs.It is important to store and manipu-

late such data as sparse matrices and retain only non-zero en-

tries.These datasets also exhibit skew due to the power-law

distribution [14],resulting in severe computation and com-

munication imbalance when data is partitioned for parallel

execution.Figure 2 illustrates the result of na¨ıve partition-

ing of various sparse data sets:LiveJournal (68Medges) [4],

Twitter (2B edges),pre-processed ClueWeb sample

1

(1.2B

edges),and the ratings from Netﬂix prize (100M ratings).

The y-axis represents the block density relative to the spars-

est block,when each matrix is partitioned into 100 blocks

having the same number of rows/columns.The plot shows

that a dense block may have 1000× more elements than

a sparse block.Depending upon the algorithm,variance in

block density can have a substantial impact on performance

(Section 7).

1.2 Limitations of current data-parallel approaches

Existing distributed data processing frameworks,such as

MapReduce and DryadLINQ,simplify large-scale data pro-

cessing [12,17].Unfortunately,the simplicity of the pro-

gramming model (as in MapReduce) or reliance on rela-

tional algebra (as in DryadLINQ) makes these systems un-

suitable for implementing complex algorithms based on ma-

trix operations.Current systems either do not support state-

ful computations,or do not retain the structure of global

shared data (e.g.,mapping of data to matrices),or do not

allowpoint to point communication (e.g.,restrictive MapRe-

duce communication pattern).Such shortcomings in the pro-

1

http://lemurproject.org/clueweb09.php

gramming model have led to inefﬁcient implementations of

algorithms or the development of domain speciﬁc systems.

For example,Pregel was created for graph algorithms be-

cause MapReduce passes the entire state of the graph be-

tween steps [24].

There have been recent efforts to better support large-

scale matrix operations.Ricardo [11] and HAMA [30] con-

vert matrix operations to MapReduce functions but end up

inheriting the inefﬁciencies of the MapReduce interface.

PowerGraph [14] uses a vertex-centric programming model

(non matrix approach) to implement data mining and graph

algorithms.MadLINQ provides a linear algebra platformon

Dryad but does not efﬁciently handle sparse matrix compu-

tations [29].Unlike MadLINQ and PowerGraph,our aim is

to address the issues in scaling R,a system which already

has a large user community.Additionally,our techniques for

handling load imbalance in sparse matrices can be applicable

to existing systems like MadLINQ.

1.3 Our Contribution

We present Presto,an R prototype to efﬁciently process

large,sparse datasets.Presto introduces the distributed array,

darray,as the abstraction to process both dense and sparse

datasets in parallel.Distributed arrays store data across mul-

tiple machines.Programmers can execute parallel functions

that communicate with each other and share state using ar-

rays,thus making it efﬁcient to express complex algorithms.

Presto programs are executed by a set of worker processes

which are controlled by a master.For efﬁcient multi-core

support each worker on a server encapsulates multiple R in-

stances that read shared data.To achieve zero copying over-

head,we modify R’s memory allocator to directly map data

from the worker into the R objects.This mapping preserves

the metadata in the object headers and ensures that the allo-

cation is garbage collection safe.

To mitigate load imbalance,the runtime tracks the exe-

cution time and the number of elements in each array parti-

tion.In case of imbalance,the runtime dynamically merges

or sub-divides array partitions between iterations and assigns

them to a new task,thus varying the parallelism and load in

the system.Dynamic repartitioning is especially helpful for

iterative algorithms where computations are repeated across

iterations.

We have implemented seven different applications in

Presto,ranging from a recommendation system to a graph

centrality measure.Our experience shows that Presto pro-

grams are easy to write and can be used to express a wide va-

riety of complex algorithms.Compared to published results

of Hadoop and Spark [36],Presto achieves equally good

execution times with only a handful of multi-core servers.

For the PageRank algorithm,Presto is more than 40×faster

than Hadoop and 15× faster than Spark.We also show how

Presto’s multi-core support reduces communication over-

head and ﬁnally measure the impact of dynamic repartition-

ing using two real-world applications.

A

B

C

D

E

A

B

C

D

E

A

B

C

D

E

A

B

C

D

E

A B C D E

1 0 0 0 0

* * * 0 0

A B C D E

* * * * 0

A B C D E

* * * * *

A B C D E

C

E

C

E

C

E

C

E

X

A

1 1 1 0 0

B

0 1 0 1 0

C

0 1 1 0 0

D

0

0

0

1

1

Y= X * G Y

1

= Y * G Y

2

= Y

1

* G

G

D

0

0

0

1

1

E

0 0 0 0 1

Figure 3.

Breadth-ﬁrst search using matrix operations.The k

th

multiplication uncovers vertices up to k hop distance away.

2.Background

Matrix computation is heavily used in data mining,image

processing,graph analysis,and elsewhere [32].Our focus is

to analyze sparse datasets that are found as web graphs,so-

cial networks,product ratings in Amazon,and so on.Many

of these analyses can be expressed using matrix formula-

tions that are difﬁcult to write in data-parallel models such

as MapReduce.

Example:graph algorithms.Many common graph algo-

rithms can be implemented by operating on the adjacency

matrix [19].To perform breadth-ﬁrst search (BFS) from a

vertex i we start with a 1 ×N vector x which has all ze-

roes except the i

th

element.Using the multiplication y=x∗G

we extract the i

th

row in G,and hence the neighbors of ver-

tex i.Multiplying y with G gives vertices two steps away

and so on.Figure 3 illustrates BFS from source vertex A in

a ﬁve vertex graph.After each multiplication step the non-

zero entries in Y

i

(starred) correspond to visited vertices.If

we use a sparse matrix representation for G and x,then the

performance of this algorithm is similar to traditional BFS

implementations on sparse graphs.

The Bellman-Ford single-source shortest path algorithm

(SSSP) ﬁnds the shortest distance to all vertices from a

source vertex.SSSP can be implemented by starting with

a distance vector d and repeatedly performing a modiﬁed

matrix multiplication,d=d⊗G.In the modiﬁed multipli-

cation d(j)=min

k

{d(k)+G(k,j)} instead of the usual

d(j)=

∑

k

{d(k)∗G(k,j)}.In essence,each multiplica-

tion step updates the vertex distances by choosing the mini-

mum of the current distance,and that of reaching the vertex

using one more edge.

2.1 R:An array-based environment

R provides an interactive environment to analyze data.It

has interpreted conditional execution (if),loops (for,

while,repeat),and uses array operators written in C,

C++ and FORTRAN for better performance.Line 1 in Fig-

ure 4 shows how a 3 ×3 matrix can be created.The argu-

ment dim speciﬁes the shape of the matrix and the sequence

10:18 is used to ﬁll the matrix.One can refer to entire subar-

rays by omitting an index along a dimension.For example,in

1:> A<-array(10:18,dim=c(3,3))#3x3 matrix

2:> A

[,1] [,2] [,3]

[1,] 10 13 16

[2,] 11 14 17

[3,] 12 15 18

3:> A[1,]#First row

[1] 10 13 16

4:> idx<-array(1:3,dim=c(3,2))#Index vector

5:> idx

[,1] [,2]

[1,] 1 1

[2,] 2 2

[3,] 3 3

6:> A[idx]#Diagonal of A

[1] 10 14 18

7:> A%

*

%idx#Matrix multiply

[,1] [,2]

[1,] 84 84

[2,] 90 90

[3,] 96 96

Figure 4.

Example array use in R.

line 3 the ﬁrst row of the matrix is obtained by A[1,],where

the column index is left blank to fetch the entire ﬁrst row.

Subsections of a matrix can be easily extracted using index

vectors.Index vectors are an ordered vector of integers.To

extract the diagonal of A we create an index matrix idx in

line 4 whose elements are (1,1),(2,2) and (3,3).In line 6,

A[idx] returns the diagonal elements of A.In a single ma-

chine environment,R has native support for matrix multi-

plication,linear equation solvers,matrix decomposition and

other operations.For example,%∗ % is an R operator for

matrix multiplication (line 7).

3.Programming model

Presto is R with new language extensions and a runtime to

manage distributed execution.The extensions add data dis-

tribution and parallel execution.The runtime takes care of

memory management,scheduling,dynamic data partition-

ing,and fault tolerance.As shown in Figure 5,programmers

write a Presto program and submit it to a master process.

The runtime at the master is in charge of the overall execu-

tion.It executes the programas distributed tasks across mul-

tiple worker processes.Table 1 depicts the Presto language

constructs which we discuss in this section.

3.1 Distributed arrays

Presto solves the problem of structure and scalability by

introducing distributed arrays.Distributed array (darray)

provides a shared,in-memory view of multi-dimensional

data stored across multiple servers.Distributed arrays have

the following characteristics:

Partitioned.Distributed arrays can be partitioned into con-

tiguous ranges of rows,columns or blocks.Users specify

the size of the initial partitions.Presto workers store parti-

tions of the distributed array in the compressed sparse col-

umn format unless the array is deﬁned as dense.Program-

mers use partitions to specify coarse-grained parallelism by

writing functions that execute in parallel and operate on par-

Figure 5.

Presto architecture

titions.Partitions can be referred to by the splits func-

tion.The splits function automatically fetches remote

partitions and combines them to form a local array.For ex-

ample,if splits(A) is an argument to a function exe-

cuting on a worker then the whole array A would be re-

constructed by the runtime,fromlocal and remote partitions,

and passed to that worker.The i

th

partition can be referenced

by splits(A,i).

Shared.Distributed arrays can be read-shared by multiple

concurrent tasks.The user simply passes the array partitions

as arguments to many concurrent tasks.Arrays can be mod-

iﬁed inside tasks and the changes are visible globally when

update is called.Presto supports only a single writer per

partition.

Dynamic.Partitions of a distributed array can be loaded

in parallel from data stores such as HBase,Vertica,or from

ﬁles.Once loaded,arrays can be dynamically re-partitioned

to reduce load imbalance and prevent straggling.

3.2 Distributed parallelism

Presto provides programmers with a foreach construct to

execute deterministic functions in parallel.The functions do

not return data.Instead,programmers call update inside

the function to publish changes.The Presto runtime starts

tasks on worker nodes for parallel execution of the loop

body.By default,there is a barrier at the end of the loop

to ensure all tasks ﬁnish before statements after the loop are

executed.

3.3 Repartition and invariants

At runtime,programmers can use the repartition com-

mand to trigger Presto’s dynamic repartitioning method.

Repartitioning can be used to subdivide an array into a spec-

iﬁed number of parts.Repartitioning is an optional perfor-

mance optimization which helps when there is load imbal-

ance in the system.

One needs be careful while repartitioning structured data,

otherwise program correctness may be affected.For exam-

Functionality Description

darray(dim=,blocks=,

sparse=)

Create a distributed array with dimensions

speciﬁed by dim,and partitioned by blocks

of size blocks.

splits(A,i) Return i

th

partition of the distributed array A

or the whole array if i is not speciﬁed.

foreach(v,A,f()) Execute function f as distributed tasks for

each element vof A.Implicit barrier at the end

of the loop.

update(A) Publish the changes to A.

repartition(A,n=,

merge=)

Repartition A into n parts.

invariant(A,B,type=) Declare compatibility between arrays A and

B by rows or columns or both.Used by

the runtime to maintain invariants while re-

partitioning.

Table 1.

Main programming language constructs in Presto

ple,when multiplying two matrices,the number of rows

and columns in partitions of both the matrices should con-

form.If we repartition only one of the matrices then this in-

variant may be violated.Therefore,Presto allows program-

mers to optionally specify the array invariants in the pro-

gram.We show in Section 5.3 how the runtime can use the

invariant and repartition functions to automati-

cally detect and reduce imbalance without any user assis-

tance.

Note that for programs with general data structures (e.g.,

trees) writing invariants is difﬁcult.However,for matrix

computation,arrays are the only data structure and the

relevant invariant is the compatibility in array sizes.The

invariantin Presto is similar in spirit to the alignment di-

rectives used in High Performance Fortran (HPF [22]).The

HPF directives align elements of multiple arrays to ensure

the arrays are distributed in the same manner.Unlike HPF,

in Presto the invariants are used to maintain correctness dur-

ing repartitioning.

4.Applications

We illustrate Presto’s programming model by discussing the

implementation of two algorithms:PageRank and Alternat-

ing Least Squares.

PageRank.Figure 6 shows the Presto code for PageRank.

M is the modiﬁed adjacency matrix of the Web graph.PageR-

ank is calculated in parallel (lines 7–13) using the power

method [8].In line 1,M is declared as an NxN array.M is

loaded in parallel from the underlying ﬁlesystem using the

Presto driver,and is partitioned by rows.In line 3 the num-

ber of columns of M is used to deﬁne the size of a dense

vector pgr which acts as the initial PageRank vector.This

vector is partitioned such that each partition of pgr has the

same number of rows as the corresponding partition of M.

The accompanying illustration points out that each partition

of the vector pgr requires the corresponding (shaded) parti-

tions of M,Z,and the whole array xold.The Presto runtime

passes these partitions and reconstructs xold from its par-

titions before executing prFunc at each worker.In line 12

#Load data in parallel into adjacency matrix

1:M<- darray(dim=c(N,N),blocks=c(s,N),sparse=T)

2:load(M,file="...")

3:pgr<- darray(dim=c(ncol(M),1),blocks=c(s,1),sparse=F)

4:xold<- darray(dim=c(ncol(M),1),blocks=c(s,1),sparse=F)

5:...

6:invariant(pgr,M,xold,Z,type=ROW)

#Calculate PageRank (pgr)

7:repeat{

#Distributed matrix operations

8:foreach(i,1:numsplits(M),

prFunc(p= splits(pgr,i),m= splits(M,i),

x= splits(xold),z= splits(Z,i)) {

9:p<-(m%

*

%x)+ z

10:update(p)

11:})

12:if(norm(pgr-xold)<1e-9) break

13:xold<-pgr

14:}

N

P

1

P

1

s

P

1

s

P

1

P

2

P

1

P

N/s

P

1

P

2

P

N/s

…

s

P

1

P

2

P

N/s

…

s

P

1

P

2

P

N/s

…

N

N/s

M

N/s

xold

Z

N/s

pgr

N/s

Figure 6.

PageRank on a Web graph.

the norm of the two distributed arrays,pgr and xold,are

calculated in parallel.Internally,normis implemented using

foreach.

Line 6 is an example invariant for the PageRank code.

Each of pgr,M,xold,and Z should have the same number

of rows in each partition.By specifying this invariant,the

programmer constrains the runtime to adhere to this compat-

ibility between arrays even during automatic repartitioning.

Alternating Least Squares.The Alternating Least Squares

(ALS) algorithm with weighted regularization is known to

perform well for matrix problems that arise in recommen-

dation systems,as exempliﬁed by the Netﬂix Prize compe-

tition [2].Figure 7 shows the implementation of a parallel

version of the algorithmin Presto [37].In Line 1 we declare

R,a sparse distributed array partitioned by columns.R is a

nu×nm array where nu and nm are the number of users and

number of movies respectively.The ratings are loaded from

the ﬁlesystemin line 2.In line 3 and 4,we create distributed

arrays to hold the feature matrices for users and movies,and

partition the arrays by columns.Each iteration of the ALS al-

gorithmconsists of two steps:the ﬁrst step updates the movie

feature matrix (M) based on the existing user feature matrix

(U),while the second step uses M to compute updates for U.

Lines 9–29 show how the ﬁrst step of the algorithm is ex-

pressed in Presto.

The foreach loop updates each partition of M in paral-

lel.To update a particular partition,we pass the correspond-

ing partition of the ratings matrix R and the entire user fea-

ture matrix U (lines 9−12).One of the advantages of Presto

is that existing R functions can be used directly.For exam-

ple,in line 23 R’s linear solver minimizes the sum of the

squares of the differences between the known and the pre-

dicted ratings and computes the new values of M.Similarly,

#Load ratings into a sparse matrix

1:R <- darray(dim=c(nu,nm),blocks=c(nu,ms),sparse=T)

2:load(R,file="...")

#Create feature matrices for users and movies

3:M <- darray(dim=c(nf,nm),blocks=c(nf,ms),sparse=F)

4:U <- darray(dim=c(nf,nu),blocks=c(nf,us),sparse=F)

5:...

6:#Initialize the features

7:...

8:repeat {

#Update movie features based on user ratings

9:foreach(i,1:numsplits(M),

10:function(ms = splits(M,i),

11:us = splits(U),

12:rs = splits(R,i)) {

13:lamI <- diag(lambda,nrow=nf,ncol=nf)

#Function to update single movie’s features

15:update

movie <- function(movieRatings) {

16:#Get users who have rated this movie

17:users <- which(movieRatings!= 0)

18:Um <- matrix(nrow=nf,data=us[,users])

19:

20:#Update by minimizing least squares

21:vec <- Um %

*

% movieRatings[users]

22:mat <- Um %

*

% t(Um) + (length(users)

*

lamI)

23:m

new <- solve(mat,vec)

24:return(m

new)

25:}

26:#Calculate updates for movies in this split

27:ms <- apply(rs,2,update

movie)

28:update(ms)

29:})

30:#Similarly update user features using movies

31:...

32:#Check for convergence by computing RMSE

33:rmse <- compute

rmse(U,M)

34:if (rmse < threshold) { break }

35:}

Figure 7.

ALS algorithmon a Netﬂix ratings dataset

in line 27 the familiar R construct apply is used to invoke

the function update

movie on all the movies present in

the partition.Finally,in line 28 the newvalues of movie fea-

tures are made globally visible by invoking update.

5.Systemdesign

The Presto master acts as the control thread.The workers

execute the loop body in parallel whenever foreach loops

are encountered.The master keeps a symbol table which

maps variables to their physical location.This map is used

by workers to exchange information using pairwise commu-

nication.In this paper we describe only the main mecha-

nisms related to multi-core support,and optimizations for

sparsity and caching.

5.1 Versioning arrays

Presto uses versioning to ensure correctness when arrays are

shared across different machines.Each partition of a dis-

tributed array has a version.The version of a distributed ar-

ray is a concatenation of the versions of its partitions,similar

in spirit to vector clocks.Updates to array partitions create

a new version of the partition.By updating the version,con-

current readers of previous versions will not be affected by

the update.For example,the PageRank vector pgr in Fig-

ure 6 starts with version 0,0,..,0.In the ﬁrst iteration,ev-

ery task reads in one partition pgr with version 0.At the

Worker

DRAM

Shareddata

R

instance

O h

Connections

R

instance

R

instance

O

t

h

er

workers

Connections

Networklayer

Figure 8.

Multiple R instances share data hosted in a worker.

Robject

1

Headercorruption

Data

Header

R instance

R instance

write writecorruption!

R

instance

R

instance

Robject

Data

Header

R instance

R instance

gc()

accesserror!

2

Danglingpointer

R

instance

R

instance

Figure 9.

Simply sharing R objects can lead to errors.

end of the task when update is called,a newpartition with

version 1 is created.Hence,after the ﬁrst iteration the dis-

tributed array’s newversion becomes 1,1,..,1.Presto uses

reference counting to garbage collect older versions of ar-

rays not used by any task.

5.2 Efﬁcient multi-core support

Since R is not thread safe,a simple approach to utilize

multi-cores is to start multiple worker processes on the same

server.There are three major drawbacks:(1) on the server

multiple copies of the same array will be created,thus in-

hibiting scalability,(2) copying the data across processes,

using pipes or network,takes time,and (3) the network com-

munication increases as we increase the number of cores be-

ing utilized.

Instead,Presto allows a worker to encapsulate multiple

R processes that can communicate through shared mem-

ory with zero copying overhead (Figure 8).The key idea

in Presto is to efﬁciently initialize R objects by mapping

data using mmap or shared memory constructs.However,

there are some important safety challenges that need to be

addressed.

Issues with data sharing.Each R object consists of a

ﬁxed-size header,and an array of data immediately follow-

ing the header.The header (among other things) has infor-

mation about the type and size of the corresponding data

part.Simply pointing an Rvariable to an external data source

leads to data corruption.As shown in Figure 9,if we were to

share an R object across different R instances two problems

can arise.First,both the instances may try to write instance

speciﬁc values to the object header.This conﬂict will lead

to header corruption.Second,R is a garbage-collected lan-

guage.If one of the instances garbage collects the object then

the other instance will be left with a dangling pointer.

page

Local R object data part

Local R

object

header

page boundary

page boundary

R object

allocator

glibc malloc

Presto malloc

Local objects

Shared objects

Shared data

R’s virtual memory space

1

2

Allocate object

Map shared data

obj. start address

Figure 10.

Shared object allocation in an R instance.

Safe data sharing.We solve the data sharing challenge by

entrusting each worker with management of data shared by

multiple R processes.We only share read-only data since

only one process may write to a partition during a loop iter-

ation and writes always create a new version of a partition.

Presto ﬁrst allocates process local objects in each R instance

and then maps the shared data on the data part of the object.

Since the headers are local to each Rinstance,write conﬂicts

do not occur on the header.

There is another issue that has to be solved:the mmap call

locates data only to an address at a page boundary.However,

R’s internal allocator does not guarantee that the data part

of an object will start at a page boundary.To solve this

issue,Presto overrides the behavior of the internal allocator

of R.We use malloc

hook to intercept R’s malloc() calls.

Whenever we want to allocate a shared R object we use our

custommalloc to return a set of pages rounded to the nearest

multiple of the page size.Once the object has been allocated

the shared data can be mapped using mmap.

Figure 10 shows that R objects are allocated through the

default malloc for local objects and through Presto’s malloc

function for shared objects.The shared objects consist of a

set of pages with the data part aligned to the page bound-

ary.The ﬁrst page starts with an unused region because the

header is smaller than a full page.

When the objects are no longer needed,these spe-

cially allocated regions need to be unmapped.Presto uses

free

hook to intercept the calls to the glibc free() func-

tion.Presto also maintains a list of objects that were spe-

cially allocated.The list contains the starting address and al-

location size of the shared objects.Whenever free is called,

the runtime checks if the object to be freed is present in the

list.If it is then munmap is called.Otherwise,the glibc free

function is called.Note that while the malloc hook is used

only when allocating shared R objects,the free hook is ac-

tive throughout the lifetime of the program,because we do

not know when R may garbage collect objects.

5.3 Dynamic partitioning for sparse data

While shared memory constructs help in reducing the net-

work overhead,the overall time taken for a distributed com-

putation also depends on the execution time.Partitioning a

sparse matrix into contiguous ranges of rows or columns

may lead to uneven distribution of nonzero elements and

cause a skew in task execution times.Moreover,the num-

ber of tasks in the system is tied to the number of partitions

which makes it difﬁcult to effectively use additional work-

ers at runtime.Presto uses dynamic partitioning to mitigate

load imbalance,and to increase or decrease the amount of

parallelism in the program at runtime.One can determine

optimal partitions statically to solve load imbalance but it is

an expensive solution.Such partitions may not remain opti-

mal as data is updated and static partitioning does not adjust

to changes in the number of workers.

Presto uses two observations to dynamically adjust parti-

tions.First,since our target algorithms are iterative,we re-

ﬁne the partitions based on the execution of the ﬁrst few it-

erations.Second,by knowing the invariants for the program

we can re-partition data without affecting correctness.

The Presto runtime tracks both the number of elements in

a partition (e

i

) and the execution time of the tasks (t

i

).It uses

these metrics to decide when to repartition data to reduce

load imbalance.The runtime starts with an initial partition-

ing (generally user-speciﬁed),and in subsequent iterations

may either merge or sub-divide partitions to create newones.

The aimof dynamic partitioning is to keep the partition sizes

and the execution time of each task close to the median [5].

The runtime tracks the median partition size (e

m

) and task

execution time (t

m

).After each iteration,the runtime checks

if a partition has more (fewer) elements than the median

by a given constant (partition threshold e.g.e

i

/e

m

≥δ) and

sub-divides (merges) them.In the PageRank program (Fig-

ure 6),after repartitioning the runtime simply invokes the

loop function (pgFunc) for a different number of partitions

and passes the corresponding data.No other changes are re-

quired.While our current implementation only uses parti-

tion sizes for repartitioning,we plan to explore other metrics

which combine partition sizes and execution times.

For dynamic partitioning,the programmer needs to spec-

ify the invariants and annotate functions as safe under repar-

titioning.For example,a function that assigns the ﬁrst ele-

ment of each partition is unsafe.Such a function is closely

tied to each partition,and if we sub-divide an existing par-

tition then two cells will be updated instead of one.In our

applications,the only unsafe functions are related to initial-

ization such as setting A[i]=1 in breadth-ﬁrst search.

5.4 Co-location,scheduling,and caching

Presto workers execute functions which generally require

multiple array partitions,including remote ones.Presto uses

three mechanisms to reduce communication:locality based

scheduling,partition co-location,and caching.

The Presto master schedules tasks on workers.The mas-

ter uses the symbol table to calculate the amount of re-

mote data copy required when assigning a task to a worker.

It then schedules tasks to minimize data movement.Parti-

tions that are accessed and modiﬁed in the same function

can be co-located on the same worker.As matrix compu-

tations are structured,in most cases co-locating different

array partitions simply requires placing the i

th

partition of

the corresponding arrays together.For example,in PageR-

ank,the i

th

partition of vectors pgr,M,and Z should be

co-located.Instead of another explicit placement directive,

Presto reuses information provided by the programmer in the

invariant function to determine which arrays are related

and attempts to put the corresponding partitions on same

workers.This strategy of co-location works well for our ap-

plications.In the future,we plan to consider work-stealing

schedulers [6,28].

Presto automatically caches and reuses arrays whose ver-

sions have not changed.For example,in the PageRank code

Z is never modiﬁed.After the ﬁrst iteration,workers always

reuse Z as its version never changes.The runtime simply

keeps the reference to partitions of Z alive and is informed

by the master when a new version is available.Due to auto-

matic caching,Presto does not need to provide explicit di-

rectives such as broadcast variables [36].

5.5 Fault tolerance

Presto uses primary-backup replication to withstand failures

of the master node.Only the meta-data information like the

symbol table,programexecution state,and worker informa-

tion is replicated at the backup.The state of the master is

reliably updated at the backup before a statement of the pro-

gram is considered complete.R programs are generally a

couple of hundred lines of code,but most lines perform a

compute intensive task.The overhead of check-pointing the

master state after each statement is lowcompared to the time

spent to execute the statement.

We use existing techniques in literature for worker fault

tolerance.The master sends periodic heartbeat messages to

determine the progress of worker nodes.When workers fail

they are restarted and the corresponding functions are re-

executed.Like MapReduce and Dryad we assume that tasks

are deterministic,which removes checkpointing as data can

be recreated using task re-execution.The matrix computa-

tion focus of Presto simpliﬁes worker fault-tolerance.Arrays

undergo coarse-grained transformations and hence it is suf-

ﬁcient to just store the transformations reliably instead of

the actual content of the arrays.Therefore,Presto recursively

recreates the corresponding versions of the data after a fail-

ure.The information on how to recreate the input is stored

in a table which keeps track of what input data versions and

functions result in speciﬁc output versions.In practice,ar-

rays should periodically be made durable for faster recovery.

6.Implementation

Presto is implemented as an R add-on package and provides

support for the newlanguage features described in Section 3.

Dense and sparse matrices are stored using R’s Matrix li-

Application

Algorithm

R

Presto

Characteristic

LOC

LOC

PageRank

Eigenvector calculation

20

41

Vertex centrality

Graph Algorithm

40

128

Edge centrality

Graph Algorithm

48

132

SSSP

Graph Algorithm

30

62

Netﬂix recom-

mender [37]

Matrix decomposition

78

130

Triangle count [18]

Top-k eigenvalues

65

121

k-Means clustering

Dense linear algebra

35

71

Input data Size Application

Twitter V=54M,E=2B Triangle counting

Twitter-S V=41M,E=1.4B PageRank,Centrality,SSSP

ClueWeb-S V=100M,E=1.2B PageRank

ClueWeb V=2B,E=6B PageRank

Netﬂix V=480K,E=100M Collaborative Filtering

Table 2.

Presto applications and their input data.

brary.Our current prototype has native support for a limited

set of distributed array operators such as load,save,matrix

multiplication,addition,and so on.Other operators and al-

gorithms can be written by programmers using functions in-

side foreach.The implementation of both Presto master

and workers use ZeroMQ servers [16].Control messages,

like starting the loop body in a worker or calls to garbage

collect arrays,are serialized and sent using Google’s pro-

tocol buffers.Transfers of arrays between workers are im-

plemented directly using BSD sockets.The Presto package

contains 800 lines of R code and 10,000 lines of C++ code.

7.Evaluation

Programmers can express various algorithms in Presto that

are difﬁcult or inefﬁcient to implement in current systems.

Table 2 lists seven applications that we implement in Presto.

These applications span graph algorithms,matrix decompo-

sition,and dense linear algebra.The sequential version of

each of these algorithms can be written in fewer than 80 lines

in R.In Presto,the distributed versions of the same applica-

tions take at most 135 lines.Therefore,only a modest effort

is required to convert these sequential algorithms to run in

Presto.

In this paper we focus on PageRank,vertex central-

ity,single-source shortest path (SSSP),triangle counting,

and collaborative ﬁltering.We compare the performance of

Presto to Spark [36],which is a recent in-memory systemfor

cluster computing,and Hadoop-mem,which is Hadoop-0.20

but run entirely on ramfs to avoid disk latencies.Spark per-

forms in-memory computations,caches data,and is known

to be 20× faster than Hadoop on certain applications.In all

the experiments we disregard the initial time spent in load-

ing data from disk.Subsequent references to Hadoop in our

experiments refer to Hadoop-mem.

Our evaluation shows that:

•

Presto is the ﬁrst R extension to efﬁciently leverage

multi-cores by reducing memory and network overheads.

•

Presto can handle load imbalance due to sparsity by dy-

namic partitioning.

•

Presto is much faster than current systems.On PageRank

Presto is 40×faster than Hadoop,15×faster than Spark,

and comparable to MPI implementations.

Our experiments use a cluster of 50 HP SL390 servers with

Ubuntu 11.04.Each server has two 2.67GHz (12-core) Intel

Xeon X5650 processors,96GB of RAM,120GB SSD,and

the servers are connected with full bisection bandwidth on

a 10Gbps network.Presto,Hadoop,and Spark are run with

the same number of workers or mappers.Hadoop algorithms

are part of Apache Mahout [1].

7.1 Application description

Since we have discussed PageRank and SSSP in Section 2,

we brieﬂy describe centrality measure and triangle counting

algorithms.

Centrality.Vertex or edge betweenness centrality deter-

mines the importance of a vertex or edge in a network (e.g.,

social graph) based on the number of shortest paths that in-

clude the vertex or edge.We implement Brandes’ algorithm

for unweighted graphs [7].Each betweenness algorithmcon-

sists of two phases:ﬁrst the shortest paths from each vertex

to all other vertices are determined (using BFS) and then

these paths are used to update the centrality measure using

scalar transformations.In our experiments we show the re-

sults of starting froma vertex whose BFS has 13 levels.

Triangle counting.In large social network graphs,anoma-

lous behavior can be detected by counting the number of tri-

angles that every vertex belongs to [18].Since a direct count

is expensive for large graphs,the number of triangles is ap-

proximated using the top eigenvalues [33].We implement

the iterative Lanzcos algorithm with selective reorthogonal-

ization to ﬁnd the top-k eigenvalues of a matrix.In each

iteration of the algorithm,the sparse input matrix is mul-

tiplied by a dense vector representing the Lanczos vector

from the previous iteration.The result is then orthogonal-

ized to form a new basis vector and the eigenvalues for the

input matrix can be computed using the last k orthogonalized

Lanczos vectors.To handle numerical inaccuracies,the al-

gorithmuses selective-reorthogonalization,where basis vec-

tors are selectively chosen for reorthogonalization.Not every

step in this algorithmneeds to be distributed across the clus-

ter,and using Presto,we can effectively mix parallel com-

putation with computation on the master.For example,the

matrix-vector multiplication is distributed across machines,

but ﬁnding the eigenvalues from the basis vectors is per-

formed on the master using existing R functions.

7.2 Advantages of multi-core support

With Presto’s multi-core support,the memory footprint and

communication overhead are lesser than when using a sin-

gle R-instance per core.In this section we vary the number

of cores and show the time spent during computation,com-

posite creation (constructing a distributed array fromits par-

titions),and data transfer.We use Presto-NoMC to denote

Pagerank

0

5

10

15

20

25

# cores

Time per iteration (sec)

Composite creation

Transfers

Compute

2 4 6 8

With mc

support

No mc

support

With mc

support

No mc

support

With mc

support

No mc

support

With mc

support

No mc

support

Input data

Vertices

#Cores

Additional mem-

ory used (no MC)

Twitter-S

41M

8

2.1G

ClueWeb-S

100M

8

5.3G

Figure 11.

Multi-core (MC) support lowers total execution time

and memory usage on a single server.Lower is better.

the system which does not have multi-core support and has

single core workers.

Single server:low memory overhead.The ﬁrst advantage

of multi-core support is that there is no need to copy data be-

tween two R instances that are running on the same server.

Unlike other R packages,Presto can safely share data across

processes through shared memory.Figure 11 shows the av-

erage iteration time of PageRank on the 1.5B edge Twitter

graph when executed on a single server.The data transferred

in this algorithm is the PageRank vector.In Presto there is

no transfer overhead as all the R instances are on the same

server and can share data.At 8 cores Presto-NoMC spends

7% of the time in data transfers and takes 5% longer to

complete than Presto.The difference in execution time is

not much as communication over localhost is very ef-

ﬁcient even with multiple workers per server.However,the

real win for multi-core support in a single server is the re-

duction in memory footprint.The table in Figure 11 shows

that at 8 cores the redundant copies of the PageRank vec-

tor in Presto-NoMC increase the memory footprint by 2 GB,

which is 10%of the total memory usage.For the Clueweb-S

dataset Presto-NoMC uses up to 5.3 GB of extra memory.

Multiple servers:low communication overhead.The sec-

ond advantage of Presto is that in algorithms with all-to-all

communication (broadcast),the amount of data transferred

is proportional only to the number of servers,not the number

of R instances.Figure 12 shows the signiﬁcance of this im-

provement for experiments on the Twitter-S graph.In these

experiments we ﬁx the number of servers to 5 and vary the

total number of cores.Figure 12(a) shows that the network

transfer overhead for Presto-NoMC is 2.1× to 9.7× higher

than Presto as we vary the total cores from 10 to 40.Worse

still,at 40 cores the PageRank code on Presto-NoMC not

only stops scaling rather it takes more time to complete than

Pagerank

0

2

4

6

8

# cores

Time per iteration (sec)

Composite

creation

Transfers Compute

10 20 40

With mc

support

No mc

support

With mc

support

No mc

support

With mc

support

No mc

support

Vertex centrality

0

100

200

300

# cores

Time (sec)

Composite creation

Transfers

Compute

10 20 40

With mc

support

No mc

support

With mc

support

No mc

support

With mc

support

No mc

support

Figure 12.

Multi-core support reduces communication overhead

in (a) PageRank (b) Centrality.Lower is better.

with 20 cores due to higher transfer overhead.In compari-

son,Presto can complete an iteration of PageRank in about

3 seconds,though there is only marginal beneﬁt of adding

more than 20 cores for this dataset.Figure 12(b) shows

similar behavior for the centrality measure algorithm.Us-

ing Presto the execution time for a single vertex decreases

from 244 seconds at 10 cores to 116 seconds at 40 cores.

In comparison,with no multi-core support Presto-NoMC in-

curs very high transfer overhead at 40 cores and the execu-

tion time is worse by 43%and takes 168 seconds.

7.3 Advantages of dynamic partitioning

While multi-core support lowers the memory and communi-

cation overhead,dynamic repartitioning of matrices reduces

imbalance due to data sparsity.We evaluate the effective-

ness of dynamic partitioning using two algorithms:ﬁrst by

running PageRank on the ClueWeb graph with 2B vertices

and 6B edges and secondly by using the Lanczos method to

ﬁnd top-k eigenvalues on the Twitter graph with 54M ver-

tices and 2B edges.

7.3.1 PageRank

We ﬁrst look at the PageRank experiments which were run

using 25 servers each with 8 R instances.Even though we

use 200 cores in this experiment,we initially partition the

graph into 1000 parts.This allows the scheduler to intel-

ligently overlap computations and attempts to improve the

balance.In this section we show that dynamic repartitioning

improves performance even in such a case.

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

22

Split size (GB)

Iteration count

Figure 13.

We trace the repartitioning seen in the initial four

matrix blocks.Black boxes represents heavy blocks chosen for

repartitioning and gray boxes indicate newly created blocks.

10

15

20

25

30

35

artition Size GB

0

50

100

150

200

250

300

350

400

2000

3000

4000

5000

6000

7000

8000

0 5 10 15 20

Cumulative partitioning time (s)

Time to convergence (s)

Number of Repartitions

Convergence Time

Time spent partitioning

Figure 14.

Convergence time decreases with repartitioning.The

cumulative partitioning time is the time spent in repartitioning.

Effects of repeated partitioning.Figure 13 shows howthe

repartitioning algorithmproceeds on the ClueWeb dataset.A

black colored partition indicates that the particular block was

heavy and chosen for repartitioning.The newly created array

partitions are shown in gray.In Figure 13 the ﬁrst block (also

the densest) is continuously repartitioned fromiteration 1 to

iteration 7 and then again at iterations 11,13,15,and 20.

Overall,repartitioning reduces the size of this partition from

23GB to 2.2GB.

However there is a cost associated with repartitioning.In

our current implementation,PageRank iterations are paused

while the graph is being repartitioned.To quantify the cost-

beneﬁt trade-off,we estimate the total running time of

PageRank as we increase the number of repartitions.Assum-

ing that we need to perform 50 iterations for convergence,

Figure 14 shows the estimated time to converge as we vary

the number of repartitions.We calculate the total execution

time after a certain number of re-partitions by assuming no

more repartitions will occur.For example,at x-axis value

of 5,Presto has performed ﬁve repartitions and the conver-

gence time is 5,126 seconds if no further repartitions occur.

The convergence time reduces by 32%(40 minutes) after the

ﬁrst four repartitions,but the beneﬁts diminish beyond that.

Note that the cumulative time spent in partitioning is a small

fraction of the total execution time (between 0.3%and 3%).

Beneﬁts of reducing imbalance.Reducing the imbalance

among partitions helps decrease the PageRank iteration

time.Figure 15 shows the time taken by each worker dur-

0 50 100 150

1

4

7

10

13

16

19

22

25

Time (seconds)

Workers

Fetch

Execute

0 50 100

150

1

4

7

10

13

16

19

22

25

Time (seconds)

Workers

Fetch

Execute

Figure 15.

Per worker execution time for PageRank (a) before

repartitioning (b) after four repartitions.Shorter bar is better.

0

500

1000

1500

2000

2500

3000

3500

0 5 10 15

20

Time (seconds)

Iteration Count

No Repart

i

t

i

on

i

ng

With Repartitioning

Figure 16.

Comparison of overall execution time with and with-

out repartitioning.Lower is better.

ing one iteration of PageRank.The horizontal bars depict

what part of the total time was spent in transferring data ver-

sus the time taken to perform the computation.Since there

is a barrier at the end of an iteration,the iteration time is de-

termined by the maximum execution time among the work-

ers.Figure 15(a) shows that the slowest worker takes 147

seconds initially but after four repartitions (Figure 15(b)) it

ﬁnishes in 95 seconds thus reducing the per-iteration time.

Reducing imbalance is especially important for iterative

algorithms as the overall execution time can be signiﬁcantly

high due to the skew among workers.As seen in Figure 16,

re-partitioning reduces the completion time by around 822

seconds (13.7 minutes) when the PageRank algorithmis run

for 20 iterations.

7.3.2 Lanczos calculation

We ran the Lanczos algorithm on the Twitter graph using

20 servers.As the dataset was relatively smaller (2B edges),

we divide the graph into 20 partitions.Each server uses as

many R instances as the number of partitions present on the

server.Similar to the PageRank experiments,we study the

beneﬁts of dynamic partitioning by looking at the imbalance

among different workers while executing a single iteration of

0 5 10 15

20

1

4

7

10

13

16

19

Time (seconds)

Workers

Fetch

Execute

0 5 10 15

20

1

4

7

10

13

16

19

Time (seconds)

Workers

Fetch

Execute

Figure 17.

Per worker execution time for Lanczos algorithm on

Twitter dataset (a) before repartitioning (b) after eight repartitions.

Shorter bar is better.

the Lanczos algorithm.The time taken in a single iteration

of the Lanczos algorithmis dominated by the sparse matrix-

dense vector multiplication.The execution time for this step

grows linearly as the number of non-zero entries in the

matrix increase.Figure 17 shows the time taken by each

worker during one iteration with and without repartitioning.

We observe that the Twitter dataset contains one partition

which is much larger than the others,and that repartitioning

reduces the per iteration execution time from19s to 7s.

7.4 Scalability

We evaluate the scalability of Presto using two algorithms:

collaborative ﬁltering [29] (CF) and single-source shortest

path (SSSP).In the following set of experiments we use 8

cores per server and measure the time taken as we increase

the number of cores.In our ﬁrst experiment,we load the Net-

ﬂix ratings dataset [2] as a matrix (R) and run two steps of

a collaborative ﬁltering algorithm.The ﬁrst step computes

R

t

×R,and the second step multiplies R with the output of

the ﬁrst step computing (R×R

t

) ×R.Figure 18 shows the

time taken by each step of the algorithm as we increase the

number of cores.Presto scales quite well in this case with a

speedup of 4.89×(755s to 154s) while using 6×more cores

(8 to 48).These numbers also indicate that Presto’s perfor-

mance is competitive to published results for MadLINQ[29]

which takes 840s on 48 machines for the same algorithm.As

the performance numbers for MadLINQare froma different

hardware conﬁguration,instead of direct comparison,our re-

sults only indicate that Presto can match the performance

of existing matrix-based systems.We also tried to compare

Presto’s performance to a vanilla-R implementation of col-

laborative ﬁltering.While vanilla-R took 385s to perform

R

t

×R,the second multiplication (R×R

t

×R) failed to com-

plete as the intermediate data did not ﬁt in the server’s 96GB

Netflix Collaborative Filtering

0

200

400

600

800

1000

# cores

Total time (sec)

Load t(R)xR Rxt(R)xR

8 16 24 32 40 48

Figure 18.

Running time for collaborative ﬁltering on the Netﬂix

dataset as we increase the number of cores.

Twitter

0

20

40

60

80

100

120

140

# cores

Total time (sec)

Composite creation

Transfers

Compute

16 32 48 64 80 96 112 128

Figure 19.

SSSP scalability on the Twitter-S dataset.

memory.This example highlights the need for distributing

the computation across more servers.

Figure 19 uses SSSP on the 1.5B edge Twitter-S dataset

to showthe performance scaling of Presto.While Presto can

scale to hundreds of cores,and the execution time continues

to decrease,in this case the scaling factor is less than the

ideal.For example,when increasing the cores from 16 to

128 (8×),the execution time drops from 125 seconds to

41 seconds (3×).The less than ideal scaling is a result of

the communication overhead involved in SSSP,which is

proportional to the number of vertices in the graph.In future

we plan to rewrite the SSSP algorithmto use block partitions

of the matrix (instead of row partitions) so that no single R

instance requires the full shortest path vector.

7.5 Comparison with MPI,Spark,and Hadoop

PageRank experiments on the 1.2B edge ClueWeb-S graph

shows that Presto is more than 40×faster than Hadoop,more

than 15×faster than Spark,and can outperformsimple MPI

implementations.

MPI.We implemented PageRank using sparse matrix and

vector multiplication in MPI.The communication phase in

the code uses MPI

Allgather to gather the partitions of

the PageRank vector from processes and distribute it to all.

Figure 20(a) shows that Presto outperforms the MPI code

MPI Pagerank

0

2

4

6

8

10

12

14

# cores

Time per iteration (sec)

Transfers Compute

8 16 32 64

Presto

MPI

Presto

MPI

Presto

MPI

Presto

MPI

Spark PageRank

1

2

5

10

20

50

100

200

500

# cores

Time per iteration (sec)

Transfers Compute

8 16 32 64

Presto

Presto

Presto

Presto

Spark

Spark

Spark

Spark

Hadoop PageRank

1

2

5

10

20

50

100

200

500

# cores

Time per iteration (sec)

8 16 32 64

Presto

Presto

Presto

Presto

Hadoop−mem

Hadoop−mem

Hadoop−mem

Hadoop−mem

Figure 20.

Performance advantage over (a) MPI (b) Spark and (c) Hadoop.Lower is better.

sometimes by 2×.There are two reasons for this perfor-

mance difference.First,the MPI code does not handle com-

pute imbalance.For example,at 64 cores one MPI process

ﬁnishes in just 0.6 seconds while another process takes 4.4

seconds.Since processes wait for each other before the next

iteration,the compute time is determined by the slowest pro-

cess.Second,while MPI’s network overhead is very low at

8 processes,it increases with the increase in the number of

cores.However,for Presto the network overhead is propor-

tional to the number of multi-core servers used,and hence

does not increase at the same rate.With more effort one can

implement multi-threaded programs executing at each MPI

process.Such an implementation will reduce the network

overhead but not the compute imbalance.

Spark.We use Spark’s PageRank implementation [36] to

compare its performance with Presto.Spark takes about

64.185 seconds per-iteration with 64 cores.The per-iteration

time includes a map phase which computes the rank of ver-

tices and then propagates themto reducers that sumthe val-

ues.We found that the ﬁrst phase was mostly compute inten-

sive and took around 44.3 seconds while the second phase

involved shufﬂing data across the network and took 19.77

seconds.At fewer cores,the compute time is as high as

267.26 seconds with 8 cores.The main reason why Spark is

at least 15×slower than Presto is because it generates a large

amount of intermediate data and hence spends more time

than Presto during execution and network transfers.Note

that the Y-axis in the plot is log scale.

Hadoop.Figure 20(c) compares the performance of Ma-

hout’s PageRank implementation to that of Presto.Since

mappers and reducers overlap during the Hadoop compu-

tation,we depict only the overall execution time.Each it-

eration of Mahout’s PageRank takes 161 seconds with 64

mappers.In comparison each iteration of PageRank in Presto

takes less than 4 seconds.Aportion of the 40×performance

difference is due to the use of Java.However unlike Presto,

MapReduce has the additional overhead of the sort phase

and the time spent in deserialization.Presto preserves the

matrix structure in between operations,and also eliminates

the need to sort data between iterations.

Existing R packages.To obtain a baseline for R-

implementations,we measured the time taken for a single

PageRank iteration using vanilla-R.R takes 30 seconds per

iteration in our setup and was faster than Presto which takes

58 seconds when using a single-core.We found that Presto

was slower due to the overheads associated with mapping

and processing 128 partitions.When the dataset was merged

to forma single partition (similar to vanilla-R case) Presto’s

performance matches that of vanilla-R.However,partition-

ing the dataset is helpful when using multiple cores.Presto

running on 8 cores takes less than 10 seconds for each

PageRank iteration.

Unfortunately,existing parallel R packages only allow

side-effect free functions to be executed in parallel.It means

that R objects in workers are deleted across iterations.Thus,

to run more than one iteration of parallel PageRank the

whole graph needs to be reloaded in the next iteration mak-

ing the measurements ﬂawed.Instead,we ran a microbench-

mark with 8 cores where the sparse matrices were not ex-

changed and only a dense vector of 100M entries was ex-

changed after each round (similar to the PageRank vector).

By efﬁciently using multi-cores and worker-worker commu-

nication Presto is more than 4×faster than doMC,a parallel-

R package.

8.Discussion

Presto makes it easy for users to algorithmically explore

large datasets.It is a step towards a platform on which

high level libraries can be implemented.We believe that

Presto packages that implement scalable machine learning

and graph algorithms will help the large R user base reap the

beneﬁts of distributed computing.

However,certain challenges remain both in the current

prototype and in the applicability of R to all problems.First,

the current prototype is limited by main memory:datasets

need to ﬁt the aggregate memory of the cluster.While most

pre-processed graphs are in the low terabyte size range,for

larger datasets it may be economical to use an out-of-core

system.We are working on adding out-of-core support for

distributed arrays in future versions of Presto.

Second,Presto assumes that there is one writer per parti-

tion during a single foreach execution.Instead of using locks

to synchronize concurrent accesses,in Presto multiple tasks

explicitly write to their partitions and then combine or re-

duce the data in another foreach loop.For example,in

k-means the centers are calculated and stored in separate ar-

rays by each task and then summed up in another loop.This

programming model retains the simplicity of R and we have

found it sufﬁcient for all the algorithms implemented so far.

This model may not be appropriate for implementing irreg-

ular applications like Delaunay mesh reﬁnement that require

ﬁne grained synchronization [20].

When applied to different datasets,array-based program-

ming may require additional pre-processing.For example,

Presto is based on R and is very efﬁcient at processing ar-

rays.However,graphs may have attributes attached to each

vertex.An algorithmwhich uses these attributes (e.g.,search

shortest path with attribute pattern) may incur the additional

overhead of referencing attributes stored in R vectors sepa-

rate from the adjacency matrix.In general,real world data

is semi-structured and pre-processing may be required to

extract relevant ﬁelds and convert them into arrays.Unlike

the Hadoop ecosystem which has both storage (HDFS) and

computation (MapReduce),Presto only has a efﬁcient com-

putation layer.In our experience,it’s easier to load data into

Presto if the underlying store has tables (databases,HBase,

etc.) and supports extraction mechanisms (e.g.,SQL).

9.Related Work

Dataﬂow models.MapReduce and Dryad are popular

dataﬂow systems for parallel data processing [12,17].To

increase programmer productivity high-level programming

models–DryadLINQ [35] and Pig [27]—are used on top of

MapReduce and Dryad.These systems scale to hundreds

of machines.However,they are best suited for batch pro-

cessing,and because of their restrictive programming and

communication interface make it difﬁcult to implement ma-

trix operations.Recent improvements,such as HaLoop [9],

Twister [13],and Spark [36],do not change the program-

ming model but improve iterative performance by caching

data or using lineage for efﬁcient fault tolerance.CIEL in-

creases the expressibility of programs by allowing newdata-

dependent tasks during job execution [26].However,none

of these systems can efﬁciently express matrix operations.

Piccolo runs parallel applications that can share state us-

ing distributed,in-memory,key-value tables [28].Compared

to MapReduce,Piccolo is better suited for expressing ma-

trix operations.However,Piccolo’s key-value interface opti-

mizes for lowlevel reads and writes to keys instead of struc-

tured vector processing.Unlike Presto,Piccolo does not han-

dle sparse datasets and the resulting load imbalance.

Pregel and GraphLab support bulk synchronous process-

ing (BSP [34]) to execute parallel programs [23,24].With

BSP,each vertex processes its local data and communicates

with other vertices using messages.Both systems require an

application to be (re)written in the BSP model.Presto shows

that the widely used R system can be extended to give sim-

ilar performance without requiring any programming model

changes.Presto’s execution time of PageRank on the Twit-

ter graph (Figure 11,8 cores,7.3s) compares favorably to

published results of PowerGraph (512 cores,3.6s) [14].

Matrix computations.Ricardo [11] and HAMA [30] use

MapReduce to implement matrix operations.While they

solve the problem of scaling to large datasets,the imple-

mentation is inefﬁcient due to the restrictive MapReduce in-

terface.In light of this observation,MadLINQ provides a

platformon Dryad speciﬁcally for matrix computations [29].

Similar to Presto,MadLINQ reuses existing matrix libraries

on local partitions,is fault tolerant and distributed.While

MadLINQ’s techniques are efﬁcient for dense matrices,their

systemdoes not efﬁciently handle sparse datasets,or support

dynamic partitioning to overcome load imbalance.

Popular high-performance computing (HPC) systems like

ScaLAPACK do not support general sparse matrices.The

few systems that do support sparse matrices (SLEPc [15],

ARPACK[21]) typically provide only eigensolvers.To write

a new algorithm,such as the betweenness centrality,one

would have to implement it with their low level interfaces

including FORTRANcode.None of these systems have load

balancing techniques or fault tolerance.MATLAB’s parallel

computing toolbox and existing efforts in parallelizing Rcan

run single programs on multiple data.Unlike these systems,

Presto can safely share data across multiple processes,has

fewer redundant copies of data,and can mitigate load imbal-

ance due to sparse datasets.

Parallel languages.HPC applications use explicit mes-

sage passing models like MPI.MPI programmers have the

ﬂexibility to optimize the messaging layer but are difﬁcult

to write and maintain.Newparallel programming languages

like X10 [10] and Fortress [31] use the partitioned global

address space model (PGAS).These languages are not op-

timized for matrix operations and the programmer has to

deal with low level primitives like synchronization and ex-

plicit locations.For example,in X10 programmers specify

on what processors computations should occur using Place.

None of these languages are as popular as R,and users will

have to rewrite hundreds of statistical algorithms that are al-

ready present in R.

10.Conclusion

Presto advocates the use of sparse matrix operations to sim-

plify the implementation of machine learning and graph al-

gorithms in a cluster.Presto uses distributed arrays for struc-

tured processing,efﬁciently uses multi-cores,and dynami-

cally partitions data to reduce load imbalance.Our experi-

ence shows that Presto is a ﬂexible computation model that

can be used to implement a variety of complex algorithms.

Acknowledgments:We thank the anonymous reviewers

and our shepherd,Jean-Philippe Martin,for their valuable

feedback.Aurojit Panda and Evan Sparks suggested im-

provements to earlier drafts of this paper.Finally,we thank

John Byrne,Kyungyong Lee,Partha Ranganathan,and Van-

ish Talwar for assisting us in developing Presto.

References

[1] Apache mahout.http://mahout.apache.org.

[2] Netﬂix prize.http://www.netflixprize.com/.

[3] The R project for statistical computing.http://www.r-

project.org.

[4] Stanford network analysis package.http://snap.

stanford.edu/snap.

[5] G.Ananthanarayanan,S.Kandula,A.Greenberg,I.Stoica,

Y.Lu,B.Saha,and E.Harris.Reining in the outliers in map-

reduce clusters using Mantri.In In OSDI’10,Vancouver,BC,

Canada,2010.

[6] R.D.Blumofe and C.E.Leiserson.Scheduling multithreaded

computations by work stealing.In SFCS ’94,pages 356–368,

Washington,DC,USA,1994.

[7] U.Brandes.A faster algorithm for betweenness centrality.

Journal of Mathematical Sociology,25:163–177,2001.

[8] S.Brin and L.Page.The anatomy of a large-scale hypertextual

Web search engine.In WWW7,pages 107–117,1998.

[9] Y.Bu,B.Howe,M.Balazinska,and M.D.Ernst.HaLoop:

Efﬁcient iterative data processing on large clusters.Proc.

VLDB Endow.,3:285–296,September 2010.

[10] P.Charles,C.Grothoff,V.Saraswat,C.Donawa,A.Kielstra,

K.Ebcioglu,C.von Praun,and V.Sarkar.X10:An object-

oriented approach to non-uniformcluster computing.In OOP-

SLA’05,pages 519–538,2005.

[11] S.Das,Y.Sismanis,K.S.Beyer,R.Gemulla,P.J.Haas,

and J.McPherson.Ricardo:Integrating R and Hadoop.In

SIGMOD Conference’10,pages 987–998,2010.

[12] J.Dean and S.Ghemawat.MapReduce:Simpliﬁed data pro-

cessing on large clusters.Commun.ACM,51(1),2008.

[13] J.Ekanayake,H.Li,B.Zhang,T.Gunarathne,S.-H.Bae,

J.Qiu,and G.Fox.Twister:A runtime for iterative MapRe-

duce.In HPDC ’10,pages 810–818,2010.

[14] J.E.Gonzalez,Y.Low,H.Gu,D.Bickson,and C.Guestrin.

PowerGraph:Distributed Graph-Parallel Computation on Nat-

ural Graphs.In OSDI’12,Hollywood,CA,October 2012.

[15] V.Hernandez,J.E.Roman,and V.Vidal.Slepc:A scalable

and ﬂexible toolkit for the solution of eigenvalue problems.

ACMTrans.Math.Softw.,31(3):351–362,Sept.2005.

[16] P.Hintjens.ZeroMQ:The Guide,2010.

[17] M.Isard,M.Budiu,Y.Yu,A.Birrell,and D.Fetterly.Dryad:

Distributed data-parallel programs from sequential building

blocks.In EuroSys ’07,pages 59–72,2007.

[18] U.Kang,B.Meeder,and C.Faloutsos.Spectral Analysis

for Billion-Scale Graphs:Discoveries and Implementation.In

PAKDD (2),pages 13–25,2011.

[19] J.Kepner and J.Gilbert.Graph Algorithms in the Language of

Linear Algebra.Fundamentals of Algorithms.SIAM,2011.

[20] M.Kulkarni,K.Pingali,B.Walter,G.Ramanarayanan,

K.Bala,and L.P.Chew.Optimistic parallelism requires ab-

stractions.In PLDI ’07,pages 211–222.

[21] R.B.Lehoucq,D.C.Sorensen,and C.Yang.ARPACK users’

guide - solution of large-scale eigenvalue problems with im-

plicitly restarted Arnoldi methods.Software,environments,

tools.SIAM,1998.

[22] D.Loveman.High performance Fortran.IEEE Parallel &

Distributed Technology:Systems &Applications,1(1):25–42,

1993.

[23] Y.Low,J.Gonzalez,A.Kyrola,D.Bickson,C.Guestrin,and

J.M.Hellerstein.GraphLab:A New Framework for Parallel

Machine Learning.CoRR,pages 1–1,2010.

[24] G.Malewicz,M.H.Austern,A.J.Bik,J.C.Dehnert,I.Horn,

N.Leiser,and G.Czajkowski.Pregel:Asystemfor large-scale

graph processing.In SIGMOD ’10,pages 135–146,2010.

[25] Q.E.McCallum and S.Weston.Parallel R.O’Reilly Media,

Oct.2011.

[26] D.G.Murray and S.Hand.C

IEL

:A universal execution

engine for distributed data-ﬂow computing.In NSDI ’11,

Boston,MA,USA,2011.

[27] C.Olston,B.Reed,U.Srivastava,R.Kumar,and A.Tomkins.

Pig latin:A not-so-foreign language for data processing.In

SIGMOD’08,pages 1099–1110,2008.

[28] R.Power and J.Li.Piccolo:Building fast,distributed pro-

grams with partitioned tables.In OSDI ’10,Vancouver,BC,

Canada,2010.USENIX Association.

[29] Z.Qian,X.Chen,N.Kang,M.Chen,Y.Yu,T.Moscibroda,

and Z.Zhang.MadLINQ:large-scale distributed matrix com-

putation for the cloud.In EuroSys ’12,pages 197–210,2012.

[30] S.Seo,E.J.Yoon,J.Kim,S.Jin,J.-S.Kim,and S.Maeng.

Hama:An efﬁcient matrix computation with the mapreduce

framework.In In CLOUDCOM’10,pages 721–726.

[31] G.L.Steele,Jr.Parallel programming and code selection in

fortress.In PPoPP ’06,pages 1–1,2006.

[32] G.Strang.Introduction to Linear Algebra,Third Edition.

Wellesley Cambridge Pr,Mar.2003.

[33] C.E.Tsourakakis.Fast counting of triangles in large real net-

works without counting:Algorithms and laws.In ICDM’08,

pages 608–617.IEEE,2008.

[34] L.G.Valiant.A bridging model for parallel computation.

Commun.ACM,33:103–111,August 1990.

[35] Y.Yu,M.Isard,D.Fetterly,M.Budiu,U.Erlingsson,P.K.

Gunda,and J.Currey.DryadLINQ:A system for general-

purpose distributed data-parallel computing using a high-level

language.In OSDI ’08,pages 1–14,2008.

[36] M.Zaharia,M.Chowdhury,T.Das,A.Dave,J.Ma,M.Mc-

Cauley,M.J.Franklin,S.Shenker,and I.Stoica.Resilient

distributed datasets:a fault-tolerant abstraction for in-memory

cluster computing.In NSDI’12,San Jose,CA,2012.

[37] Y.Zhou,D.Wilkinson,R.Schreiber,and R.Pan.Large-Scale

Parallel Collaborative Filtering for the Netﬂix Prize.In AAIM

’08,pages 337–348,Shanghai,China,2008.

## Σχόλια 0

Συνδεθείτε για να κοινοποιήσετε σχόλιο