ScaleOut hServer™ V2 Java Programmer's Guide - ScaleOut Software

squawkpsychoticΛογισμικό & κατασκευή λογ/κού

2 Δεκ 2013 (πριν από 3 χρόνια και 6 μήνες)

176 εμφανίσεις


Version
5.0.25

-

© Copyright 2013 by ScaleOut Software, Inc.

ScaleOut
h
Server


V2

Java
Programmer’s

Guide


The ScaleOut h
Server


V2

Java API

library integrates
a
Hadoop
MapReduce

execution e
ngine

with ScaleOut h
Server
’s in
-
memory data grid (IMDG)
.
This
open source
library
1

consists of
several
components:
a

Hadoop
MapReduce
execution

engine
, which runs MapReduce jobs in
memory
without using Hadoop j
ob trackers or task trackers, and
four

I
/
O components to pass data between
the
IMDG and
a
MapReduce job.
The
I
/
O

component
s

include

the
Named Map Input Format
, the

Named

Cache
I
nput
F
ormat
,

and
the
Grid
Output F
ormat
, which
together
allow
MapReduce applications
to use
the
IMDG
as a data so
urce and/or result storage for MapR
educe jobs.

In
addition, t
he
Dataset Input F
ormat

accelerates the

performance of
MapR
educe
jobs

by
caching HDFS datasets in the IMDG.


Using ScaleOut hServer, d
evelopers can
write

and run standard
Hadoop

MapReduce
appl
ications

in Java, and these applications can be executed
stand
-
alone
by ScaleOut hServer

s execution engine. The Apache
Hadoop distribution does
not

need to be insta
lled to run MapReduce
programs; it
optionally
can be installed

to
make use of other
Hadoop
components
, such as the Hadoop Distributed File System (HD
F
S).
(
If
HDFS is used to store data

sets analyzed by MapReduce, ScaleOut
hServer should be installed on the same cluster of servers to minimize
network overhead.
) ScaleOut hServer

s execution engine offers very fast job scheduling (measured
in milliseconds), highly optimized
data
combining and
shuffling, in
-
memory storage of
intermediate key/value pairs within the IMDG,
optional use of sorting,
and
fast,

pipelined

access to
in
-
memory data within the IMDG for
analysis
.

In addition,
ScaleOut hServer automati
cally sets the
number of splits,
partitions
, and slots

for IMDG
-
based data. Lastly, the
performance of the Hadoop
MapReduce engine automatically
scales

as s
ervers are added to the cluster
,

and IMDG
-
based data is
automatically redistributed across the cluster

as needed
.


D
evelopers can use

ScaleOut hServer

s

Java APIs
2

to create, read, update, and delete objects within
the IMDG.
This enables MapReduce applications to input

live


data sets

which are stored and
updated within the IMDG.
Complex
IMDG
-
based objects can be stored

within
a

named cache
,
which provides comprehensive semantics, such as
object timeouts, dependency relationships,
pessimistic locking
, and access by remote IMDGs
.
These objects are input to MapReduce
application
s using the
Named Cache

input format.
Alternatively, large populations of small



1

The
open source
ScaleOut hServer Java API library

(
soss
-
hserver
-
2
.0.jar
) is licensed under the Apache License,
Version 2.0 (
http://www.apache.org/licenses/LICENSE
-
2.0
).

2

The ScaleOut StateServer Java API library (soss
-
jnc
-
5.1.jar) is licensed under the ScaleOut StateServer End User
License Agreement.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
2

key/value pairs

can be efficiently stored within a
named map
, which provides highly efficient
memory usage and streamlined sema
n
tics following the Java concurrent

map model. These objects
can be input to MapReduce applications using the
Na
med Map

input format. The
Grid

o
utput
format

can be used to output objects from MapRe
duce applications to
a
named cache or a named
map.



Note that some advanced features of the Ja
va APIs, such as event handling, parallel query

and
parallel met
hod invocation

are only available under a full ScaleOut StateServer
®

or ScaleOut
Analytics Server
®

license.


This programming guide is intended to b
e a supplement to the Java API documentation and
the
ScaleOut StateServer (SOSS)
Help F
ile included with ScaleOut h
Server. It focuses on the Java
components
used

with Hadoop MapReduce applications.

1.
Installation

of the IMDG

Please refer to ScaleOut

StateServer
help file

for instructions

on installing the IMDG service on a
cluster

of servers
.

ScaleOut

hServer installs
the ScaleOut

StateServ
er
grid service on all servers
.
When
a
Map Reduce job

is
started, ScaleOut hServer

automatically starts Java Virtual Machines
(JVM
s) on all servers

(called an
invocation grid
)

to implement its scalable Hadoop MapReduce
engine.




Here are some

quick start


instructions for installing the IMDG on Linux. On each server in the
cluster
:

1.

Download the rpm file from the Scal
eOut Software web site.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
3

2.

Install the

rpm
:

sudo rpm

ivh

so
s
s
-
5.0.
2
5
-
2
2
5
.el5.x86_64.rpm

(It will be installed into
/usr/local/soss5
.
)

3.

Verify the daemon is running:

soss query

4.

Configure the network settings to use the desired network, for example
:

soss set
net_interface=10.0.3.0

(You also can edit the
soss_params.txt

file and restart the daemon.)

5.

Join this

server to the cluster of IMDG servers:

soss join

The IMDG servers will automatically discover each other and balance the storage workload among
all server
s.

Sufficient

physical
memory should be provisioned for the IMDG to
hold all data set objects

and
their
associated replicas

following the best practices described in the SOSS Help File
.
By default,
named cache objects have one replica on a different server to
ensure

high availability in case a
server fails.
For example, if a 100GB data set is to be stored in the IMDG, this will require
approximately
200GB of
aggregate
memory for the data set and its replicas
(using the default
parameters). If the cluster has four servers, this will require 50GB per server. Note that additional
memory is required for object metadata and other data structures used by the IMDG.

To maximize
the
performance

of MapReduce applications,
n
amed map objects a
nd intermediate
key/va
lue pairs
do not use replicas
; r
eplicas optionally can be enabled for named maps

(described below)
.

For MapReduce appli
cations which input data from HDFS and store results in HDFS, the IMDG

s
memory is only used to store intermediate results
sent from
the

map
pers to the
reduce
rs. If
multiple

grid servers are added to the cluster, their
memory

automatically

is

combined
to

store very large
sets of
intermediate results.

R
eplicas
are not used for intermediate results, and these

res
ults are
cleared when a Map
R
educe job completes.

The
Java API library for ScaleOut hServer can be fo
und in
soss
-
hserver
-
2
.0
.jar
, and the Java API
library for creating, reading, updating, and deleting objects can be found in
soss
-
jnc
-
5.1
.jar
.

These jars and their

dependencies

are located

in the
java_api

directory

of the
ScaleOut
Stat
eServer

installation directory
.

2. Running Hadoop MapReduce jobs with ScaleOut hServer

ScaleOut hServer

executes

MapReduce jobs without
using

the Hadoop job tracker/task tracker
infrastructure. The operations are performed through
an
invocation grid

(IG)
,

that is,
a

set of worker
JVM
s
, each of which is started by
its corresponding

IMDG
grid service
. The
intermediate
data
between mappers and reducers are
stored in

the IMDG.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
4

If the input or output format specified for the MapReduce job do
es

not use HDFS as a
data store
, i
t
is not required to
install
the

Apache (or other) Hadoop distribution a
nd
start
Hadoop processes on
the
IMDG
servers
.
If the job
uses
HDFS for input and/or output, the name

nodes and data nodes
must be running for the job to complete
.

2.1 Requirements

The following

requirements apply to MapReduce applications executed using ScaleOut hServer:



The MapReduce application must be compatible with the

Apache
Hadoop 1.x.x

distribution
.




The job
must
use the new MapReduce API (
org.apache.hadoop.mapreduce
)
.



If
a

combiner is specified, it
must

emit
no more than one

key value pair per call, and the
emitted
key
must

be the same as the parameter key.



The input/output keys and values of the mapper and the reducer
must

implement
Writable

or
Serializable
. If
s
orting is enabled, the mapper output key
must

implement
WritableComparable
/
Comparable
.


2.
2

Configuring the

IMDG to run MapReduce jobs

After the IMDG is installed as described in section 1,
two
additional steps should
be take
n

to
configure it to run MapReduce jobs.



Each IMDG
grid service (
daemon
)

relies on the
JAVA_HOME

environment variable in its
context to find the Java installation
directory
used to start up worker JVMs.

This variable is
set in the
/etc/init.d/sossd
script. This script should be edited if the default
JAVA_HOME

value (
/usr/lib/jvm/jre
) does not point to the Java installation directory
.

(
A
lternatively,
/usr/lib/jvm/jre

can be configured as a symbolic link to the Java installation directory
.
)




T
he
ScaleOut hServer libra
r
y JARs should be present
i
n the classpath of the invocation
JVM.

Note
:
If the MapReduce

job

is intended to be run from within
an
other application and
not from the command line, the ScaleOut hServer libraries should be present in the classpath

of that application
.

2.
3

Using
h
ServerJob to submit jobs to the IMDG

To
execute a

MapReduce job
using

ScaleOut hServer
, the
HServerJob

class should be used instead
of
Job

for configuring the MapReduce job.
The
HServerJob

supports the constructor signatures
of
Job

and, since it extends the
Job

class, the methods for configuring the job parameters
are
unchanged
. For example,
to apply this change to
the W
ord
C
ount example:




ScaleOut hServer™

V2

Java Programmer’s Guide


Page
5

//Th
is job will run
using the Hadoop

job tracker
:

public

static

void

main
(
String
[]

args
)



throws


Exception
{



Configuration conf
=

new

Configuration
();



Job job
=

new

Job
(
conf
,

"wordcount"
);




job
.
setOutputKeyClass
(
Text
.
class
);


job
.
setOutputValueClass
(
IntWritable
.
class
);



job
.
setMapperClass
(
Map
.
class
);


job
.
setReducerClass
(
Reduce
.
class
);



job
.
setInputFormatClass
(


TextInputFormat
.
class
);


job
.
setOutputFormatClass
(


TextOutputFormat
.
class
);



FileInputFormat
.
addInputPath
(


job
,

new

Path
(
args
[
0
]));


FileOutputFormat
.
setOutputPath
(


job
,

new

Path
(
args
[
1
]));




job
.
waitForCompletion
(
true
);


}

//This job will run
using

ScaleOut hServer:

public

static

void

main
(
String
[]

args
)



throws

Exception
{



Configuration conf
=

new

Configuration
();



Job job
=

new

HServerJob
(
conf
,

"wordcount"
);




job
.
setOutputKeyClass
(
Text
.
class
);


job
.
setOutputValueClass
(
IntWritable
.
class
);



job
.
setMapperClass
(
Map
.
class
);


job
.
setReducerClass
(
Reduce
.
class
);



job
.
setInputFormatClass
(


TextInputFormat
.
class
);


job
.
setOutputFormatClass
(


TextOutputFormat
.
class
);



FileInputFormat
.
addInputPath
(


job
,

new

Path
(
args
[
0
]));


FileOutputFormat
.
setOutputPath
(


job
,

new

Path
(
args
[
1
]));




job
.
waitForCompletion
(
true
);


}


There is a constructor signature in
HServerJob

which takes
an
additional
boolean

parameter

to
control whether the
reducer input keys

are sorted
:

public

HServerJob
(
Configuration conf
,

String jobName
,

boolean

sortEnabled
)


To
maximi
ze

performance,
t
his parameter allows
sorting of
the reducer input keys for
each partition

to be disabled
.

2.
4

Running
a MapReduce Job from the Hadoop Command Line

If
the

Apache Hadoop distribution is installed on the IMDG

s cluster, you can use the Hadoop
command line to run a MapReduce job within the IMDG.
T
o do this, be sure that t
he
ScaleOut
hServer libra
r
y JARs
are

present
i
n the classpath of the invocation JVM. This
can

be
accomplished

by adding

the

HAD
O
OP_CLASSPATH variable to the
conf/hadoop
-
env.sh

in th
e Hadoop
installation directory
, as follows:


export
HADOOP_CLASSPATH=

/usr/local/soss/java_api/*:/usr/local/soss/java_api/lib/*

These changes are sufficient to run
a
MapReduce application from the
Hadoop
com
mand line. For
example, if the
W
ord
C
ount example is modified as
described

in
the
previous section and packaged
as

wordcount
-
hserver.jar
, it can be run from the c
ommand

line as follows:


$ hadoop jar wordcount
-
hserver.jar org.myorg.WordCount input

output


2.
5

Explicitly specifying the invocation grid

An invocation grid (IG) represents a
set of JVMs attached to the grid se
rvice processes within the
IMDG

and
is
used to
execute MapReduce applications within
the

IMDG
.
Each invocation grid is
identified by a user
-
specified name

and
specifies

the necessary dependencies (JARs, classes,
folders, or files) and JVM parameters.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
6

By d
efault, ScaleOut hServer
automatically
creates
an

IG for
each

MapReduce
application

s
execution
; the IG
includes the JAR file
for

the MapReduce job as the only
dependency
.
The

IG i
s
loaded
and
the
dependencies are

copied to the worker nodes and t
he worker JVMs
before the job is
run
. After
execution completes
, the IG is unloaded
,

and

the JVMs
shut down.

ScaleOut hServer
supports the use of m
ultiple IGs
to independently run
dif
ferent

jobs

at the s
ame time
.

To avoid
creating

an
IG

for each job, the IG can be
managed

manually
and

provide
d
as a
constructor parameter to the
HServerJob

instance.
T
his is advantageous for

the following reasons:



Loading the IG can take a considerable time (up to several seconds).

If
a

relatively short job
is run several times
in a short time span, each job run

can share a single invocation grid

to
maximize

performance
.



If the job

s

dependencies include multiple JARs and classes, they can be specified explicitly
through the invocation grid.



A
custom IG can be used to pass parameters to the w
orker JVMs, such as memory or
garbage collector

settings.



An
IG can be reused for NamedMap para
llel
invocations or queries

that may be
needed

in
conjunction

w
ith MapReduce applications.

If an
HServerJob

creates its own IG
, the job will automatically unload
the IG

upon completion.
However, i
f the
HServerJob

is provided with
a
preexisting IG, it will not be automatically unloaded
a
fter the completion of the job
.
In this case, t
he
unload()

method

should be called on the IG object
to dispose of the IG
when

it
is no longer needed for
further

MapReduce jobs or other parallel

invocations
.

Invocation grids
are created by configuring
a
builder object and calling
load()
. If the IG is intended
to be used for
h
Server invocations, the buil
der

should be created with

HServerJob
.
getInvocationGridBuilder(…
)
instead of
the
InvocationGridBuilder

constructor.

In the following

example
, a
custom
-
built IG
is configured with custom JARs, class dependencies,
and JVM parameters, used
to perform multiple jobs

in rapid succession
,

and

then explicitly
unloaded
:


public

static

void

main
(
String argv
[])

throws

Exception
{




//Configure and load the invocation grid


InvocationGrid grid
=

HServerJob
.
getInvocationGridBuilder
(
"myGrid"
).



//

Add JAR files as IG dependencies


addJar
(
"main
-
job.jar"
).



addJar
(
"first
-
library.jar"
).



addJar
(
"second
-
library.jar"
).



//

Add classes as IG dependencies


addClass
(
My
Mapper
.
class
).


addClass
(
My
Reducer
.
class
).



//

Define custom JVM parameters


setJVMParameters
(
"
-
Xms512M
-
Xmx1024M"
).



load
();

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
7




//Run 10 jobs on the same invocation grid


for
(
int

i
=
0
;

i
<
10
;

i
++)


{


Configuration conf
=

new

Configuration
();




//The preloaded invocation grid is passed as the parameter to the job


Job job
=

new

HServerJob
(
conf
,

"Job number "
+
i
,

false
,

grid
);




//
......
Configure the job

here
.........




//Run the job


job
.
waitForCompletion
(
true
);


}




//Unload the invocation grid when we are done


grid
.
unload
();

}



2.
6

Single result
optimization


It is often useful for a job to

produce
a
single object as the result
, for example, when combining the
output of all mappers into a single output value
. To
accomplish this
,

the
map output key space
should
consist

of a single key
, the reducer input and output types should match (i.e.
,

the reducer

can
be used as a combiner),

and the reducer/combiner should produce no more than one key value pair
per call.

If these conditions are met,
the
output

of the job
is

a single object, which is the result of
combining all the values for
a
single
map output
key.

ScaleOut hServer
identifies and

optimizes this usage model
,
allowing

this type of job

to run
efficiently and without
the
need for
an
output for
mat.
Using the
runAndGetResult()

method of
HServerJob

to run the optimized job, t
he result object is returned directly to the application
.

The job
should include a combiner to run this
optimization
.

To illustrate
a
single result optimization,
the
W
ord
C
ount

example can be modified to count the
occurrences of
a

specific word:

public

class

SingleWordCount
{


private

final

static

String wordPropertyName
=

"com.scaleoutsoftware.soss.hserver.examples.lookupWord"
;



//The mapper is changed to emit value
s

only for matching words


public

static

class

TokenizerMapper


extends

Mapper
<
Object
,

Text
,

NullWritable
,

IntWritable
>

{



private

final

static

IntWritable one
=

new

IntWritable
(
1
);


private

String lookupWord
;



@Override


protected

void

setup
(
Context context
)

throws

IOException
,

InterruptedException
{


super
.
setup
(
context
);


String strings
[]

=

context
.
getConfiguration
().
getStrings
(
wordPropertyName
);


if

(
strings
.
length
==

1
)

{


lookupWord
=

strings
[
0
];


}

else

throw

new

IOException
(
"Word property is not set."
);

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
8




}



public

void

map
(
Object key
,

Text value
,


Context context
)

throws

IOException
,

InterruptedException
{



StringTokenizer itr
=

new

StringTokenizer
(
value
.
toString
());


while

(
itr
.
hasMoreTokens
())

{


if

(
itr
.
nextToken
().
equals
(
lookupWord
))

{


//Emit only for the matching words


context
.
write
(
NullWritable
.
get
(),

one
);


}


}


}


}



//The reducer is unchanged except for the key type


public

static

class

IntSumReducer


extends

Reducer
<
NullWritable
,

IntWritable
,

NullWritable
,

IntWritable
>

{




private

IntWritable result
=

new

IntWritable
();



@Override


public

void

reduce
(
NullWritable key
,

Iterable
<
IntWritable
>

values
,


Context context
)

throws

IOException
,

InterruptedException
{


int

sum
=

0
;


for

(
IntWritable val
:

values
)

{


sum
+=

val
.
get
();


}


result
.
set
(
sum
);


context
.
write
(
key
,

result
);


}


}



public

static

void

main
(
String
[]

args
)

throws

Exception
{


Configuration conf
=

new

Configuration
();


String
[]

otherArgs
=

new

GenericOptionsParser
(
conf
,

args
).
getRemainingArgs
();


if

(
otherArgs
.
length
!=

2
)

{


System
.
err
.
println
(
"Usage: singlewordcount <in> <word>"
);


System
.
exit
(
2
);


}


conf
.
setStrings
(
wordPropertyName
,

otherArgs
[
1
]);


HServerJob job
=

new

HServerJob
(
conf
,

"Single word count"
);



job
.
setJarByClass
(
SingleWordCount
.
class
);


job
.
setMapperClass
(
TokenizerMapper
.
class
);



//The map output types should be specified


job
.
setMapOutputKeyClass
(
NullWritable
.
class
);


job
.
setMapOutputValueClass
(
IntWritable
.
class
);



job
.
setCombinerClass
(
IntSumReducer
.
class
);


FileInputFormat
.
addInputPath
(
job
,

new

Path
(
otherArgs
[
0
]));



System
.
out
.
println
(
"
W
ord
\
""
+
otherArgs
[
1
]+
"
\
" was used "
+
job
.
runAndGetResult
()+
"
times."
);


}

}

3
.
Storing input/output data sets in
the IMDG


After you install ScaleOut h
Server

on the
servers

of your
cluster, it will automatically discover and
self
-
aggre
gate into an in
-
memory data grid

(IMDG)

spanning
the cluster. Using ScaleOut
State
Server’s Java A
PIs, your application can create, read, update, and delete key/value pairs i
n the
ScaleOut hServer™

V2

Java Programmer’s Guide


Page
9

IMDG to manage fast
-
changing
,

live


data
, keeping the data in
the grid up to date as changes occur
. At the same time, your
Hadoop MapReduce program can read a collection of key/v
alue
pairs from the IMDG

using

th
e

input formats

provided by
ScaleO
ut hServer
. Th
ese

input formats
,

which subclass
GridInputFormat
,

retrieve

key/val
ue pairs from the IMDG and
feed

them to Hadoop’s mappers with minimum latency.
Likewise, the output of Hadoop’s reducers optionally can be
stored
back into
ScaleOut
hServer’s IMDG using the

Grid
Output
Format

and its associated
Grid Record Writer
,
making these results available for subsequent Hadoop processing
without data leaving the IMDG. You also can output results to HDFS or another persistent store.

The

diagram below
illustrates
the use of the Grid Record Reader and Grid Record Writer.

The Grid
Record Reader is designed to input key/value pairs to Hadoop’s mappers with minimu
m latency.
The
Grid
Input
F
ormat
automatically creates splits
of the specified input key/value collection to
avoid network overhead when

retrieving key/
value pairs on all worker nodes
:



There are two ways to store objects in the IMDG,
through
a

NamedMap

or through
a
NamedCache
:




Named
C
ache

is optimized for bigger objects

(e.g., 10KB

or larger)

and has advanced
features, such as property
-
based
query
, dependencies,

and
pessimistic
locking
. The keys are
restricted to strings, UUIDs, and byte arrays. To construct the named cache
,

use
CacheFactory
.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
10



NamedMap

is a distributed

Java

C
oncurrentMap
optimized for storing small objects

efficiently.
A
named
map
supports arbitrary keys

and

is coherent across all clients connected
to the same IMDG
. To construct the named map
,

use
NamedMapFactory
.


3.1 NamedMap feature
overview

Using a

NamedMap

is the preferred way to store most MapRe
duce input and output data sets

because it provides efficient way
to store

large numbers of relatively small keys and values.

The
key features of
a

NamedMap

are:



Bulk operations.

To efficiently put or remove a
large number of keys, use a
BulkPut

view of the
map which can be obtained by calling the
getBulkPut(...)

method. This
combines

multiple
map operations
into chunks which provide

higher
overall bandwidth.
putAll(...)
also

will provide the
same performance gain if the keys and values are
pre
-
computed and put into an intermediate map.



Client cache

with coherency policy
.

A
customizable number of the recently read values
can be stored in memory in the client cache. On
subsequent reads, cached values
for a key are
returned if they are not older than the coherency
interval. A coherency interval of 0 means that
cached values are never used
,

and every read
requires

a call to the data grid.
The client c
ache

s

size can be configured
by

setClientCacheSize(...)
,

and
the
coher
ency interval
is
configured

by

setCoherencyIntervalMilliseconds(...)

.
By default, the client cache is turned off (coherency
interval is 0).



Parallel

method

invocation
.

Parallel

method

invocations

can

run simultaneously on all the
hosts in the data grid, with each host performing oper
ations on its local subset of keys. This
helps to avoid moving data across the network and provides the best performance. Parallel
invocations

are defined by subclassing

NamedMapInvokable
.
They require that an
invocation grid is assigned to the named map
by

setInvocationGrid(...)
.
3



Parallel q
uer
y
.

Parallel q
uery
returns a list of matching keys. To query a map, use

runQuery(...)

with a

QueryCondition

implementation as a parameter.

Section 3.6 contains
more information on queries and parallel
method invocations
.



Custom serialization.

Custom serialization can be used to efficiently store keys and values
in memory. Custom serializers, which implement

CustomSerializer
, should be provided to
the map factory method

NamedMapFactory.getMap(...)
. Each instance of the client
application across the grid should have the same custom serializers assigned to the map.




3

Parallel method

invocation and parallel query
re
quire a ScaleOut Analytics Server
®

license.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
11



Replication
.
A

NamedMap

can be configured to create replicas of its contents on multiple
hosts
. Enabling replication provides fault tolerance, i.e., it ensures that no data is lost in case
of host failure at the expense of increased memory and network utilization.
The number of
replicas is determined by the
max_r
eplicas

parameter

in the
soss_params.txt

parameters file
.
By default, replication is disabled.

Please refer to the ScaleOut StateServer Ja
va API documentation for
more

details.

3
.
2

The

Named
Map
InputFormat


The
NamedMapInputFormat

reads all the entries from the named map and sends the keys and
values it retrieve
s

to the mappers.

The input named map should be set as a configuration property by calling
setNamedMap(...)
. The
following example illustrates how to use
NamedMapInputFormat

and associate it with a named
map within the IMDG:


//
Create the named map

NamedMap
<
Integer
,

String
>

namedMap
=

NamedMapFactory
.
getMap
(
"myMap"
);

namedMap
.
clear
();



//
Insert entries

-

d
o a bulk put for better performance

BulkPut
<
Integer
,

String
>

bulkPut
=

namedMap
.
getBulkPut
();

for
(
int

i
=
0
;

i
<
1000000
;

i
++)

{


bulkPut
.
put
(
i
,

"initial.value"
);

}

bulkPut
.
close
();



//
Set up the job

Job job
=

new

HServerJob
(
new

Configuration
());

...

job
.
setInputFormatClass
(
NamedMapInputFormat
.
class
);

NamedMapInputFormat
.
setNamedMap
(
job
,

namedMap
);


3
.
3

The

NamedCache
InputFormat


The
NamedCacheInputFormat

reads all the keys and values from the named cache within the
IMDG

and sends them to the map
pers
. It assumes that all the objects in
the named

cache have the
same type. The key

that the mapper receive
s

for each object is its
CachedObjectId

identifier within
the
named cache
.

The key/
value pairs are served

to the mappers

in random order.



The

input
cache name should be set as
a
conf
iguration property by calling

setNamedCache(...)
.
The
following example illustrates
how to use the
NamedCacheInputFormat

and associat
e it with a
named cache within the IMDG:



//
Create the named cache and put some values in it


NamedCache cache
=

CacheFactory
.
getCache
(
"MyNamedCache"
);

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
12


cache
.
put
(
"key1"
,

"foo"
);


cache
.
put
(
"key2"
,

"bar"
);




//
Set up the job


Job job
=

new

HServerJob
(
new

Configuration
());


...


job
.
setInputFormatClass
(
NamedCache
InputFormat
.
class
);


NamedCache
InputFormat
.
setInputObjectClass
(
job
,

String
.
class
);


NamedCache
InputFormat
.
setNamedCache
(
job
,

"MyNamedCache"
);


3
.
4

The
Grid
Output
Format

The
Grid Record Writer, which is generated by the Grid Output Format (of type
GridOutputFormat
)
,

writes
key/value pairs

emitted by the reducers
to
a

named
map or a named
cache

within the IMDG
.
The g
rid output format does not preserve
the
order of
key/value
pairs. If
two values have t
he same key
, only one of them will be saved
.

ScaleOut hServer does not perform
sorting of the keys if the Grid Output format is used, because named map
s and
named cache
s

do

not
preserve ordering.

3
.
4
.
1

Using
a
NamedMap
for O
utput

To configure
the
Grid
Output
Format to use

a

named map for o
utput
,
the
named map should be
passed as a configuration property by calling
setNamedMap(…)
.
The following

example illustrates
how to set

up the grid output format and
associate

it with a named map in the IMDG:


NamedMap
<
IntWritable
,

Text
>

outputMap
=

NamedMapFactory
.
getMap
(
"myMap"
);

....

job
.
setOutputFormatClass
(
GridOutputFormat
.
class
);

GridOutputFormat
.
setNamedMap
(
job
,

outputMap
);


3
.
4
.2

Using

a

NamedCache
for O
utput

T
o configure

the

Grid
Output
Format

to use
a
named cache for
o
utput
,
the
cache name should be set
as
a
configuration property by calling
setNamedCache(...)
. The following

example illustrates
how
to set
up the grid output format and
associate
it with a named cache in the IMDG:


NamedCache writablecacheO
=

CacheFactory
.
getCache
(
"MyOutputCache"
);

....

job
.
setOutputFormatClass
(
GridOutputFormat
.
class
);

GridOutputFormat
.
setNamedCache
(
job
,

"MyOutputCache"
);



If
a

named cache is used for output, t
he reducer
’s

output key should be one of the following:
Text
,
String
,
CachedObjectId
,
UUID
or
byte[]
. Values should implement
Writable

or
Serializable
. If the
values are
Writable
,
a
custom serializer

should be set for the named cache before accessing the data
set through
the
named cache
’s access

methods (
see
section
3.5
).

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
13

3
.
5

Using the IMDG’s
NamedMap/
NamedCache

with Writables

By default, it is assumed that the objects stored in the
named map or
the
n
amed

cache

are serialized
and deserialized using
the
standard Java serialization framework. To use other serialization
frameworks, a custom serializer
must

be
provided to each instance of the ScaleOut StateServer
named cache. ScaleOut h
Server includes
a
custom serializer

of type
WritableSerializer

for Hadoop
Writable
types, so
that these objects can

be conveniently stored and retrieved from the
IMDG
.
The
WritableSerializer
takes the
Writable
-
implementing type of objects it
serializes and
deserialize
s

as a

constructor parameter.

To assign
a
custom
serializer

to the named map, use
the

factory method which takes custom
serializers for key and value:

NamedMap
<
Text
,

IntWritable
>

map
=

NamedMapFactory
.
getMap
(
"myMapW"
,



new

WritableSerializer
<
Text
>(
Text
.
class
),



new

WritableSerializer
<
IntWritable
>(
IntWritable
.
class
));

map
.
put
(
new

Text
(
"myText"
),

new

IntWritable
(
1
));



To
install

a

custom serializer
for a

named cache,
it

should be passed to the named cache
instance
by
calling

setCustomSerialization()

:


// Construct the named cache and

set

custom serialization


NamedCache cache
=

CacheFactory
.
getCache
(
"cache"
);

cache
.
setCustomSerialization
(

new

WritableSerializer
(
LongWritable
.
class
));


//

Put

some numbers in the named cache as objects
:

LongWritable number
=

new

LongWritable
();

for

(
long

i
=

0
;

i
<

NUMBER_OF_OBJECTS
;

i
++)

{


number
.
set
(
i
);


cache
.
put
(
""

+

i
,

number
);

}


// Retrieve

them back
:

for

(
long

i
=

0
;

i
<

NUMBER_OF_OBJECTS
;

i
++)

{


number
=

((
LongWritable
)

cache
.
get
(
""

+

i
));


if

(
i
!=

number
.
get
())

System
.
out
.
println
(
"Objects do not match."
);

}

3
.
6

Parallel
Method Invocations

and
Q
ueries
on the NamedMap

The
NamedMap
supports
parallel
method invocations

and
queries
which are concurrently executed
across all nodes in the IMDG,
with
each node working on its local set of keys and values.
Please
refer to the
NamedMap

documentation for more details.

P
arallel
met
hod invocations

are

run on the map contents to produce a single resul
t

object
.
They

have two phases:
eval(
),

which
returns a
result
for
each key in the map and
merge()
,

which performs
a
pairwise merge of the results.
Parallel operation
s

can have a parameter object,
which is
passed as
a parameter to the
eval()

and
merge()

methods
when
they
are invoked by the worker nodes.

These

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
14

parallel operations can be treated as a lightweight alternative to the MapReduce job performed on a
NamedMap
.
Parallel
method invoc
ations

are defined by subclassing
NamedMapInvokable

and
passing the implemen
tation, parameter object, and optional timeout
to

NamedMap.
invok
e
(…)
.

Queries
filter keys in the
NamedMap

based on the provided condition

and return a collection of the
matching keys
.

Queries are run by providing an implementation of the
QueryCondition
interface

to
NamedMap.runQuery()
.
The sample program in the next section contains
an
example of query
usage.

P
arallel operation
s

and queries require
an
IG assigned to the NamedMap. If
an
IG was
previously
constructed to run
a
ScaleOut hServe
r job, it can be reused, as in
the following

example.

3
.
7

Sample Program
:
M
odified WordCount example

The following program
is a modified WordCount example which
illustrates
many of the approaches
outlined in previous sections:



It uses

the

NamedMap

as input and output for
a ScaleOut hServer job,



It implements a

custom
Writable

serialization
,

and



It p
erforms

a
parallel query of the output map to determine which words are use
d

more
frequently than
a
provided threshold.


package

com
.
scaleoutsoftware
.
soss
.
hserver
.
examples
;


import

com
.
scaleoutsoftware
.
soss
.
client
.
InvocationGrid
;

import

com
.
scaleoutsoftware
.
soss
.
client
.
InvokeException
;

import

com
.
scaleoutsoftware
.
soss
.
client
.
map
.
NamedMap
;

import

com
.
scaleoutsoftware
.
soss
.
client
.
map
.
NamedMapFactory
;

import

com
.
scaleoutsoftware
.
soss
.
client
.
map
.
QueryCondition
;

import

com
.
scaleoutsoftware
.
soss
.
hserver
.
GridOutputFormat
;

import

com
.
scaleoutsoftware
.
soss
.
hserver
.
HServerJob
;

import

com
.
scaleoutsoftware
.
soss
.
hserver
.
NamedMapInputFormat
;

import

com
.
scaleoutsoftware
.
soss
.
hserver
.
WritableSerializer
;

import

org
.
apache
.
hadoop
.
conf
.
Configuration
;

import

org
.
apache
.
hadoop
.
io
.
IntWritable
;

import

org
.
apache
.
hadoop
.
io
.
Text
;

import

org
.
apache
.
hadoop
.
mapreduce
.
Job
;

import

org
.
apache
.
hadoop
.
mapreduce
.
Mapper
;

import

org
.
apache
.
hadoop
.
mapreduce
.
Reducer
;

import

org
.
apache
.
hadoop
.
util
.
GenericOptionsParser
;


import

java
.
io
.
IOException
;

import

java
.
util
.
StringTokenizer
;



public

class

NamedMapWordCount

{


public

static

class

TokenizerMapper


extends

Mapper
<
Object
,

Text
,

Text
,

IntWritable
>

{



private

final

static

IntWritable

one

=

new

IntWritable
(
1
);


private

Text

word

=

new

Text
();



public

void

map
(
Object

key
,

Text

value
,

Context

context


)

throws

IOException
,

InterruptedException

{


StringTokenizer

itr

=

new

StringTokenizer
(
value
.
toString
());


while

(
itr
.
hasMoreTokens
())

{

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
15


word
.
set
(
itr
.
nextToken
());


context
.
write
(
word
,

one
);


}


}


}




public

static

class

IntSumReducer


extends

Reducer
<
Text
,

IntWritable
,

Text
,

IntWritable
>

{


private

IntWritable

result

=

new

IntWritable
();



public

void

reduce
(
Text

key
,

Iterable
<
IntWritable
>

values
,


Context

context


)

throws

IOException
,

InterruptedException

{


int

sum

=

0
;


for

(
IntWritable

val

:

values
)

{


sum

+=

val
.
get
();


}


result
.
set
(
sum
);


context
.
write
(
key
,

result
);


}


}



public

static

void

main
(
String
[]

args
)

throws

Exception

{


Configuration

conf

=

new

Configuration
();


String
[]

otherArgs

=

new

GenericOptionsParser
(
conf
,

args
).
getRemainingArgs
();


if

(
otherArgs
.
length

!=

3
)

{


System
.
err
.
println
(
"Usage: wordcount <input map> <output map> <threshold>"
);


System
.
exit
(
2
);


}



final

int

threshold

=

new

Integer
(
otherArgs
[
2
]);





//
Create named maps


NamedMap
<
IntWritable
,

Text
>

inputMap

=

NamedMapFactory
.
getMap
(
otherArgs
[
0
],


new

WritableSerializer
<
IntWritable
>(
IntWritable
.
class
),


new

WritableSerializer
<
Text
>(
Text
.
class
));



NamedMap
<
Text
,

IntWritable
>

outputMap

=

NamedMapFactory
.
getMap
(
otherArgs
[
1
],


new

WritableSerializer
<
Text
>(
Text
.
class
),


new

WritableSerializer
<
IntWritable
>(
IntWritable
.
class
));



o
utputMap
.clear();
//
clear output map



//Create the invocation grid


InvocationGrid

grid

=

HServerJob
.
getInvocationGridBuilder
(
"WordCountIG"
).


addJar
(
"
myjob
.jar"
).


load
();



//Create hServer job


Job

job

=

new

HServerJob
(
conf
,

"word count"
,

false
,

grid
);


job
.
setJarByClass
(
NamedMapWordCount
.
class
);


job
.
setMapperClass
(
TokenizerMapper
.
class
);


job
.
setCombinerClass
(
IntSumReducer
.
class
);


job
.
setReducerClass
(
IntSumReducer
.
class
);


job
.
setOutputKeyClass
(
Text
.
class
);


job
.
setOutputValueClass
(
IntWritable
.
class
);


job
.
setInputFormatClass
(
NamedMapInputFormat
.
class
);


job
.
setOutputFormatClass
(
GridOutputFormat
.
class
);



//Set named maps as input and output


NamedMapInputFormat
.
setNamedMap
(
job
,

inputMap
);


GridOutputFormat
.
setNamedMap
(
job
,

outputMap
);


ScaleOut hServer™

V2

Java Programmer’s Guide


Page
16


//Execute job


job
.
waitForCompletion
(
true
);



//Assign invocation grid to the map, so parallel operation can be performed


outputMap
.
setInvocationGrid
(
grid
);



//Run query to find words that are used
more than threshold frequency


Iterable
<
Text
>

words

=

outputMap
.
runQuery
(
new

UsageFrequencyCondition
(
threshold
));



//Unload the invocation grid


grid
.
unload
();



//Output resulting words and their frequencies


System
.
out
.
println
(
"Following words were used more than "

+

threshold

+

" times:"
);


for
(
Text

word

:

words
)


{


System
.
out
.
println
(
"
\
""
+
word
.
toString
()+
"
\
" was used "

+

outputMap
.
get
(
word
)

+

"
times."
);


}


}



//Implementation
of the query condition. Condition is true if


//the usage frequency exceeds threshold frequency


static

class

UsageFrequencyCondition

implements

QueryCondition
<
Text
,

IntWritable
>


{


private

int

frequency
;



UsageFrequencyCondition
(
int

frequency
)

{


this
.
frequency

=

frequency
;


}



@Override


public

boolean

check
(
Text

key
,

IntWritable

value
)

throws

InvokeException

{


return

value
.
get
()

>

frequency
;


}


}

}

4
.
Using ScaleOut hServer

as an HDFS Cache

ScaleOut hServer enables key/value pairs read from HDFS or another source to be cached within its
IMDG to reduce data access time on subsequent MapReduce runs with the same data set.

This is
accomplished

by wrapping the original input fo
rmat with
ScaleOut hServer’s

dataset input format
to create a Dataset Record R
eader.

If the specified data set is not already stored within the IMDG or
if it has been modified
, the Dataset Record Reader

intercepts
key/value pairs as they flow
in
to the
mapper and
caches
them within the IMDG
. On subsequent runs

in which the data set is available
within the IMDG, the
Dataset Record Reader

retrieves key/
value pairs
from the IMDG, bypassi
ng
the underlying record reader, and serves

them to the mapper
.

The
abo
ve

diagram conceptually
shows how th
e Dataset Record Reader is used to wrap the underlying record reader and integrate
with ScaleOut hServer’s IMDG.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
17

4
.1 Dataset Input Format

To enable
the Dataset Record Reader, the

original input
format
should be replaced
with the
Dataset Input F
ormat
of type
DatasetInputFormat.
The o
riginal input format is
then

passed
to the Dataset Input F
ormat

as a
configuration
property.

The f
ollowing example

illustrate
s

the necessary changes
in the program code

to configure the Dataset Input
F
ormat:



Original code:


job
.
setInputFormatClass
(
TextInputFormat
.
class
);


Modified code:


job
.
setInputFormatClass
(
DatasetInputFormat
.
class
);



DatasetInputFormat
.
setUnderlyingInputFormat
(
job
,

TextInputFormat
.
class
);



Note that i
n current version of ScaleOut h
Server
,

only file
-
based input formats which
subclass

FileInputFormat
are supported as underlying input formats.

4
.2
Handling Modifications to the Input Files

When splits are created by the
D
ataset
I
nput
F
ormat, the

modification times of the input file
(
s
)

are
compared to the modification times recorded in the
IMD
G
. If these times do not match, one of the
two
actions can be taken

based on the
EnableAppends

property, which can be set
with

DatasetInputFormat.setEnableAppends(...).

If this property is set to
the default value of
false
, the
cached data set within the IMDG

is deleted
,

and a new set of splits is calculated based on the new file. This set of spl
its is recorded in the IMDG

during
the subsequent job run.

If the file is deleted and replaced by another file, the property should
be
false

to avoid allowing the dataset input f
ormat to serve recorded splits from the old file.

If the

property is set to
true
, it is assumed that the file was

appended. The
dataset input f
ormat will
use the splits that were already recorded and add splits corresponding to the appended
portion of the
file by
read
ing from HDFS and recording the splits

on the next run.

4
.3

Dataset

Input

Format

O
ptimization for
Text
K
ey
/V
alue Pairs
of Known L
ength

If the key and value produced by underlying input format is a
Text
object, and the
re is known fixed
size for keys and
values
,

both server memory consumption and deserialization overhead

can be
optimized
by
storing
Text

contiguously in a buffer; this avoids

storing the length of each individual
object.

ScaleOut hServer™

V2

Java Programmer’s Guide


Page
18

To take advantage of this optimization
,

the object sizes in bytes should be set in the job
configuration by using
setTextKeyValueSize
(…)
method of the

DatasetInputFormat:


job
.
setInputFormatClass
(
DatasetInputFormat
.
class
);



DatasetInputFormat
.
setUnderlyingInputFormat
(
job
,

TextInputFormat
.
class
);


DatasetInputFormat
.
setTextKey
Value
Size
(
job
,

10
,

90
);


4
.
4

Memory Usage

Sufficient

physical
memory should be provisioned for
the IMDG to store the data s
et objects and
their replicas.
If the IMDG runs out of memory during the recording phase, the
affected

split
s will
not be recorded and instead
will be read from HDFS
using the underlying record reader
on the next
run. Although t
his behavior will impact access time,
it is recommended that the maximum memory
usage by
the IMDG

be

limited
by

setting

the
max_memory

parameter in the
soss_params.txt

file on
every server in the cluster. Setting t
his parameter
ensures that

adequate
physic
al

memory
is
provisioned f
or the Hadoop infrastructure
and MapR
educe jobs.

If additional
servers running ScaleOut hServer are added to the cluster
, the
cached

data sets
automatically will be rebalanced
across all servers
to take advantage of the larger
IMD
G
. This will
move some of the
cached

splits to
new servers for access by additional Hadoop mappers to increase
the overall throughput of the MapReduce run.

4
.5

Manag
ing Cached Data Sets in the IMD
G

The
soss

command line control program can be used to manage HDFS data sets cached within the
IMDG. To list the set of cached data sets, use the
soss show_ds

command:

%
soss show_ds

1 HDFS cached data sets found


Dataset ID: 1689276489
--

Name: com.scaleou
tsoftware.soss.hserver.

FileImageorg.apache.hadoop.mapreduce.lib.input.TextInputFormathdfs://10.0.4.2

7/tmp/in/part1hdfs://10.0.4.27/tmp/in/part2


To remove a cached data set, use the

soss remove
_ds

command and either specify the data
set identifier or specify “
all
” to remove all data sets:

%
soss
remove
_ds

all

Removing Dataset ID: 1689276489


4
.
6

Performance Optimizations in the Dataset Record Reader

T
he Dataset Record Reader is designed to store key/value pairs in the IMDG with minimum
network overhead and maximum storage efficiency.
By u
sing splits defined for the HDFS file, it
creates “chunks” of key/pairs in the IMDG using overlapped updates to the

IMDG while each
ScaleOut hServer™

V2

Java Programmer’s Guide


Page
19

HDFS record reader reads from the HDFS file and supplies key/value pairs to its mapper. These
chunks are stored as highly av
ailable objects within the IMDG.


Likewise, on subsequent Hadoop MapReduce runs in which key/value pairs are availab
le in the
IMDG, the
Dataset Record Reader bypasses the underlying HDFS record reader and supplies
key/value pairs from the IMDG. ScaleOut hServer uses the same set of splits to efficiently retrieve
the key/value chunks from the IMDG in an overlapped manne
r that minimizes latency. To
minimize network overhead, chunks are served from the ScaleOut hServer service process running
on the same Hadoop worke
r node as th
e requesting mapper.