Qualitative Evaluation - wsclients

shrubberystatuesqueData Management

Dec 1, 2012 (4 years and 10 months ago)

301 views






UNSW

S
1

2009

Research Project

V
ersion: draft

Evaluating Cloud
Application Development
and Runtime Platforms

SOC Teaching & Learning Materials Portal

Project Team:

Supervisors
-

A/Prof. Anna Liu (annaliu@cse)
and Dr. Helen Paik (hpaik@cse)

Students
:

Fei Teng (ften303@cse), Liang Zhao (lzha077@cse)
and


Xiaomin Wu (xmwu432@cse)

1


TABLE OF CONTENTS







2


1

INTRODUCTION




3







4


2

EVALUATION METHODOLO
GY

2.1

OVERVIEW

Cloud
platform
s are still evolving, but m
any companies
have already

join
ed

into this

competition
,
keen
ing
to promote
their own products to
be the first person to try the cloud tomato
. Since there is
no wide
-
accepted public standard
a
t the mom
ent
, it makes the ev
aluation difficult.

Three cloud platforms

are invoked in this evaluation
, and a
pplications running on
them

are built
from

various

languages and technologies.
All
premises of these states lead to severe difficulty of
cross platform comparison.
For the purpose of keeping
measurements

commonality
,

different kinds
of timestamp are used
during the evaluation.

To maximize the coverage of the evaluation, three scenarios are illustrated to help build test

case
s.



Client


Cloud Host
Evaluations
: A user
visit a web application on the cloud from an end user
application, the client response time would be the
user

s
first concern to the cloud.



Cloud Host


Cloud Database
Evaluations
:

A user may send/
receive
an article or a form to
/from

the cloud database thr
ough
the web application, the database processing time is
a main factor
taking

into consideration. Meanwhile, if thousands of users take the same action concurrently,
the performance of the cloud database would also be interesting.



Client


Cloud Database
Evaluations
:

For a large file
transferring
,
whether or not a user can
make a peer
-
to
-
peer connection between client and the cloud database without going via the
cloud host. And also the
performance

of this connection.


2.2

EVALUATION TERMINOLO
GY

The entire evaluation is mainly based on time that observed and measured
from
both client and
Figure

1
:

Time
-
relevant Terminologies

5


server side. Before taking further steps to introduce evaluation methodologies, it is
necessary

to
make a
clarification

on these time
-
relevant terminologies.

The

figure
above
shows a full round
-
trip
amount a client, a

cloud hosting
server

and a
cloud
database.



Client
R
esponse
T
ime
: it

is a sum of
the amount of
network round
-
trip time
,

between the client
and th
e cloud host
/database
,

and the amount of time
that required
processing

a

client

s
request

on the cloud host/database
.



Processing
T
ime
: it

is

amount of time the server need
ed to process a block of logic codes.
In
order to get an accurate processing time, a

t
imer is set to start and end right before and
after

the code block. If a database transaction is involved in the code block the processing time will
also include database operating time and
network round
-
trip time
between the cloud host
ing
server

and the
cloud
database
.



Database
Processing
Time:
I
t is practically impossible to measure
accurate time that a

cloud
database take
s

to
finish
a
database transaction
. Alternatively, the processing time

could be used
,
once the timer is set before and after database
API
methods
; w
hile for
measuring

database
processing time directly between the client and the

cloud

database

(
only
appl
ying

to Azure
Stor
age
,

Amazon SimpleDB and

Amazon

S3)
, the client
response time between

the
client
and

the database could be applied.




6


2.3

QUALITATIVE EVALUATI
ON METHODOLOGY






7


2.4

QUANTITATIVE EVALUAT
ION METHODOLOGY

2.4.1

OVERVIEW

Two types of test

strategie
s are adopted in the qualitative
evaluation
.



Stress Test

Strategy
:
T
o obtain architecture insights
.

All cloud host
relevant

test

case
s
adopt
this
test

strategy
.




Singleton
R
equest
T
est

Strategy
: To make
numerical

measurement
s

on
time and
throughput.

All cloud database
relevant

test

case
s implement this test
strategy
.

2.4.2

STRESS
T
EST

STRATEGY

The stress test
strategy
works

as above flow chart states
.

At the very beginning,
the n
umber of
round
s

and
initial
concurrent
threads

will be set
.

Then, a
ll

threads will be
initiate
d
at the
first

stage

before fir
ing
.

W
ithin e
ach

thread, three requests will be
launch
ed continuously

to maintain a
constant
contention

on

cloud hosts
.

For

e
ach

round, the number of concurrent
threads

will be
increase
d

by 200

to increase the stress onto
cloud hosts
.

A test case will be terminated when all
rounds are finished.

This test
strategy
is used for Client


Cloud Host
Evaluations
and Cloud Host


Cloud Database
Evaluations
.




Figure

2
: The Flow Cha
r
t of
the
Stress Test

8


2.4.3

SINGLETON
R
EQUEST
T
EST

STRAGTEGY

The flow chart

of
singleton

request test

strategy

is similar to the one of stress test

strategy
, but with
number of concurrent
threads

stays

at 1

constantly
. Therefore, the
number of rounds here indicates

the

total amount of requests will be sent to cloud. In order to avoid stress affection, all requests will
be sent one by one from client.

This test
strategy
is adopted in Cloud Host


Cloud Database
Evaluations
and Client


Cloud Database
Evaluations
.




Figure

3
: The Flow Chart of
the
S
ingleton
R
equest
T
est

9


Figure

4
: Windows Azure Cloud

2.5

EVALUATION APPLICATI
ON ARCHITECTURE

2.5.1

APPLICATION ARCHITEC
TURE ON CLOUD PLATFO
RM
S

2.5.1.1

WINDOWS AZURE CLOUD

This diagram illustrates the

web
application

programming model
use
d

for

Microsoft Windows Azure
cloud.
Windows Azure provides a Windows
-
based environment for running applications and storing
data on servers in Microsoft data centres in distributed manner. Currently, all sever applications
running on the cloud are based on Mic
rosoft .Net Framework 3.5 SP1

and PHP
. Amount various
models in the Framework,
W
indows
C
ommunication
F
oundation

work
s

as web role to
adopt

server
-
side codes in C#
,
provid
ing

web service
.

By invoking

Azure SDK, W
indows
C
ommunication
F
oundation

communicate
s
with multiple
Azure Data Storage
s,

which are sitting on the cloud via
RESTful protocol.

Microsoft also provide
d

another kind of
cloud storage service,
named

SQL Data Servic
e.

B
ut the
protocol was later
deprecated
. Instead,

Microsoft announced
a
new protoco
l
, called

Tabular

Data
Stream
,

w
ould

be release in the future.

Since SQL Data Service is deprecated in the middle of the
evaluation, and
development

documents about
Tabular

Data Stream

are still
incomplete
, the
evaluation of cloud database on Microsoft Azure Cloud mainly focuses on
Azure Data Storage
s.



10


2.5.1.2

AMAZON ELASTIC COMPU
TE CLOUD

Amazon EC2 is a highly configurable cloud. Developers are flexible to setup software and applications
that
they are in favour on any Amazon Instances. For the instance used in the evaluation,

Tomcat is
setup
as
S
ervlet running on top of Java 1.6 SDK. Third

party framework Apache

CXF is used to provide
SOAP protocol. By using JPA1.0,

Apache

CXF
is enabled to

communicate with Postgre
SQL,

which is
installed on the same instance machine as where web
application

is hosted.
With

Amazon API,
Apache
CXF
is allowed
to

invoke

Amazon

s
cloud database
s,

Amazon
Sim
p
leDB and Amazon S3 via
SOAP or RESTful protocol.




Figure

5
: Amazon Instance

11


2.5.1.3

APP ENGINE CLOUD

Currently
, there are two
kinds of
programming languages available on Go
ogle App Engine,
Python
and J
ava
V
irtual
M
achine supported languages
.
JVM
support is newly released on April 7, 2009, still
being under development and only for an early look
ing
.
C
omparatively
,
Python

support

is original
ly

delivered
since the first

release

of Google App En
gine. It is
much more stable

in practise

at the
moment
.

Therefore, in the evaluation, Python and its
third

party frameworks, ZSI and Zope I
nterface,
offering SOAP protocol supports in Python, are selected to develop and deploy the server
-
side codes
.

In add
ition
,
Google App Engine SDK is used
t
o make a connection
via inner protocol
between
server
and Google
stateful services,
App Engine D
atastore
, offer
ing

cloud database behind web
applications
,

and
App Engine

MEMCACHE
, providing
storage for

data caching.




Figure

6
: App Engine Cloud

12


2.5.2

APPLICATION
ARCHITECTURE ON CLIE
NT

2.5.2.1

APPLICATION MODEL ON

CLIENT

Th
e

diagram
above
illustrates the testing model
used

for cross
-
platform
evaluations

based on
C
ontract
-
F
irst Web
S
ervice
.

As mentioned above, three
platforms

offer
diverse

programming languages for web server
development. For

Microsoft Windows Azure
, it
accept
s

C# and PHP
; for

Google App Engine
, it allows
P
ython and
JVM
-
supported languages;
and
for
Amazon
, it

provide
s

a virtual machine

to
adopt
programming language
s based on user

s choice
.

For the purpose of keeping
as much commonality as possible

among three cloud platforms, the
C
ontract
-
F
irst Web
S
ervice is
dedicate
d

to
the evaluation.

As following guidelines:



A

WSD
L file
will

be built

first
.



T
hree
hosting servers

implement all function
s

defined in the WSDL

file
.



A unified client application is created from
the WSDL file
.

S
o
it

can communicate with different
platform
s

via the same protocol.




Figure

7
: Application Model on Client

13


2.5.2.2

APPLICATION
ARCHITECTURE ON CLIE
NT

2.5.2.2.1

C
ONTRACT
-
F
IRST WEB
S
ERVICE

BASED CLIENT

The
application

model above mainly fits this application architecture.

2.5.2.2.2

RESTF
UL BASED CLIENT

This application architecture is used for Client


Cloud Database
evaluation
.
When doing the
PUT
/GET/DELETE

REST
f
ul action,
the
client
read
s

a

binary file from the local machine, and sends

them

directly to Azure Blob S
torage

or Amazon S3
.




Figure

8
: Contract
-
First Web Service Based Client

Figure

9
: Restful Based Client

14


3

QUALITATIVE EVALUATI
ON

3.1

SUMMARY OF EVALUATIO
N



3.2

STRESS
R
OUND
-
TRIP
T
EST





15


3.3

SINGLETON
D
ATABASE

W
RITE
/READ

T
EST

3.3.1

OVERVIEW

This is a
test case of
Cloud Host


Cloud Database
Evaluations,

based on
the singleton request test
strategy, following
a scenario that a

user sends/receives an article or a form to/from the cloud
database through the web application.
Base on the
scenario,

requests
of

various sizes
, simulating a
character (1 B), a message (100 B), an article (1
K
B) and a small file (1 MB),

will be sent one by one to
the
web
application

which host
ed

on the
cloud
.

The database
processing time will be measured on

the cloud

hosting server
, and sent back

to client.

For each test, the number of requests is fixed.

In terms of

specifications
, Amazon SimpleDB,
String Model

and

Text Model
in
App Engine Datastore
,

and Microsoft Windows Azure Table Storage are advertised to store structure data
.

W
hile Amazon S3,

Blob Model

in

App Engine Datastore and Microsoft Windows Azure Blob Storage are aim
ed

for
storing binary data.

In
the evaluation
,
request

which
is
no larger than

1KB will be stored into
structure data

oriented database,
and the one which is

larger than 1KB will be put into binary data
oriented database.


3.3.2

SINGLETON
D
ATABASE

WRITE

T
EST

16


O
n the view of average data processing time,
overall, each

singleton
data
processing time

for
writing
small requests (1 B, 100 B, and 1 KB) on cloud databases
varies in a small range
.

It suggests that the
size of small requests doesn

t affect too much on data processing tim
e on every cloud databases.

T
he
figure

also
state
s

that
Amazon
LocalDB
shows its strength
from

1

B
to

1

K
B
.

It is mainly due to
the stressless test
environment
, so that a local database without any
optimizing

can still
handle
requests

normal
ly
.

In
addition
, building the local database and the web application in the same
Amazon Instance might shorten the data processing time,
comparing to the time
uses by
other cloud
hosting servers and
corresponding

cloud databases, which may not sit in the same cloud, lead
ing to a
smallest time.

When the
size of request

goes to 1M, Amazon S3
almost

has
the same performance as App Engine
Datastore while Azure Blog Storage
takes less time than others
.

By diving request size by data processing time, the CDF of singlet
on write operation is presented
above.
It

reflects the different write speed
on each cloud database
for
different request sizes
.

Overall,
by increasing the size of the reque
st, the transfer speed is getting faster
progressively
.
It
indicates

that the connection between the cloud hosts and the cloud databases is fast and stable on
three cloud
platforms
.

Comparatively
,

Amazon SimpleDB has the slowest
speed,

which is worse than
App Engine Datastore and Azure Table Storage

In the first three tests

(1 B, 100 B, and 1 KB),
App Engine
D
atastore, Azure Table Storage, Amazon
LocalDB and Amazon SimpleDB

are
conducted. The order is quite stable in which Amazon LocalDB
performs
much
faster

than others
.

As for the 1

M
B

test, three
cloud

platforms perform similar
ly
. Approximately 80% of their requests


speeds almost approach

10

MB

per Second
.


Figure

10
: Average Singleton Writ
ing

Time on Cloud Databases

Figure

11
: CDF of
Singleton

Write

Throughput on
Cloud Databases

17




18


3.3.3

SINGLETON
D
ATABASE

R
EAD
T
EST

T
h
ese two
diagram
s

indicate

data processing time
of
reading requests
and CDF of read throughput
on
different
cloud databases
.

Figure

13
:
Average Singleton Reading Time on Cloud Databases

Figure

12
:
CDF of Singleton
Read

Throughput on Cloud Databases

19


On
e

interesti
ng point could be drawn is that comparing with singleton database write test, the
data
processing

time for all cloud platforms decreases dramatically except for Amazon S3
,

which
takes
longer time
than
it is in singleton writing test
.

Another
observation

is that Amazon SimpleDB changes
its position

from the last one to second last
.
Azure Table Storage performs better than Amazon SimpleDB in write, but worse in read
.


3.4

STRESS
D
ATABASE
W
RITE/
R
EAD
T
EST

3.4.1

OVERVIEW

Based on the stress test strategy,
another case of
Cloud Host


Cloud Database
Evaluations

is
performed to simulate a
scenario

that

multiple users take the write/read action concurrently
. In this
test case, the number of requests varies, while the size of each request is fixed. Due to Googl
e App
Engine

s free incoming
bandwidth

limitation, at maximum 56 MB per Minute, the number of
concurrent requests in each round varies from 300 to 3300, and the size of the request is set to 1 KB.

In the stress database test, not only error rate tables, error detail lists, and a CDF of throughput will
be established, but also
high network fluctuations
on the test
over time

will be observed, by
setting
c
ron job
to
schedule

repeat
edly

tests over 24 ho
urs.

3.4.2

ERROR DETAIL LIST

A variety of errors
occurred

during the stress database test. In terms of
phase
s where errors are
thrown
,

all
faults are categorized
into three categories
.



Connection Error
:
Errors are

encounter
ed

if
request
s

do not reach
cloud host
s
,
due to network
connection

problems
, such as packages lose, proxy gateway temporary unavailable.




Server Error
:
F
ault
s
occur within cloud hosting servers
,
for instance,
web application are
not
able to
allocate

resource
s

to more

requests
.



Database Error
: Errors come from cloud database
in the
period

of data processing time.


E
rror
details in each category are listed as following.

Error
Category

Error Messages

Reason

Happened on

Database
Error
s

datastore_errors:

Timeout

Multiple action perform at the
same entry, one will be p
rocessed
others will be failed

Request take
s

too much time to
process

Google App Engine
Datastore

datastore_errors:

TransactionFailedError

An error occurred for the API
request datastore_v3.RunQuery()

Google App Engine
Datastore

apiproxy_errors:

Error

T
oo much contention on these
datastore entities

Google App Engine
Datastore

20


Amazon SimpleDB is
currently unavailable

Too many concurrent requests

Amazon SimpleDB

Server Error
s

Unable to read data
from the transport
connection

WCF failed to open connection

Microsoft Windows
Azure

500 Server Error

HTTP 500 ERROR : Internal Error

Google App Engine

Zero Sized Reply


Amazon

Connection
Error
s

Read timed out

HTTP time out

Microsoft Windows
Azure

Amazon

Access Denied

HTTP 401
ERROR

Microsoft Windows
Azure

Google App Engine

Amazon

Too many open files

Java IO exception, due to machine
limit, too many concurrent
requests (too many threads) have
been launched

Microsoft Windows
Azure

Amazon

Network Error
(tcp_error)

Local proxy
connection error

Microsoft Windows
Azure

Google App Engine

Unknown Host
Exception


Microsoft Windows
Azure


3.4.3

OVERALL ERROR PRECEN
TAGES AMOUNT REQUEST
S

O
n each client application,
the n
umber of round
s

is set to 6 and the initial

concurrent
threads are
100
, according to the
stress test strategy
. Therefore, all
six rounds
will start
1
00,
3
00, 500,
7
00,
9
00
and
11
00 concurrent thread
s
gradual
ly, with three
continuous

requests

in each thread.

The overall
requests sent will be 10800

for each client application
.

To maximize the stress from the client side,
three

computers are deployed to run

client applications
collectively

at the scheduled

time. In all, 32400 requests in each scheduled test.

21


Th
e overall
writing
error percentages

chart
is generated from the average results
of retained
data

from scheduled tests over 24 hours.

Th
e

chart illustrates the
cloud
host and
cloud
database
performance from the aspect

of
error
rat
es
.

Considering the large amount of requests sent from three client applications,
although App Engine
Datastore and Amazon SimpleDB threw
average

31.67 and 111.17 faults in each scheduled test,
separately
, the overall performance of
all cloud databases
are

still acceptable, keeping the database
correct rates at a high level. Even the
worst

one, still makes the
rate more

than 99.67%.

Among all cloud platforms, Google App Engine drops the mo
st number of connection errors,
containing

500 Server Error


messages. The

For both
cloud
host

and
cloud
database performance,
Azure

is more stable t
han the other two cloud
platforms
.

From the database performance aspect,
the local database

is stable as

well. It is no doubt that

traditional
database is

still
compe
titive
in the contemporary time and stage.

Figure

14
: Overall Error
Percentages

amount Writing Requests

22


For
t
h
e overall reading error percentages

chart
, both Amazon SimpleDB and Azure Table Storage
performance are slightly better
than App Engine Datastore.


From the database performance aspect, LocalDB is stable as well, we can see that traditional hosting
still compatible in the contemporary time and stage.





3.4.4

CDF OF
STRESS

WRITE/
READ THROUGHPUT ON C
LOUD
DATABASES

Figure

15
:
Overall Error Percentages amount
Read
ing Requests

23


By picking the third round




3.5

SINGLETON
L
ARGE
F
ILE
W
RITE/
R
EAD/
D
ELETE
T
EST










Figure

16
:
CDF of
Stress

Write/
Read Throughput on Cloud Databases

24


4

QUANTITATIVE EVALUAT
ION






25


5

PLATFORM OVERVIEWS