Qualitative Evaluation - wsclients

gasownerΔιαχείριση Δεδομένων

31 Ιαν 2013 (πριν από 4 χρόνια και 7 μήνες)

212 εμφανίσεις






UNSW

S
1

2009

Research Project

V
ersion: draft

Evaluating Cloud
Application Development
and Runtime Platforms

SOC Teaching & Learning Materials Portal

Project Team:

Supervisors
-

A/Prof. Anna Liu (annaliu@cse)
and Dr. Helen Paik (hpaik@cse)

Students
:

Fei Teng (ften303@cse), Liang Zhao (lzha077@cse)
and


Xiaomin Wu (xmwu432@cse)

1


TABLE OF CONTENTS







2


1

INTRODUCTION




3


2

EVALUATION METHODOLO
GY

2.1

OVERVIEW

Cloud
platform
s are still
in the stage of
evolving
, without any

wide
-
accepted public standard at the
moment

yet. But it cannot stop
companies

keen
ing
to promote
their
products to
be the first person
to try the cloud tomato
.

In this evaluation,

three big companies


cloud platforms

are
involved
.

They are Microsoft Windows
Azure, Amazon

Elastic Compute Cloud

and Google App Engine.
Since there is no wide
-
accepted
public standard at the moment
,
each platform comes with its own featured technologies
and
programming languages. It
lead
s

to severe difficulty of cross
ing

comparison

of platforms
.
For the
purpose of keeping measurements commonality,

timing,
throughput

and error rates

are the main
focus in
qualitative

evaluation
, while
user experience
s

are t
he key consideration

in
quantitative

evaluation
.


2.2

QUALITATIVE EVALUATI
ON METHODOLOGY

2.2.1

EVALUATION TERMINOLO
GY

2.2.1.1

TIMING

The entire
qualitative

evaluation is mainly based on
kinds of
timestamp
s

that observed and
measured
from client

and cloud hosting server

side. Before taking further steps to introduce
evaluation methodologies, it is necessary to make a clarification on these time
-
relevant
terminologies. The figure
below

shows a full round
-
trip amount a client, a cloud hosting server and a
cloud database.

Figure

1
:
Time
-
relevant Terminologies

4




Client Response Time: it is the amount of network round
-
trip time between the client and the
cloud host/database,
plus
the amount of time that required processing a clie
nt’s request on the
cloud host/database.



Processing Time: it is amount of time the server needed to process a block of logic codes. In
order to get an accurate processing time, a timer is set to start and end right before and after
the code block. If a dat
abase transaction is involved in the code block
,

the processing time will
also include database operating time and network round
-
trip time between the cloud hosting
server and the cloud database.



Database Processing Time: It is practically impossible to me
asure accurate time that a cloud
database takes to finish a database transaction. Alternatively,
for measuring database
processing time between the
cloud hosting server

and the cloud database
,

the processing time
could be used

as setting a
timer before and

after database API methods
.

W
hile for measuring
database processing time directly between the client and the cloud database (only applying to
Azure Storage

and Amazon S3), the client response time between the client and the database
could be applied.


2.2.1.2

REQ
UEST

To
easier
i
ndentify all requests sent, they are categorized into four types
according

to their response
results.



Incomplete

Request:
It is a type of requests that a client fails to send or receive.



Completed Request: It is a request that a client
sends

successfully and receives a response from
the cloud host/database at last.



Failed Request: It is a completed request
, but its response

contains
an error message.



Successful Request: It is a
fully completed request, with
out

any
errors.


2.2.2

TEST CASES

To
maximize the coverage of the evaluation, three scenarios are illustrated to help build test cases.



Client


Cloud Host Evaluations: A user visit
s

a web application on the clo
ud from an end user
application
.

T
he client response time would be the user’s
first concern to the cloud.



Cloud Host


Cloud Database Evaluations: A user may send/receive an article or a form to/from
the cloud database through the web application, the database processing time is a main factor
taking into consideration. Meanwhile, if

thousands of users take the same action concurrently,
the performance of the cloud database would also be interesting.



Client


Cloud Database Evaluations: For a large file transferring, whether or not a user can
make a peer
-
to
-
peer connection between cli
ent and the cloud database without going via the
cloud host. And also
,

the performance of this connection.

More details will be presented in the section of each test later.

5



2.2.3

TEST STRATEGY

In order to perform all test cases above, t
wo types of test
strategies are adopted in the qualitative
evaluation.



Stress Test Strategy: To obtain architecture insights. All cloud host relevant test cases adopt this
test strategy.



Singleton Request Test Strategy: To make numerical measurements on time and throughpu
t.
All cloud database relevant test cases implement this test strategy.


2.2.3.1

STRESS TEST STRATEGY

The stress test strategy works as above flow chart states. At the very beginning, the number

of
rounds and initial concurrent threads will be set. Then, all threads will be initiated at the first stage
before firing. Within each thread, three requests will be launched continuously to maintain a
constant contention on cloud hosts. For each round,
the number of concurrent threads will be
increased by 200 to increase the stress onto cloud hosts. A test case will be terminated when all
rounds are finished.

This test strategy is used for Client


Cloud Host Evaluations and Cloud Host


Cloud Database
E
valuations.



Figure

2
: The Flow Chart of the
Stress Test

6


2.2.3.2

SINGLETON REQUEST TE
ST STRAGTEGY

The flow chart of singleton request test strategy is similar to the one of stress test strategy, but with
number of concurrent th
reads stays at 1 constantly. Therefore, the number of rounds here indicates
the total amount of requests will be sent to cloud. In order to avoid stress affection, all requests will
be sent one by one from client.

This test strategy is adopted in Cloud Hos
t


Cloud Database Evaluations and Client


Cloud Database
Evaluations.




Figure

3
: The Flow Chart of
the
S
ingleton
R
equest
T
est

7


Figure

4
: Microsoft Windows Azure Cloud

2.3

EVALUATION APPLICATI
ON ARCHITECTURE

2.3.1

APPLICATION ARCHITEC
TURE ON CLOUD PLATFO
RMS

2.3.1.1

MICROSOFT WINDOWS AZ
URE CLOUD

This
diagram illustrates the web application programming model used for Microsoft Windows Azure
cloud. Windows Azure provides a Windows
-
based environment for running applications and storing
data on servers in Microsoft data centres in distributed manner. Curre
ntly, all sever applications
running on the cloud are based on Microsoft .Net Framework 3.5 SP1 and PHP. Amount various
models in the Framework, Windows Communication Foundation works as web role to adopt server
-
side codes in C#, providing web service. By
invoking Azure SDK, Windows Communication
Foundation communicates with multiple Azure Data Storages, which are sitting on the cloud via
RESTful protocol.

Microsoft also provided another kind of cloud storage service, named SQL Data Service. But the
protoco
l was later deprecated. Instead, Microsoft announced a new protocol, called Tabular Data
Stream, would be release in the future. Since SQL Data Service is deprecated in the middle of the
evaluation, and development documents about Tabular Data Stream are s
till incomplete, the
evaluation of cloud database on Microsoft Azure Cloud mainly focuses on Azure Data Storages.



8


2.3.1.2

AMAZON ELASTIC COMPU
TE CLOUD

Amazon EC2 is a highly configurable cloud. Developers are fl
exible to setup software and applications
that they are in favour on any Amazon Instances. For the instance used in the evaluation, Tomcat is
setup as Servlet running on top of Java 1.6 SDK. Third party framework Apache CXF is used to provide
SOAP protocol
. By using JPA1.0, Apache CXF is enabled to communicate with PostgreSQL, which is
installed on the same instance machine as where web application is hosted. With Amazon API,
Apache CXF is allowed to invoke Amazon’s cloud databases, Amazon SimpleDB and Amaz
on S3 via
SOAP or RESTful protocol.




Figure

5
: Amazon Instance

9


2.3.1.3

GOOGLE APP ENGINE CL
OUD

Currently, there are two kinds of programming languages available on Google App Engine, Python
and Java Virtual Machine supported
languages. JVM support is newly released on April 7, 2009, still
being under development and only for an early looking. Comparatively, Python support is originally
delivered since the first release of Google App Engine. It is much more stable in practise a
t the
moment. Therefore, in the evaluation, Python and its third party frameworks, ZSI and Zope Interface,
offering SOAP protocol supports in Python, are selected to develop and deploy the server
-
side codes.

In addition, Google App Engine SDK is used to ma
ke a connection via inner protocol between server
and Google stateful services, App Engine Datastore, offering cloud database behind web applications,
and App Engine MEMCACHE, providing
storage for

data caching.




Figure

6
: Google App Engine Cloud

10


2.3.2

APPLICATION ARCHITEC
TURE ON CLIENT

2.3.2.1

APPLICATION MODEL ON

CLIENT

The diagram above illustrates the testing model used for cross
-
platform evaluations based on
Contract
-
First Web Service.

As mentioned above, three platforms offer di
verse programming languages for web server
development. For Microsoft Windows Azure, it accepts C# and PHP; for Google App Engine, it allows
Python and JVM
-
supported languages; and for Amazon, it provides a virtual machine to adopt
programming languages ba
sed on user’s choice.

For the purpose of keeping as much commonality as possible among three cloud platforms, the
Contract
-
First Web Service is dedicated to the evaluation. As following guidelines:



A WSDL file will be built first.



Three hosting servers
implement all functions defined in the WSDL file.



A unified client application is created from the WSDL file. So it can communicate with different
platforms via the same protocol.




Figure

7
: Application Model on Client

11


2.3.2.2

APPLICATION ARCHITEC
TURE ON CLIENT

2.3.2.2.1

CONTRACT
-
FIRST WEB SERVICE BA
SED CLIE
NT

The application model above mainly fits this application architecture.

2.3.2.2.2

RESTFUL BASED CLIENT

This application architecture i
s used for Client


Cloud Database evaluation. When doing the
PUT/GET/DELETE RESTful action, the client reads a binary file from the local machine, and sends
them directly to Azure Blob Storage or Amazon S3.



Figure

8
: Contract
-
First Web Service Based Client

Figure

9
: Restful Based Client

12


2.4

QUANTITATIVE EVALUAT
ION METHODOLOGY





13


3

QUALITATIVE EVALUATI
ON

3.1

SUMMARY OF EVALUATIO
N



3.2

STRESS ROUND
-
TRIP TEST



Figure

10
: Average Client Response Time in Stress Round
-
trip Test

The three figures above indicate
change
s

of
average client response time when cloud hosting servers
are
under
variant

amount of stress, at
various

time and date


From figures, latencies are dramatically increased after 2100 concurrent request
s
.

T
wo statement
s

could be raised.

Firstly, i
t
could be
di
fficult for a limited number of computers to challenge the entire

0
2000
4000
6000
8000
10000
12000
300
900
1500
2100
2700
3300
0
2000
4000
6000
8000
10000
12000
14000
16000
300
900
1500
2100
2700
3300
0
2000
4000
6000
8000
10000
12000
300
900
1500
2100
2700
3300
Average Client Response Time
(Millionseconds)

Number of Concurrent Requests

Amazon EC2
Google App Engine
Microsoft Windows Azure
14


cloud hosting server
s
.

Secondly, e
ven
the latencies were

due to the burden
raised
on cloud hosting
server
s
,
a
quota of 2100 concurrent requests
is

efficient enough for nowadays enterprise
s
.

Taking

the
ticket booking system of
2008 Beijing Olympic
G
ame

as an example
,
it crashed when its burden
reached at

2200

Requests per Seconds.




15


3.3

SINGLETON DATABASE W
RITE
/READ

TEST

3.3.1

OVERVIEW

This is a
test case of
Cloud Host


Cloud Database
Evaluations,
based on
the singleton request test
strategy, following
a scenario that a user sends/receives an article or a form to/from the cloud
database through the web application. Base on the scenario, requests
of

various sizes
, simulating a
character (1 B), a mess
age (100 B), an article (1 KB) and a small file (1 MB),

will be sent one by one to
the
web
application
which host
ed

on the
cloud.

The database
processing time will be measured on

the cloud hosting server
, and sent back to client.

For each test, the number
of requests is fixed.

In terms of specifications
, Amazon SimpleDB,
String Model and Text Model in
App Engine Datastore
,

and Microsoft Windows Azure Table Storage are advertised to store structure data
.

W
hile Amazon S3,

Blob Model in

App Engine Datastore an
d Microsoft Windows Azure Blob Storage are aim
ed

for
storing binary data.
In
the evaluation
,
request

which
is
no larger than

1KB will be stored into
structure data

oriented database,
and the one which is

larger than 1KB will be put into binary data
oriented database.


3.3.2

SINGLETON DATABASE W
RITE TEST

16


O
n the view of average data processing time,
overall, each

singleton
data
processing time

for
writing
small requests (1 B, 100 B, and 1 KB) on cloud databases
varies in a small range
.

It suggests that the
size of small requests doesn’t affect too much on data processing tim
e on every cloud databases.

T
he
figure

also
state
s

that
Amazon
LocalDB
shows its strength
from

1

B
to

1

K
B
. It is mainly due to
the stressless test environment, so that a local database without any optimizing can still
handle
requests

normally
.

In addition, building the local database and the web application in

the same
Amazon Instance might shorten the data processing time,
comparing to the time
uses by
other cloud
hosting servers and corresponding cloud databases, which may not sit in the same cloud, leading to a
smallest time.

When the
size of request

goes to 1M, Amazon S3
almost

has
the same performance as App Engine
Datastore while Azure Blog Storage
takes less time than others
.

By diving request size by data pro
cessing time, the CDF of singlet
on write operation is presented
above. It

reflects the different write speed
on each cloud database
for
different request sizes
.

Overall,
by increasing the size of the request, the transfer speed is getting faster progressively.
It
indicates that the connection between the cloud hosts and the cloud databases is fast and stable on
three cloud platforms.

Comparatively,

Amazon SimpleDB has the
slowest
speed,

which is worse than
App Engine Datastore and Azure Table Storage

In the first three tests

(1 B, 100 B, and 1 KB),
App Engine
D
atastore, Azure Table Storage, Amazon
LocalDB and Amazon SimpleDB

are
conducted. The order is quite stable in which

Amazon LocalDB
performs
much
faster

than others
.

As for the 1

M
B

test, three
cloud

platforms perform similar
ly. Approximately 80% of their requests’
speeds almost approach

10

MB

per Second
.



Figure

11
: Average Singleton Writ
ing

Time on Cloud Databases

Figure

12
: CDF of
Singleton

Write

Throughput on Cloud Databases

17


3.3.3

SINGLETON DATABASE R
EAD TEST

T
h
ese two
diagram
s

indicate

data processing time
of
reading requests
and CDF of read throughput
on
different
cloud
databases
.

Figure

14
:
Average Singleton Reading Time on Cloud Databases

Figure

13
:
CDF of Singleton
Read

Throughput on Cloud Databases

18


On
e

interesting point could be drawn is that comparing with singleton database write test, the
data
processing

time for all cloud platforms decreases dramatically except for Amazon S3
,

which
takes
longer time
than
it is in singleton writing test
.

Another
observation

is that Amazon SimpleDB changes
its position

from the last one to second last
.
Azure Table Storage performs better than Amazon SimpleDB in write, but worse in read
.


3.4

STRESS DATABASE WRIT
E/READ TEST

3.4.1

OVERVIEW

Based on the stress test
strategy, another case of Cloud Host


Cloud Database Evaluations is
performed to simulate a scenario that multiple users take the write/read action concurrently. In this
test case, the number of requests varies, while the size of each request is fixed. Du
e to Google App
Engine’s free incoming bandwidth limitation, at maximum 56 MB per Minute, the number of
concurrent requests in each round varies from 300 to 3300, and the size of the request is set to 1 KB.

In the stress database test, not only error rate
tables, error detail lists, and a CDF of throughput will
be established, but also high network fluctuations on the test over time will be observed, by setting
cron job to schedule repeatedly tests over 24 hours.

3.4.2

ERROR DETAIL LIST

A variety of errors occurr
ed during the stress database test. In terms of phases where errors are
thrown, all faults are categorized into three categories
.



Connection Error:
Errors are

encountered

if
request
s

do not reach
cloud host
s, due to network
connection problems, such as pac
kages lose, proxy gateway temporary unavailable.




Server Error
:
F
ault
s occur within cloud hosting servers, for instance, web application are not
able to allocate resources to more requests.



Database Error
: Errors come from cloud database
in the period of
data processing time.


E
rror
details in each category are listed as following.

Error
Category

Error Messages

Reason

Happened on

Database
Error
s

datastore_errors:

Timeout

Multiple action perform at the
same entry, one will be processed
others will be
failed

Request takes too much time to
process

Google App Engine
Datastore

datastore_errors:

TransactionFailedError

An error occurred for the API
request datastore_v3.RunQuery()

Google App Engine
Datastore

apiproxy_errors:

Error

Too much contention on
these
datastore entities

Google App Engine
Datastore

19


Amazon SimpleDB is
currently unavailable

Too many concurrent requests

Amazon SimpleDB

Server Error
s

Unable to read data
from the transport
connection

WCF failed to open connection

Microsoft Windows
Azure

500 Server Error

HTTP 500 ERROR : Internal Error

Google App Engine

Zero Sized Reply


Amazon

EC2

Connection
Error
s

Read timed out

HTTP time out

Microsoft Windows
Azure

Amazon

EC2

Access Denied

HTTP 401 ERROR

Microsoft Windows
Azure

Google App
Engine

Amazon

EC2

Too many open files

Java IO exception, due to machine
limit, too many concurrent
requests (too many threads) have
been launched

Microsoft Windows
Azure

Amazon

EC2

Network Error
(tcp_error)

Local proxy connection error

Microsoft
Windows
Azure

Google App Engine

Unknown Host
Exception


Microsoft Windows
Azure


3.4.3

OVERALL ERROR PRECEN
TAGES AMOUNT REQUEST
S

O
n each client application,
the number of rounds is set to 6 and the initial concurrent threads are
100
, according to the stress
test strategy
. Therefore, all
six rounds
will start
1
00,
3
00, 500,
7
00,
9
00
and
11
00 concurrent thread
s gradual
ly, with three continuous requests in each thread.

The overall
requests sent will be 10800

for each client application
.

To maximize the stress fr
om the client side, three computers are deployed to run

client applications
collectively
at the scheduled

time. In all, 32400 requests in each scheduled test.

20


The overall
writing
error percentages chart is generated from the average results
of retained
data
from scheduled tests over 24 hours.

Th
e

chart illustrates the
cloud
host and
cloud
database
performance from the aspect

of error rates
.

Considering the large

amount of requests sent from three client applications,
although App Engine
Datastore and Amazon SimpleDB threw
average
31.67 and 111.17 faults in each scheduled test,
separately, the overall performance of
all cloud databases
are still acceptable, keepin
g the database
correct rates at a high level. Even the worst one, still makes the rate more than 99.67%

of successful
requests
.

Among all cloud platforms, Google App Engine drops the mo
st number of connection errors,
containing “500 Server Error” messages. The
largest server error rate and database error rate
happened on time after May 21 16:30 EST 2009, when was May 20 23:30 PST 2009 to Google App
Engine as well.
C
hecking its host and d
atabase status diagram on that day, there
were

some

large
latencies
on both host and database
,

around one or half hour earlier than the scheduled test.
It could
be a cause. But since it is hard to prove that earlier
latencies
affected the later test, it is

still

not
certain that the
latencies were

the main reason which led to high error rates.

Comparing to cloud platforms,
the local database
, Amazon LocalDB,

is
significant
stable
. It is no
doubt that

traditional
database is

still
competitive
in the contempo
rary time
.

Figure

15
: Overall Error
Percentages

amount Writing Requests

21


For
the overall reading error percentages chart,
the correct rates of cloud database and cloud host
are even higher, almost 99.99% of successful requests
.

It
is worth notifying that almost one third

failed connection requests
are happened
when client
applications
were

connect
ing

to Microsoft Windows Azure.

Most of errors were raised due to “Read
time out”
. It
occurs

9728.20 times on average in each scheduled test, varying
in a range of
8156
times to 12775 times.


One evident
can be observed
from client application log
s

is

that,
client
application
s

are

waiting
for

responses from the web application
,
but
nothing com
es

back,
leading to the
occurrence

of
time out
error.

Another evident can
be
read from all the test cases which were applied onto Google App Engine

is
that
, read time out
has

never been happened for once.

From these two
eviden
ces
,
it is certain

that,



Message
was sent out



Client testing application was doing its job correctly


Base on above,
two

assumption
s

could be made
.



C
loud hosting server is too busy, do not have enough resource to accept other
up
coming
requests.



Due to security concern, packages are dropp
ed

by
firewall

when there are too many concurrent
requests
.

Figure

16
:
Overall Error Percentages amount
Read
ing Requests

22




(Assumption of “Access Denied”)


From the database performance aspect, LocalDB is stable as
well,

we can see that traditional hosting
still compatible in the contemporary time and stage.





3.4.4

CDF

OF STRESS WRITE/READ

THROUGHPUT ON CLOUD
DATABASES

By picking the third round




3.5

SINGLETON LARGE FILE

WRITE/READ/DELETE TE
ST

Figure

17
:
CDF of
Stress

Write/
Read Throughput on Cloud Databases

23











24


4

QUANTITATIVE EVALUAT
ION






25


5

PLATFORM OVERVIEWS