Experience Predicting Application Server Memory Usage

watermelonroachdaleInternet και Εφαρμογές Web

30 Ιουλ 2012 (πριν από 5 χρόνια και 22 μέρες)

303 εμφανίσεις

Experience Predicting

Application Server Memory Usage


Thomas Reidemeister

Mohammad A. Munawar

Paul A.S.Ward

University of Waterloo

April 2005

2

Outline


Introduction


Difficulty of predicting in dynamic systems



Case Study: Java
-
based Application server


A simple approach to predicting memory
-
usage


Some experiments


Work in progress

3

Introduction


Need for monitoring system health


Performance


Security


Availability


Correctness,
etc


Sources of information


Events (logs, notifications,
etc
)


System status (resources, parameters,
etc)



Purpose


Anomaly detection and diagnosis


Dynamic reconfiguration


Planning

4

Building a Model

Workload
Generator

System Under
Observation


Model
Builder

Response

Monitor


Model

5

Using the Model

System Under
Operation


System
Model

Response

Monitor

Problem
Determination
Tool

Workload

6

Dynamic Environments


Changing nature


System configurations can change


System components (software/hardware) can be
updated/added/removed


Examples


Operating systems, Applications using Virtual Machines


Java
-
based applications



Java Virtual Machine (JVM)


Automatic memory management


Dynamic class
-
loading


Just
-
in
-
time compilation


Online byte
-
code instrumentation,
etc

7

Uncertainty


Why is prediction is difficult?


Beside lack of knowledge of the future, we have to
deal with:


Availability of information


Quality of available information


Performance impact


Dynamic environments (adaptive behaviour)


Complexity of the underlying process and dependencies

8

Case Study


Predicting memory
-
usage of a Java
-
based application server


Memory leaks and resource exhaustion is a severe problem


Automatic memory management makes it interesting


Prediction can help with


Load
-
balancing


Capacity planning


Possibly leak
-
detection


What can we achieve with the simplest approach and
information that is readily available?

9

Approach

Create
test-suites of
most common
requests
Execute
requests
sequentially &
repeat
Build an
approximate
memory profile
per request-type
Monitor actual
requests
received per
interval
Use profile to
estimate
memory-usage
Project number
of requests of
each type for the
next interval
PROFILING STAGE
PREDICTION STAGE
10

Setup


Software:


IBM WebSphere Application Server 5.1 running
Trade3


IBM DB2 UDB 8.1


‘httperf’ and httpunit
-
based custom workload generators


Hardware


P4 2.0 GHz, 1024MB machine with Windows Server 2003
running WebSphere


P4 2.0 GHz, 512MB machine with Redhat Linux running DB2


Dual
-
processor Xeon 2.8GHz with Redhat Linux for load
generation


Connected with a Gbit
-
network

11

Profiling


Obtain low
-
level memory allocation information from the
JVM


Use timing of request execution to match associated
memory allocation


Execute requests sequentially and at reasonably distant time
-
points to better isolate their effects


Repeat requests a sufficient number of times and with
different parameter values to obtain good representative
statistics


Build histograms for each request
-
type and select mode as
the representative value

12

/trade/portfolio.jsp
0
50
100
150
200
250
300
350
Memory allocated
Frequency
Profiling (cont.)


Examples

0
5000
10000
15000
0
20
40
60
80
100
120
Time (ms)
Memory (bytes)
Memory bytes allocated by an idle server

Histogram of a request
-
type

13

Profiling (cont.)


Issues


Typically, only millisecond time
-
resolution is available.
How much memory does a sub
-
millisecond request use?
When requests overlap in a millisecond
-
interval, we
apportion equally


Should we isolate unusual, irregular memory
-
allocation
due to internal house
-
keeping?


Profiles can change. A search request may require more
memory depending how many add/updates have taken
place since the last profiling time


Profiling requires a dedicated and isolated server

14

Monitoring


Precise memory
-
usage information is not readily available


For example, Websphere interface to query memory
-
usage
has a KB resolution.


Reported memory
-
usage usually includes objects no more
referenced and waiting to be freed


Relatively precise information can be obtained from the
JVM, if we profile GC events. However, the profiling
interface does not provide a complete picture


Let us monitor at regular interval, but at some reasonable
granularity. We thus have an
approximate actual usage
.

15

Monitoring (cont.)

Memory
Usage
Time
Monitoring
Interval
M
e
m
o
r
y

u
s
e
d

i
n
t
h
e

i
n
t
e
r
v
a
l
Actual
Observed
16

Prediction


Predict number of requests of each type that we expect to
see


Use the above together with the profiles created earlier to
estimate memory
-
usage


Any technique to project future requests can be used.
e.g.

simple Exponential Smoothing


What prediction period to use?


Fixed
-
duration: memory usage information is approximate


Variable
-
duration (
e.g
. time to next GC event): need for estimating
interval length

17

Prediction (cont.)


Experiments


How close do we get to actual memory usage, if we
knew exactly the number of each request
-
type received
during an interval?


Let us use variable
-
duration intervals, which correspond
to GC events
observed
to estimate actual memory
-
usage


The prediction model is instantiated with request counts
and request
-
type profile (mode
-
values)


No model fine
-
tuning

18

Prediction (cont.)

0
10000
20000
30000
40000
50000
60000
70000
80000
20
41
73
102
132
169
226
252
277
307
329
369
419
465
501
539
Time (s)
Memory-usage (KB)
Estimated Actual
Predicted
19

Results


We slightly under
-
estimate in general, but closely
follow the trend when a sufficient number of
requests are received during the prediction interval


With some calibration, we should be able to achieve
better results


Caching is an important factor that needs to be
incorporated in our prediction model

20

Improving Prediction


Build better profiles


Isolate object allocation due to server house
-
keeping


Deal with intervals with small number of requests


Use a different model?


Take caching into account


Model effect of present memory allocations on future
requests


21

Fine
-
grained Monitoring


Our approach can be used for coarse
-
grained monitoring


What if we wanted closer monitoring of the absolute
memory used at any time?


Can we develop a general technique that can, for example,
help detect memory
-
leaks?


Again, we would need to model the behaviour of the JVM in
details


E.g.

take into account the effect of requests, occurrence of GC
events, caching,
etc


Exploring Dynamic Bayesian Networks

22

Summary


Described a simple approach to predicting memory
-
usage of
an application server


Mentioned difficulties and issues


Showed that this approach is able to closely follow memory
-
usage trend


Depending on purpose, such simple prediction models can
be very effective


Prediction in dynamic systems is tricky and require that we
resort to approximations


But, we need to be intelligent about how to approximate

23

Our Research


Resource monitoring for anomaly
-
detection in
dynamic environments


Associating status information (
e.g.

resource usage)
with discrete system events for problem detection
and diagnosis


Adaptive monitoring

24

End


Thank you.

Questions are welcome.