TOWARDS
AN
EVOLUTIONARY MODEL G
ENERATION
FOR ERP PERFORMANCE
SIMULATION
Daniel Tertilt
, Stefanie Leimeister
Fortiss
–
Research Institute a
t Technische Universit
ae
t M
ue
nchen
Guerickest
rasse 25, 80805 M
ue
nchen, Germany
Stephan Gradl, Manuel Mayer, Helmut
Krcmar
Chair for Information Systems, Technis
che Universitaet Muenchen
Boltzmannstraße 3,
85748
Garching, Germany
ABSTRACT
The performance of ERP systems is a critical success factor for the reliable operation of a business. A promising
approach to cope wi
th the complexity of nowadays' ERP systems and to predict their actual behavior is simulation.
Commercial ERP systems, however, only provide limited insight and thus several components have to be handled as
black boxes and require a modeling approach. In t
his paper we depict an approach to increase the accuracy of ERP
system performance simulation by using an evolutionary algorithm for modeling the black boxes performance behavior.
We can show that
evolutionary algorithms
are able to
generate performance mo
dels for ERP components based on
measured performance data
that describe the performance behavior of these components accurately. Furthermore we
point
out the characteristics of the
algorithm
, as well as its advantages and disadvantages, and give an outloo
k about the
future research.
KEYWOR
D
S
Performance m
odeling,
performance s
imulation,
ERP,
evolutionary a
lgorithm
1.
INTRODUCTION
The performance of an
enterprise resource planning
(
ERP
)
system is a business critical non

functional
requirement
(Schneider, 2006)
, and strongly dependent on the infrastructure it is hosted on. Bögelsack et al.
(2010)
showed that changes in the infrastructure c
an significantly influence the performance and
consequentially the usability of an ERP system.
Performance predictability for ERP systems is thus very
desirable,
as it allows the handling of performance problems before they occur
(Balsamo et al., 2004)
,
thereby
reducing the risk of infrastructure changes significantly.
At the same time performance prediction i
s
hard to achieve
,
due to the complexity of modern ERP
systems
(Anderson and Mißbach, 2005)
.
Often this complexity is
managed
by anal
yzing the internal structure
of the ERP system (white box approach), identifying the correlation between the internal components, and
performing performance prediction by simulation.
Simulation though brings some “smallest elements”,
components that cannot
be seen inside (e.g.
to protect
intellectual property),
and that have to be
handled as
black boxes.
The performance behavior of these black boxes is often modeled using simple mathematical
functions like the mean value on measured performance data (see e.
g.
(Woodside, 2002)
)
, though ignoring
the characteristics o
f the underlying infrastructure
.
As Noblet et al.
(2004)
state simulation results can only be valid and trusted if the simulation is as close to
reality as possible. Our aim is to optimize the simulation results by increasing the accuracy of the black box
performance models. For this, we
deve
lop
an approach to model the performance behavior of these black
boxes using an evolutionary algorithm
(Zitzler and Thiele, 1999)
performing a multi

objective optimization
(
Zitzler and Thiele, 1998)
on measured performance data. In contrast to exact mathematical or algori
thmic
modeling, the evolutionary approach promises usable approximations even of multidimensional models in an
acceptable time span
(Gwozdz and Szlachcic, 2009)
, providing the possibility to consider multiple factors for
the respons
e time prediction of a black box.
2.
RELATED WORK
The approach presented in this paper is intended to enhance the interface between the ERP performance
measurement and the ERP performance simulation.
The most related work is shown in table 1
, where we also
point out which subject the documents deal with.
The
two
documents that deal with modeling as well as
simulation describe a hybrid approach, as it is developed in this paper.
Table
1
.
Related Work
concerning performance measurement, modeli
ng and simulation
Document
Measurement
Modeling
Simulation
Bögelsack et al.
(2008)
Pllana et
al.
(2008)
No
No
No
Yes
Yes
Yes
Jehle
(2009)
Gradl et al.
(2009)
Rolia et al
.
(2009)
Kraft et al.
(2009)
Bögelsack et al.
(2010)
Yes
No
No
No
Yes
No
No
Yes
Yes
No
No
Yes
No
Yes
No
3.
ERP COMPONENT
PERFORMANCE MODEL
ING
3
.1
Structure of the Algorithm
Oriented at natural evolution the evolutionary algorithm sets up a population of concurring threads, each
trying to generate a model that best matches the given set of measured performance data of the ERP black
box component. A s
election of the fittest is done by competition, and evolution evolves by constantly passing
better models to threads that lost competitions, and by mutation of these models.
Competition in this context
means that the fitness values of two threads (defined
by the fitness function described later) are compared,
resulting in the thread with the better fitness value to win the competition.
In order to create the performance model, several items have to be introduced. First, a central component is
created at startup. This component m
anages the population of threads that generate the performance model by
creating them at startup and by selecting an available opponent whenever a thread is in time to compete
Create
Fetch
opponent
Central
Component
Report
Population
Compete
Fig.
1
.
Schematic representation of the functionality of the evolutionary algorithm.
against another thread.
Furthermore, the central component monitors the results
of each competition, to check
if the end criterion is reached.
The population size
is restricted by the
system
resources
the algorithm is executed on
. A higher
population will result in higher parallelization of the evolution.
Figure 1 is a schematic sketc
h of the
functionality of the algorithm.
3
.2
Model Representation
Each thread stores a representation of its actual model in memory. This model is represented by an object tree
that has method objects as nodes and fixed value or variable objects
as leafs.
Method objects represent a mathematical operator.
Mathematical operators can be binary operators like
addition, subtraction, multiplication, division and power
, but
also unary operators like sine
or cosin
e
. They
are nodes in the tree, as they require param
eters, either again method objects, or fixed value or variable
objects. Fixed value objects are leafs that are set to a fixed numeric value. Variable objects are
leafs
that are
set to the parameter values of the measured performance data used as basis for
the modeling on evaluation
time. They represent the variables in the model.
Figure 2
shows
the representation of a
n exemplary
model
(
)
to illustrate the
structure
of the model as an object tree
.
The nodes
add
and
div
are method objects, where
X1
,
X2
are leafs and variable objects, and
a
is also a leaf
and a fixed value object
(
a
represents a number in this example)
.
3
.3
Fitness Function
During the competition the models of the two opponent threads are evaluated. For every set of available
performance data the performance parameters (like number of
parallel
users, size
or type
of
request) are set
for the variable objects, and the modeled response time is calculated.
Based on the relative deviation for every given performance data entry, an error index
is calculated as
the sum of the deviation
between
every available measured
value
and
the corresponding value
calculated by
the mathematical model
, divided by the number of available performance data entries
. Formula
1
shows the
implemented directive to achieve
.
∑
.
(1)
In this form
ula,
n
is the number of available performance data
entries
,
the measured response time
for data entry i, and
the modeled response time for that data entry.
The thread with a lower error
index
wins the competition and passes its mode
l to the loser thread.
We chose this fitness function, as it describes the distance between the model an
d the measured values
accurately
, and it is eff
iciently
calculable.
The negative side of the function is that it allows big deviations in
some points, w
hen others are modeled very close to the measured values. The evaluation of other fitness
functions will be part of the algorithm optimization process.
add
div
X2
X1
a
Fig.
2
.
Example of the representation of a mathematical model as an object tree.
3
.4
Model Passing and Mutation
Beside the competition, inheritance is the second important factor of an
evolution. After a fitter thread has
been identified by competition, it has to pass its model to the loser thread in a way that the chance for
keeping the positive characteristics of the model is high, but at the same time there is a considerable chance
f
or optimizing the model by mutation.
Passing a model from the winner thread to the loser thread is done by deep cloning of the object tree
representing the model, and replacing the model stored by the loser.
After the model is passed, the loser
thread itse
lf mutates its new model. In
our approach with a given
chance
either a fixed value object,
a
variable object
or a method object
is chosen
for mutation. If a fixed value object is selected, a random value
is added to its value. The selection of a variable o
bjects
results in the allocation of
a random performance
parameter
to this
object
.
In the
case
that a method object is chosen
, the mathematical operator is set to a
randomly selected one
with the same number of parameters
.
In the latter case t
he parameters
of the method
stay
s
the same.
The optimal probabilities for mutating a fixed value,
variable
or method
object
will be analyzed by
experiments and future case studies.
By performing continuous optimization, there is a risk of getting stuck in a local opt
imum
(Rocha and
Neves, 1999)
. We mitigate this problem of lo
cal optima by re

bearing every thread that lost 10.000 times in a
row, resulting in a re

initialization of the model of this thread. The threads themselves keep track of the
number of failed competitions. When the limit is reached, they trigger re

initiali
zation, drop the existing
model, and create a new one from scratch. Even when optimization is advanced, “rebirth” opens a way for
leaving local optima for finding the global one.
3
.5
End Criterion
As the end criterion,
an error index limit has to be
defined
.
The error index is a
significant
indicator of a
model’s distance to the measured data, and a
s
it
is calculated
in every competition, the end criterion is
checked without additional effort.
Choosing the error index as end criterion though
, as menti
oned before,
involves the disadvantage of
allowing big deviations for some data entries, if the majority of the data entries are modeled very exactly.
This might lead to unacceptable prediction errors if the measured performance data is not equally distrib
uted
,
including the algorithm to stop on a model that is not usable
.
This disadvantage though is solved
by
improving the fitness function.
4.
PRELIMINARY RESULTS
A prototype of the evolutionary algorithm has been implemented, and applied to the measured data
of an
SAP benchmark
as sample data
(Jehle, 2009)
. A subset of the measured data was used for modeling, while
the rest served for validating the model.
First results show that the approach delivers usable result
s (average error less than
3%
) when the data
used for modeling was equally distributed, while the error becomes big if the data is unbalanced. Future
improvement like weighting the input data will be necessary to remove the requirement of equally distribut
ed
input data.
Another important factor for efficiency of the presented approach is the configuration of the evolutionary
algorithm, especially the mutation. First experiments
we conducted on the SAP benchmark data
proved that
the selection of a fixed valu
e or variable object in 95% of all mutations, and the selection of a method object
in 5% of all cases, results in a fast evolution with reliable convergence to an optimum.
For the relatively small set of input data with around 200 entries the prototype ret
urned a usable model
after around two to three minutes
when hosted on
two Intel Core2 Duo machines (1.6 and 3 GHz, both 4GB
RAM).
The scalability of the evolutionary algorithm itself will have to be tested in future case studies.
5.
CONCLUSION
The prototypica
l implementation of the evolutionary algorithm showed the
feasibility
of the depicted
approach. Further, the first case study pointed out the efficiency, but also difficulties of the evolutionary
algorithm.
As next steps, further, more complex case studie
s will be performed. For this, we will develop an
interface for integrating the generated models into a LQN simulation of a SAP system
(Gradl et al.
, 2009)
.
Executing the simulation with the traditional black box modeling and afterwards using the generated models
will deliver comparable results.
In parallel, the prototype will be extended and optimized. A literature review about the vehicle routing
problem, a field where evolutionary algorithms are applied since many years, revealed the complexity of
possible configurations and modifications of the algorithm. Using the LQN simulation, we will analyze
different
configurations and develop an optimal a
lgorithm for the field of ERP performance simulation.
REFERENCES
Anderson, G. W. and Mißbach, M., 2005.
Last

Testing und Performance

Tuning.
SAP Press, Bonn, Germany.
Balsamo, S. et al., 2004. Model

based performance prediction in softwar
e development: A survey.
IEEE Transactions on
Software Engineering,
Vol. 30, No. 5, pp. 295

310.
Bögelsack, A. et al. 2008. An Approach to Simulate Enterprise Resource Planning Systems.
In:
ULTES

NITSCHE, U.,
MOLDT, D. & AUGUSTO, J. C. (eds.)
6th Internati
onal Workshop on Modelling, Simulation, Verification and
Validation of Enterprise Information Systems, MSVVEIS

2008, In conjunction with ICEIS 2008.
Barcelona, Spain:
INSTICC PRESS.
Bögelsack, A. et al. 2010. Performance Overhead of Paravirtualization on a
n Exemplary ERP System.
12th International
Conference on Enterprise Information Systems.
Funchal, Madeira, Portugal.
Gradl, S. et al., 2009. Layered Queuing Networks for Simulating Enterprise Resource Planning Systems.
In:
MOLDT,
D., AUGUSTO, J. C. & ULTES

NITSCHE, U., eds. 7th International Workshop on Modelling, Simulation,
Verification and Validation of Enterprise Information Systems, MSVVEIS

2009, In conjunction with ICEIS 2009,
May 2009 2009 Milan, Italy. INSTICC PRESS, pp. 85

92.
Gwozdz, P. and Szlach
cic, E., 2009. An Adaptive Selection Evolutionary Algorithm for the Capacitated Vehicle Routing
Problem.
In:
Logistics and Industrial Informatics, 2009. LINDI 2009. 2nd International, 10

12 Sept. 2009 2009. pp.
1

6.
Jehle, H. 2009. Performance

Messung ein
es Portalsystems in virtualisierter Umgebung am Fallbeispiel SAP.
CVLBA
Workshop 2009. 3. Workshop des Centers for Very Large Business Applications (CVLBA).
Magdeburg, Deutschland:
Arndt, H.

K.; Krcmar, H.
Kraft, S. et al. 2009. Estimating service resource consumption from response time measurements.
Proceedings of the
Fourth International ICST Conference on Performance Evaluation Methodologies and Tools.
Pisa, Italy: ICST
(Institute for Computer Sciences, S
ocial

Informatics and Telecommunications Engineering).
Noblet, C. M. H. et al., 2004. Enabling UMTS end

to

end performance analysis.
In:
3G Mobile Communication
Technologies, 2004. 3G 2004. Fifth IEE International Conference on, 2004 2004. pp. 29

33.
Plla
na, S. et al., 2008. Hybrid Performance Modeling and Prediction of Large

Scale Computing Systems.
In:
Complex,
Intelligent and Software Intensive Systems, 2008. CISIS 2008. International Conference on, 4

7 March 2008 2008.
pp. 132

138.
Rocha, M. and Neves
, J. 1999. Preventing premature convergence to local optima in genetic algorithms via random
offspring generation.
Proceedings of the 12th international conference on Industrial and engineering applications of
artificial intelligence and expert systems: mu
ltiple approaches to intelligent systems.
Cairo, Eygpt: Springer

Verlag
New York, Inc.
Rolia, J. et al. 2009. Predictive modelling of SAP ERP applications: challenges and solutions.
Proceedings of the Fourth
International ICST Conference on Performance
Evaluation Methodologies and Tools.
Pisa, Italy: ICST (Institute for
Computer Sciences, Social

Informatics and Telecommunications Engineering).
Schneider, T., 2006.
SAP Performance Optimization Guide.
Galileo Press, Bonn, Boston.
Woodside, M. 2002. Tutoria
l Introduction to Layered Modeling of Software Performance. Available:
http://www.sce.carleton.ca/rads/lqns/lqn

documentation/tutorialg.pdf
.
Zitzler, E. and Thiele, L. 1998. Multiobjective Optimization Using Evolutionary Algorithms

A Comparative Case
Stu
dy.
In:
EIBEN, A. E., BÄCK, T., SCHOENAUER, M. & SCHWEFEL, H.

P. (eds.)
Parallel Problem Solving
from Nature

PPSN V, 5th International Conference, Amsterdam, The Netherlands, September 27

30, 1998,
Proceedings.
Springer.
Zitzler, E. and Thiele, L., 1999.
Multiobjective Evolutionary Algorithms: A Comparative Case Study and the Strength
Pareto Approach.
IEEE Transactions on Evolutionary Computation,
Vol. 3, No. 4, pp. 257

271.
Comments 0
Log in to post a comment