A Combination of Parallel Computing and Object- Oriented Programming to Improve Optimizer Robustness and Efficiency

parentpitaΛογισμικό & κατασκευή λογ/κού

18 Νοε 2013 (πριν από 3 χρόνια και 10 μήνες)

97 εμφανίσεις

20
th
European Symposium on Computer Aided Process Engineering – ESCAPE20
S. Pierucci and G. Buzzi Ferraris (Editors)
© 2010 Elsevier B.V. All rights reserved.

A Combination of Parallel Computing and Object-
Oriented Programming to Improve Optimizer
Robustness and Efficiency
Guido Buzzi-Ferraris, Flavio Manenti
Dipartimento di Chimica, Materiali e Ingegneria Chimica “Giulio Natta”
Politecnico di Milano, Piazza Leonardo da Vinci 32, 20133 Milano, ITALY
E-mail: guido.buzziferraris@polimi.it
Abstract
This research activity is mainly aimed at showing potentialities in coupling object-
oriented programming with parallel computing. Wide margins of benefits could be
obtained in algorithm efficiency and robustness with a relative small programming
effort. The case of unconstrained multi-dimensional optimization is proposed as
quantitative example.

Keywords: Robust optimization, Multimodality, Narrow valleys, Discontinuous
functions, Parallel Computing.
1. Introduction
We are undergoing two silent revolutions that directly involve Process Systems
Engineering (PSE) and Computer-Aided Process Engineering (CAPE) communities,
besides many other scientific and industrial areas: the object-oriented programming and
the parallel computing on personal computer. Both these transformations have been
widely discussed in the literature as they significantly modify the numerical analysis as
it was conceived since the second part of the previous century and the way to apply
numerical methods and algorithms for solving more and more complex problems and
multifaceted issues.
Nevertheless, since it is not clearly stated in the current literature, it is worth remarking
that the parallel computing is easy to integrate in object-oriented programming and their
combination seems particularly appealing as many objects generated by the same class
might run simultaneously on different processors or cluster nodes. By thinking parallel
and object-oriented both together, it is possible to write by new many algorithms which
were not considered for solving numerical problems because of their reduced
performances in the procedural programming and sequential computing.
Thus, this research activity specifically deals with the development of very robust
optimizers that exploit:
• All features of object-oriented programming (Buzzi-Ferraris, 1994), which allow
going beyond the procedural programming and its limitations.
• The shared memory nowadays commonly available on multi-processor machines
(distributed memory machines are not considered for the time being, even though
the same reasoning here described can be extended to this branch of parallel
computing).
Motivation and practical interests in some scientific and industrial areas are briefly
reported in Paragraph 2. Basic concepts of coupling parallel computing with object-
G. Buzzi-Ferraris and F. Manenti
oriented programming for improving the optimizer robustness and efficiency are stated
in Paragraph 3. Some literature tests involving multidimensionality, strong and weak
multimodality, very narrow valleys, and functions that are undefined in some regions
are proposed in Paragraph 4.
2. Motivation and Practical Interests
Looking at the increasing spread of multi-processor machines as well as the larger and
larger amount of processors available on the common PCs, it is easy to see how the
period we are living is very similar to the one of 1970s when the most powerful (and
very large-size) machines were gradually replaced by smallest personal computers with
reduced computational power, but with a large impact on research activities for their
faster spread, reduced costs, reasonably good performances, and especially for their
higher slope in innovation. In our opinion, but it is easy to find out some confirmations
yet, shared memory machines will have a faster evolution than distributed memory
architectures and looking forward this, we preferred to use openMP directives rather
than MPI ones in starting exploiting parallel computing, at least for this preliminary
research activity.
In any case, the result is practically the same by using one or the other set of directives
as efficiency and robustness of many algorithms can be significantly improved so to
increase performance solution of a series of industrial issues and to allow moving from
reacting to predicting technologies applied to industrial plants (Manenti, 2009) and from
those solutions still performed off-line to their on-line application (White, 2001) by
preserving the robustness of the selected methods.
To quote some examples typical of process industry and PSE/CAPE communities,
improvements in optimizer efficiency and robustness can provide practical benefits in
data reconciliation (Arora and Biegler, 2001; Bagajewicz, 2003), in data regression
(Buzzi-Ferraris and Manenti, 2009a, 2009b, 2010b; Manenti and Buzzi-Ferraris, 2009),
in solving nonlinear algebraic or differential systems (Cuoci et al., 2007; Manenti et al.,
2009), or in the supply chain management optimization levels (Dones et al., 2009; Lima
et al., 2009; Manenti and Rovaglio, 2008).
3. Exploiting Shared Memory to Improve Efficiency and Robustness
Conventional programs easily fail when some specific families of optimization
problems have to be solved. Very robust optimizers are required in these cases:
• When the function is multimodal and the global optimum is required
• The function and/or its derivatives are discontinuous
• The function cannot be approximated by a quadric in correspondence with the
optimum
• Very narrow valleys (or steep walls) are present
• The function is undefined in some regions and the domain cannot be analytically
described
Although no one can ensure the global optimum is found, a robust algorithm should be
effective in tackling all previous situations.
Let us start analyzing the problem of very narrow valleys. From this perspective, the
OPTNOV’s method (Buzzi-Ferraris, 1967) seems one of the most appealing approach.
It is important to realize the reason that makes traditional methods such as Simplex
(Nelder and Mead, 1965), Hooke-Jeeves (Hooke and Jeeves, 1961), Rosembrock
(Rosenbrock, 1960), Quasi-Newton algorithms and so on ineffective when the function
valleys are particularly narrow. For example, Rosembrock’s method is based on the
A Combination of Parallel Computing and Object-Oriented Programming to Improve
Optimizer Robustness and Efficiency

rotation of axes of search so to follow the bottom of the valley. Since one of rotated
axes is adopted as search direction, it may occur that moving along it does not bring to
any function improvement when the valley is very narrow as shown in Figure 1.

x
1
x
2

Figure 1. Rosembrock’s method fails with very narrow valleys.

To exploit the search direction that inaccurately detects the bottom of the valley, it is
necessary to change the point of view. OPTNOV’s method is based on some simple
ideas that make it particularly robust and efficient in the case of very narrow valleys:
• Whatever optimization algorithm is able to find the bottom of the valley by
starting from a point outside the same valley
• The line joining two points on the bottom of the valley is a reasonable valley
direction; therefore a point projected along such a direction has good probabilities
to be close to the valley
• Nevertheless, this valley direction must not be used as direction of one-
dimensional search, rather as a direction which a new point projection must be
carried out along
• This new point should not be discarded even though it is worse than the previous
one, rather it is the new starting point for the search.
• This search must be performed in the sub-space orthogonal to the valley direction
to prevent the problem of having small steps
This philosophy is particularly effective in an object-oriented programming coupled
with parallel computing as many reduced optimizations must be carried out starting
from distinct points and they can be independently solved each other. Consequently,
this philosophy of simultaneously solving different optimization problems by starting
from distinct guesses allows rationally facing even the global minimum paradigm.
The concept to build up a program for effectively tackling all aforementioned issues is
rather trivial as it is possible to develop an optimizer consisting of
N
objects, where
N

is the number of available processors and each of them uses in turn an optimizer
reasonably robust.
Hence, two distinct problems must be solved: the first is the selection of points used in
the
N
objects as initial guess and the second is which optimizer to use within each of
these
N
objects; it is worth remarking that even this optimizer must be opportunely
robust: to manage possible first- and second-order discontinuities of the function; to
overcome possible regions where the same function is undefined; to ensure the global
minimum in one-dimensional searches is found; and to efficiently tackle the problem of
slightly narrow valleys.
For the sake of clarity, let us call inner the optimizer used within each of the
N
objects
and outer the one managing the overall optimization problem. The outer optimizer only
is discussed in this paper.
G. Buzzi-Ferraris and F. Manenti
The problem of searching for the global optimum of the overall problem and to
overcome its possible narrow valleys are both tasks of the outer optimizer. The
following strategy is proposed to manage the
N
objects: three objects are required for
applying OPTNOV’s philosophy whereas the remaining
3N

objects are selected by
using the same techniques employed in optimal experimental design (Buzzi-Ferraris,
1999; Buzzi-Ferraris and Manenti, 2009a, 2010a, 2010b; Manenti and Buzzi-Ferraris,
2009). Therefore, there is the lower bound of using four processors (QUAD CORE
machines). The outer optimizer collects initial and arrival points of each inner object
and it selects the two points having the best performances among all those ones
collected. If these two points are significantly close, the best third, fourth… is selected
in spite of the second to avoid any ill-conditioning while detecting the valley direction.
B
I
A
II
III
δ
δ
ε
B
I
A
II
III
δ
δ
ε

Figure 2. Points A and B are on the bottom of the valley (A is the best one); points I, II, and III
are the possible point projections along the valley direction

Distances
δ
⁡湤=
ε
⁡=潮朠灯p湴猠捡渠扥⁲n摵捥搠潲⁥硰≤湤n搠慣捯牤楮朠瑯⁴桥⁲敳畬瑳a=
景爠r•慭灬攬⁩f⁴h攠灯p湴⁉II⁢ri湧n⁴o⁡⁢整ter=inner optimum, distances are expanded.
Points from the fourth to the
N th

are selected so to have the farthest points against
all the collected ones. This selection is efficiently carried out by using those techniques
adopted and proven for the optimal design of experiments. The following procedure is
adopted as stop criterion. At each iteration, the number of points in the neighborhood of
the optimum (given a tolerance value) is checked. If such a number is reasonable
(according to an assigned value), a possible solution is reached. Theoretically, the
number of points should be in the order of magnitude of the optimization problem
dimensions, but it is preferable to use smaller numbers when the optimization size is
large.
4. Numerical Tests
Many numerical tests were carried out to check the algorithm robustness for problems
of different dimensions. Tests of Table 1 are well-known literature functions for:

“Strong” multimodality issues: all directions are directions of function increase in
correspondence with local minima. Both Rastrigin (1) and Haupt (2) functions
were adopted and reported in Figure 3.

“Weak” multimodality issues: the function is constant along at least one direction
in correspondence of some local minima. Michalewicz’s function (3) was adopted
and reported in Figure 3.

Discontinuities and regions where the function is not defined (4). Figure 4 shows
the function against the variable
1
x
for the optimal value of
2
x
.

Extremely narrow valleys: valleys consisting of steep walls make the search of
the minimum a problematic issue for many optimizers. Buzzi-Ferraris’s function
(5) is adopted and reported in Figure 4.
( )
( )
2
1
110 10cos 2
n
RASTRIGIN i i
i
F
n x xπ
=
= − ⋅ + −

(1)
A Combination of Parallel Computing and Object-Oriented Programming to Improve
Optimizer Robustness and Efficiency

( )
( )
4
1
, 0
where: sin 2
0, 0
HAUPT
i i
i
HAUPT
F a a
a x x
F a
π
=
= − >

=

= ≤



(2)
( )
2
2
1
sin sin
n
n
i
MICHALEWICZ i
i
i x
F x
π

=
⎛ ⎞
⎛ ⎞
⎛ ⎞

⎜ ⎟
= − ⋅
⎜ ⎟
⎜ ⎟
⎜ ⎟
⎜ ⎟
⎝ ⎠
⎝ ⎠
⎝ ⎠


(3)
( )
( )
( )
1
,1 1 1 1 1
2 1 1
945 1689 950 230 25
10 10 10 6
x
BUZZI FERRARIS A
F x x x x x e
x x x


⎡ ⎤
= − + + − + + − + + +
⎣ ⎦
+ − + −

(4)
( )( )( )( )( ) ( )
2
2
,2 1 1 1 1 1 1
10000 1 3 5 7 9 8
BUZZI FERRARIS B
F x x x x x x x

= − − − − − − + −
⎡ ⎤
⎣ ⎦

(5)

HAHAUPT

Hau
p
t

-10
-5
0
5
10
-10
-5
0
5
10
0
50
100
150
200
250
Rastrigin

0
1
2
3
4
0
1
2
3
4
0
0.5
1
1.5
2
Michalewicz

Figure 3. Two-dimensional Haupt (left), Rastrigin (middle), and Michalewicz (right) functions


Solution
x1:_8.00000000000
x2:–1.050000e+006
_y:_1.092632e-023
Starting point
x1:
_
1.000000000
x2:
_
0.000000000
_
y:
_
49.00000000
10
0
-10
7
+10
7

Figure 4. Buzzi-Ferraris’s functions (A, left; B, right) to test optimizer robustness
Table 1. Optimization tests

Starting point Number of iterations Optimum value
Rastrigin
2n =

0
8.
=
x

2414 -240
Rastrigin
10n =

0
8.
=
x

8357 -1200
Haupt
0
0.
=
x

5559 -19.8630560097267
Michalewicz
2n =

0
3.
=
x

1683 -1.80130340983985
Michalewicz
10n =

0
3.
=
x

20144 -9.660152
Buzzi-Ferraris A
{
}
0
1.;1.=x

16714 6.708389
Buzzi-Ferraris B
{
}
0
1.;0.=x

1085 1.009e-018

G. Buzzi-Ferraris and F. Manenti

5. Conclusions and Future Developments
This preliminary research activity shows the way and reports some benefits coming
from the interaction of parallel computing and object-oriented programming.
Specifically, the example of C++ class for robust optimization that could generate a
series of objects so that each of them could run on a specific processor by increasing the
same optimizer robustness with a small programming effort is proposed.
References
Arora, N., L.T. Biegler. (2001). Redescending estimators for data reconciliation and parameter
estimation. Computers & Chemical Engineering, 25(11-12), 1585-1599.
Bagajewicz, M.J. (2003). Data Reconciliation and Instrumentation Upgrade. Overview and
Challenges. FOCAPO 2003. 4th International Conference of Computer-Aided Process
Operations, Coral Springs, Florida, 103-116.
Buzzi-Ferraris, G. (1967). Ottimizzazione di funzioni a più variabili. Nota I. Variabili non
vincolate. Ing. Chim. It., 3, 101.
Buzzi-Ferraris, G. (1994). Scientific C++. Building Numerical Libraries, the Object-Oriented
Way. 2nd Ed., 479pp, Addison-Wesley, Cambridge University Press, ISBN 0-201-63192-X.
Buzzi-Ferraris, G. (1999). Planning of experiments and kinetic analysis. Catalysis Today, 52,
125-132.
Buzzi-Ferraris, G., F. Manenti. (2009a). Kinetic models analysis. Chemical Engineering Science,
64(5), 1061-1074.
Buzzi-Ferraris, G., F. Manenti. (2009b). Outlier Detection. Computers & Chemical Engineering,
submitted.
Buzzi-Ferraris, G., F. Manenti. (2010a). Fundamentals and Linear Algebra for the Chemical
Engineer Solving Numerical Problems. ISBN: 978-3-527-32552-8, WILEY-VCH, Weinheim.
Buzzi-Ferraris, G., F. Manenti. (2010b). Interpolation and Regression Models for the Chemical
Engineer Solving Numerical Problems. ISBN: 978-3-527-32652-5, WILEY-VCH, Weinheim.
Cuoci, A., A. Frassoldati, G. Buzzi-Ferraris, T. Faravelli, E. Ranzi. (2007). The ignition,
combustion and flame structure of carbon monoxide/hydrogen mixtures. Note 2: Fluid
dynamics and kinetic aspects of syngas combustion. International Journal of Hydrogen
Energy, 32(15), 3486-3500.
Dones, I., F. Manenti, H.A. Preisig, G. Buzzi-Ferraris. (2009). Nonlinear Model Predictive
Control: a Self-Adaptive Approach. Industrial & Engineering Chemistry Research, to appear.
Hooke, R., T.A. Jeeves. (1961). "Direct Search" Solution of Numerical and Statistical Problems.
J. of the Assn. For Computing Machinery, 8, 212-229.
Lima, N.M.N., F. Manenti, R. Maciel Filho, M. Embiruçu, M.R. Wolf Maciel. (2009). Fuzzy
Model-Based Predictive Hybrid Control of Polymerization Processes. Industrial &
Engineering Chemistry Research, 48(18), 8542–8550.
Manenti, F. (2009). From Reacting to predicting technologies: A novel performance monitoring
technique based on detailed dynamic models. Chemical Product and Process Modeling, 4(2).
Manenti, F., G. Buzzi-Ferraris. (2009). Criteria for Outliers Detection in Nonlinear Regression
Problems. In J. Jezowski & J. Thullie (Eds.), Computer Aided Chemical Engineering (Vol. 26,
pp. 913-917).
Manenti, F., I. Dones, G. Buzzi-Ferraris, H.A. Preisig. (2009). Efficient Numerical Solver of
Partially Structured Differential and Algebraic Equation Systems. Industrial & Engineering
Chemistry Research, 48(22), 9979-9984.
Manenti, F., M. Rovaglio. (2008). Integrated multilevel optimization in large-scale polyethylene
terephthalate plants. Industrial and Engineering Chemistry Research, 47(1), 92-104.
Nelder, J.A., R. Mead. (1965). A simplex method for function minimization. Computer Journal, 7,
308-313.
Rosenbrock, H.H. (1960). An Automatic Method for Finding the Greater or Least Value of a
Function. the Computer Journal, 3, 175-184.
White, D.C. (2001). Increased Refinery Productivity through Online Performance Monitoring.
Emerson Process Management MCD Technology Division, NPRA Computer Conference.