Empirical Analysis of Model Selection Criteria for Genetic Programming in Modeling of Time Series System

wyomingbeancurdAI and Robotics

Nov 7, 2013 (3 years and 9 months ago)

78 views


Empirical Analysis of Model Selection Criteria for
Genetic Programming in Modeling of Time Series
System
A. Garg, S. Sriram, K. Tai
School of Mechanical and Aerospace Engineering
Nanyang Technological University
Singapore
Akhil1@e.ntu.edu.sg






Abstract— Genetic programming (GP) and its variants have been
extensively applied for modeling of the stock markets. To
improve the generalization ability of the model, GP have been
hybridized with its own variants (gene expression programming
(GEP), multi expression programming (MEP)) or with the other
methods such as neural networks and boosting. The
generalization ability of the GP model can also be improved by
an appropriate choice of model selection criterion. In the past,
several model selection criteria have been applied. In addition,
data transformations have significant impact on the performance
of the GP models. The literature reveals that few researchers
have paid attention to model selection criterion and data
transformation while modeling stock markets using GP. The
objective of this paper is to identify the most appropriate model
selection criterion and transformation that gives better
generalized GP models. Therefore, the present work will conduct
an empirical analysis to study the effect of three model selection
criteria across two data transformations on the performance of
GP while modeling the stock indexed in the New York Stock
Exchange (NYSE). It was found that FPE criteria have shown a
better fit for the GP model on both data transformations as
compared to other model selection criteria.
Keywords—genetic programming, model selection, stock market,
fitness function

I.

I
NTRODUCTION

Over the recent years, time series modeling have become an
active area of research. In time series modeling, stock market
prediction is of great challenge because it possesses higher
volatility, complexity and dynamics. The methods for
predicting stock market index can be divided into two
categories [1]. First category of methods includes five classical
methods such as exponential smoothing methods, regression
methods, autoregressive integrated moving average (ARIMA),
threshold methods and generalized autoregressive conditionally
hetroskedastic (GARCH) methods. The first three methods
assume the model in linear form and so may not be appropriate
for modeling highly non-linear chaotic stock markets. The last
two methods are able to capture the non-linearity in the stock
markets but rely strongly on assumption that the underlying
stock market must be stationary. However, in reality, the shift
in environmental conditions (sudden financial information)
could cause the stock market data to be change. Also, the
classical methods require human judgment to select a suitable
modeling method and then select an appropriate mathematical
model. Second category of methods includes modern heuristic
methods such as genetic programming (GP), artificial neural
network (ANN) and support vector regression (SVR) from the
field of machine learning. These methods have the ability to
handle non-static stock markets and build non-linear stock
market forecasting models. These methods does not require
model to be prescribed as in case of classical methods. Among
the heuristic methods, ANN cannot provide simple and elegant
representations of the model [33]. Therefore, the current paper
will focus on GP since it provides parsimonious models which
can explicitly described the changing trends in stock market
prices.
Several applications of GP for forecasting stock markets
have been reported in literature [2-7]. In these applications, the
GP method applied is either its variant (gene expression
programming (GEP) [8-10], multi expression programming
(MEP) [11]) or methods of ensembles [8, 12]. The variants of
GP mostly applied are based on the principle of linear genetic
programming (LGP). Since LGP is considered to be more
powerful than standard tree based GP [13], therefore it has
been preferred in modeling of stock markets. The hybrid
methods include the combination of either variants of GP or
GP and neural networks [14] or GP and genetic algorithms
(GA) [15] and GPboost [16]. These hybrid methods are applied
with an objective to improve the generalization ability of the
models. Another possible route to improve the generalization
ability of the models is the deployment of an appropriate model
selection criterion. The model selection criterion defines the
search efficiency of GP i.e. it controls the search to evolve
better fit and simpler models from the population of models.
The model selection criteria such as Akaike’s information
criteria (AIC) [17], Bayesian information criteria (BIC) [18],
final prediction error (FPE), predicted residual error sum of
squares (PRESS) [19], mean absolute percentage error
84
978-1-4673-5921-4/13/$31.00
c
￿2013 IEEE
(MAPE), root mean square error (RMSE) [20, 21] and
structural risk minimization (SRM) [22] takes into account the
number of data samples (sample size) and size of the model
(number of nodes of a GP tree) and penalizes the accuracy of
the model as the size of the model increases. In some cases, the
accuracy of the model may get compromised significantly with
an increase in size of the model which results in poor
generalization ability of the model. The exact amount of
penalization (complexity measure) of accuracy of the model is
never known and due to which, very often some models with
higher accuracy on training data and larger size perform poor
on testing data (unseen samples). There is a need of selecting
an appropriate balance between the complexity measure and
empirical error for the evolution of well generalized GP models
[1]. Every model selection criterion has different balance and
thus it becomes highly important to investigate the effect of
each criterion on the performance of GP models. In addition,
the pre-processing of data i.e. data transformation also have an
impact on the performance of GP models. Very few studies
exist in literature that compares the performance of GP in
respect to various model selection criteria [2, 23]. Iba and
Nikolaev [2] introduced various data transformations and
proposed enhanced STROGANOFF fitness function and
evaluated their effect on the performance of GP. It was found
that the GP with proposed fitness function produce better
predictive models.
The present work conducts an empirical analysis to study
the effect of various model selection criteria such as AIC, FPE
and PRESS across two data transformations on the
performance of GP while modeling the stock named ‘X’ listed
in New York stock exchange (NYSE). Currently, the authors’
have not named the company to avoid any sort of possible
conflict. The behavior of NYSE is a kind of random-walk and
is highly volatile and chaotic system. The purpose of the
empirical analysis is to identify the most appropriate model
selection criterion and transformation that could result in an
evolution of well generalized GP model. The rest of the paper
is organized as follows. Section 2 provides details on data
collection and methodology genetic programming (GP).
Section 3 shows the results and discussion. Finally, Section 4
concludes with the recommendations for the future work
II. M
ETHODOLOGY

The description on methodology comprise of two sub
sections: data preparation and genetic programming (GP).
A. Data Preparation
NYSE exchange market is considered in this study because
the return data have a large number of specific properties that
together makes the generalized forecasting model unusual. The
stock price of company ‘X’ for 290 days (July 1, 2011-
September 1, 2012) is noted. The data is divided into training,
validation and testing data in the ratio 4:1:1 respectively. Since
the actual raw data is considered to be noisy, the data is pre-
processed. The pre-processing step is essential to isolate the
significant information in the variables which is crucial for
successful learning of GP. The data is preprocessed through
two stages: (1) normalization (2) transformation. The
normalization is carried out using z-score so as to reduce the
effects due to magnitude of values. The data is then
transformed using two transformations. The two data
transformations are given as
t
1
= y
i+1
– y
i
(1)
t
2
= ln (y
i+1
/ y
i
) (2)
where y
i+1
and y
i
is the stock price on the following day and
present day respectively. Three model selection criteria to be
used are shown in Table I, where k is the number of nodes of
GP tree (size of model), SSE is the sum of square of error of
GP model on the training data and N is the number of training
samples.
T
ABLE
I

M
ODEL SELECTION CRITERION AND THEIR FORMULAE

Model selection criterion Mathematical formulae
AIC
N
log(SSE/
N
)+ k
FPE SSE/
N
(N+k/
N
-k)
PRESS SSE/
N
(1+2k/
N
)


B. Genetic Programming (GP)
GP has been applied to solve many real world problems.
One of the main applications is symbolic regression. GP is
based on the principle of Genetic algorithm (GA) which uses
Darwin’s theory of ‘survival of the fittest’. The principles of
GP and its terminology is formulated by Koza [24]. The aim of
the GP is to evolve an optimum program (model) that fits to a
given system. The models are represented in form of trees
structure and large population of these models is generated.
Each model is made of elements from the functional and
terminal set. The functional set elements can be basic
mathematical operators (+, - , ×, /) and/or Boolean algebraic
operators (eg. AND and OR) or any other user defined
mathematical symbols. The terminal set consists of input
variables of the problem. More details about this method can be
found in [25]. In present work, Multigene genetic programming
developed by Hinchliffe et al., [26] and Hiden [27] is used. The
main difference between the traditional GP and multigene GP
is that in multigene GP, number of trees makes a model. All of
the genes are combined by weights (different for each gene)
and a constant term added to it gives final formulae
(mathematical model). Multigene GP can be written as
y = a
0
+ a
1
gene
1
+a
2
gene
2
+a
3
gene
3
+…+a
n
gene
n
(3)
where a
0
is a bias term and a
i
is the weight of the ith gene.
Multigene genetic programming is implemented using the
software GPTIPS. This software is written on MATLAB.
Several applications of this software are reported in literature
[29-32]. This software tool has a inherited capacity to avoid
bloat problem by imposing restrictions on maximum value of
parameters such as the number of genes, depth of trees and
genes, number of nodes per tree, etc.
III. R
ESUTS AND DISCUSSION

GPTIPS is applied on each transformed data (stock price as
output and days as inputs (
x
1
)) for each model selection
criterion. Total of 30 runs are performed with the parameter
2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr) 85
settings shown in Table II. The best GP model is selected for
each model selection criterion across two transformations
based on minimum normalized RMSE on the validation data. A
total of 6 best GP models are selected. The performance of
each best GP model is evaluated on training, validation and
testing data. Figures 1-3 shows the performance of the best GP
model using transformation t
1
for the model selection criteria
AIC, FPE and PRESS respectively. Figures 4-6 shows the
performance of the best GP model using transformation t
2
for
the model selection criteria AIC, FPE and PRESS respectively.
Table III-IV shows the correlation coefficient of the best model
on training and testing data set for the model selection criteria
AIC, FPE and PRESS respectively. The graphs shown in
Figures 1-3 and Table III indicates that the best GP model
using FPE model selection criterion have shown better fit when
compared to the model selection criterion AIC and PRESS.
The best GP model using AIC and PRESS have shown sign of
overfitting because they have greater training, validation and
testing error. The graphs shown in Figures 4-6 and Table IV
indicates that the best GP model using FPE and PRESS model
selection criterion have shown better fit while AIC have given
the poorest performance. The possible reason for the poor
performance of AIC and PRESS for transformation t
1
and AIC
for transformation t
2
could be that these model selection criteria
have significantly penalized the accuracy of the model with an
increase in size of the model.
T
ABLE II
.
PARAMETER SETTINGS FOR GP




Figure 1. Fit of predicted values of GP model versus actual values for AIC

Figure 2. Fit of predicted values of GP model versus actual values for FPE


Figure 3. Fit of predicted values of GP model versus actual values for
PRESS

Figure 4. Fit of predicted values of GP model versus actual values for AIC

Figure 5. Fit of predicted values of GP model versus actual values for FPE

Figure 6. Fit of predicted values of GP model versus actual values for
PRESS
0
50
100
150
200
250
300
-4
-3
-2
-1
0
1
2
3
4
5
Days
Closing values
X - AIC


Actual values
Predicted values
0
50
100
150
200
250
30
0
-4
-3
-2
-1
0
1
2
3
4
5
Days
Closing values
X - FPE


Actual values
Predicted values
0
50
100
150
200
250
30
0
-4
-3
-2
-1
0
1
2
3
4
5
Days
Closing values
X - PRESS


Actual values
Predicted values
0
50
100
150
200
250
300
-0.15
-0.1
-0.05
0
0.05
0.1
Days
Closing Values
X - AIC


Actual values
Predicted values
0
50
100
150
200
250
300
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
Days
Closing Values
X - FPE


Actual values
Predicted Values
0
50
100
150
200
250
300
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
Days
Closing values
X - PRESS


Actual values
Predicted values
Parameters Values assigned
Population size 1000
Number of Generations 200
Tournament selection and
size
Lexicographic, 4
Termination criteria Maximum number of runs
Maximum number of genes 5
Functional set (multiply, plus, minus, divide, square,
cosine, sine, exp, tanh, sqrt, log, tan,
power)
Terminal set (x
1
, [-10 10])
Mutation rate 0.10
Crossover rate 0.85
Reproduction rate 0.05
86 2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr)
TABLE

III
CORRELATION COEFFICIENT FOR BEST GP MODELFOR EACH MODEL
SELECTION CRITERIA BASED ON TRANSFORM T
1

Criteria Training data set Testing data set
AIC 0.47 0.08
FPE 0.25 0.16
PRESS 0.27 0.04

TABLE

IV
CORRELATION COEFFICIENT FOR BEST GP MODELFOR EACH MODEL
SELECTION CRITERIA BASED ON TRANSFORM T
2

Criteria Training data set Testing data set
AIC 0.48 0.09
FPE 0.26 0.23
PRESS 0.35 0.23

IV. C
ONCLUSION AND FUTURE WORK

An empirical analysis to study the impact of various model
selection criteria on the performance of GP is conducted. The
results conclude that for both the transformation, the model
selection criterion FPE have shown better fit of the best GP
model when compared to other selection criterion. There is no
sign of overfitting of the model and therefore the best model
formed from the FPE model selection criteria have better
generalization ability. The transformation t
2
have shown better
fit of the best GP model for the two model selection criteria
FPE and PRESS while transformation t
1
have shown only for
FPE. This clearly indicates that the transformation t
2
have
performed better while modeling with GP. The current study is
limited to one stock prediction, three model selection criteria
and two transformations. Therefore in future, an indepth study
can be conducted by incorporating additional fitness functions
such as SRM, BIC, Allenby, etc and various other
transformations and measure their effect on the performance of
GP for multistocks prediction


REFERENCES:
[1] Wagner, N., et al., Time series forecasting for dynamic
environments: the DyFor genetic program model. Evolutionary
Computation, IEEE Transactions on, 2007. 11(4): p. 433-452.
[2] Iba, H. and N. Nikolaev. Genetic programming polynomial models
of financial data series. in Evolutionary Computation, 2000.
Proceedings of the 2000 Congress on. 2000. IEEE.
[3] El-Telbany, M.E. The Egyptian stock market return prediction: A
genetic programming approach. in International Conference on
Electrical, Electronic and Computer Engineering. 2005.
[4] Santini, M. and A. Tettamanzi, Genetic programming for financial
time series prediction. Genetic Programming, 2001: p. 361-370.
[5] Wagner, N. and Z. Michalewicz. Genetic programming with
efficient population control for financial time series prediction. in
2001 Genetic and Evolutionary Computation Conference Late
Breaking Papers. 2001.
[6] Kaboudan, M., Genetic programming prediction of stock prices.
Computational Economics, 2000. 16(3): p. 207-236.
[7] Iba, H. and T. Sasaki. Using genetic programming to predict
financial data. in Evolutionary Computation, 1999. CEC 99.
Proceedings of the 1999 Congress on. 1999. IEEE.
[8] Grosan, C. and A. Abraham, Stock market modeling using genetic
programming ensembles. Genetic Systems Programming, 2006: p.
131-146.
[9] Chen, F., Y.H. Chen, and J.Z. Zhang, Stock index modeling using
IP based gene expression programming. Jisuanji Gongcheng yu
Yingyong(Computer Engineering and Applications), 2007. 42(26):
p. 227-229.
[10] Qian, X. and C. Yang, Improved gene expression programming
algorithm tested by predicting stock indexes. CAAI Transactions
on Intelligent Systems, 2010. 4: p. 005.
[11] Grosan, C., et al. Stock market prediction using multi expression
programming. in Artificial intelligence, 2005. epia 2005.
portuguese conference on. 2005. IEEE.
[12] Kwon, Y.K. and B.R. Moon. Evolutionary ensemble for stock
prediction. in Genetic and Evolutionary Computation–GECCO
2004. 2004. Springer.
[13] Oltean, M. and C. Grosan, A comparison of several linear genetic
programming techniques. Complex Systems, 2003. 14(4): p. 285-
314.
[14] Hsu, C.M., A hybrid procedure for stock price prediction by
integrating self-organizing map and genetic programming. Expert
systems with applications, 2011. 38(11): p. 14026-14036.
[15] Chen, S.H., Genetic algorithms and genetic programming in
computational finance2002: Springer.
[16] de Souza, L.V., A.T.R. Pozo, and A.C. Neto. Using correlation to
improve boosting technique: an application for time series
forecasting. in Tools with Artificial Intelligence, 2006. ICTAI'06.
18th IEEE International Conference on. 2006. IEEE.
[17] Akaike, H., An information criterion (AIC). Math Sci, 1976.
14(153): p. 5-9.
[18] Schwarz, G., Estimating the dimension of a model. The annals of
statistics, 1978. 6(2): p. 461-464.
[19] Barron, A.R., Predicted squared error: a criterion for automatic
model selection. 1984.
[20] Miller, A., Subset selection in regression2002: Chapman &
Hall/CRC.
[21] Moody, J., Prediction risk and architecture selection for neural
networks. NATO ASI SERIES F COMPUTER AND SYSTEMS
SCIENCES, 1994. 136: p. 147-147.
[22] Vapnik, V.N., Statistical learning theory. 1998.
[23] Cherkassky, V. and Y. Ma, Comparison of model selection for
regression. Neural computation, 2003. 15(7): p. 1691-1714.
[24] Koza, J.R., Genetic programming II: automatic discovery of
reusable programs. 1994.
[25] Koza, J.R., Genetic programming II: automatic discovery of
reusable programs1994.
[26] Hinchliffe, M., et al. Modelling chemical process systems using a
multi-gene genetic programming algorithm
. in Genetic
Programming: Proceedings of the First Annual Conference (late
breaking papers). 1996.
2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr) 87
[27] Hiden, H.G., Data-based modelling using genetic programming,
1998, University of Newcastle upon Tyne.
[28] Searson, D.P., D.E. Leahy, and M.J. Willis. GPTIPS: an open
source genetic programming toolbox for multigene symbolic
regression. in Proceedings of the International MultiConference of
Engineers and Computer Scientists. 2010. Citeseer.
[29] Garg, A. & Tai, K. 2011. A Hybrid Genetic Programming–
Artificial Neural Network Approach For Modeling of Vibratory
Finishing Process. International Proceedings of Computer Science
and Information Technology (IPCSIT), Vol. 18, pp. 14-19.
[30] Garg, A. & Tai, K. 2012. Comparison of regression analysis,
Artificial Neural Network and genetic programming in Handling
the multicollinearity problem. Proceedings of International
Conference on Modelling, Identification & Control (ICMIC).
IEEE, pp. 353-358.
[31] Garg, A. and Tai, K. 2012. Review of Genetic Programming in
modeling of machining processes. International Conference on
Modelling, Identification and Control (ICMIC), IEEE, pp.653-658
[32] Baziar, M.H., et al., Prediction of strain energy-based liquefaction
resistance of sand–silt mixtures: An evolutionary approach.
Computers & Geosciences, 2011. 37(11): p. 1883-189











.




[33] Zhang, W., Z. Wu, and G. Yang, Genetic programming-based
chaotic time series modeling. Journal of Zhejiang University
Science A, 2004. 5(11): p. 1432-1439.
88 2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr)