Empirical Analysis of Model Selection Criteria for

Genetic Programming in Modeling of Time Series

System

A. Garg, S. Sriram, K. Tai

School of Mechanical and Aerospace Engineering

Nanyang Technological University

Singapore

Akhil1@e.ntu.edu.sg

Abstract— Genetic programming (GP) and its variants have been

extensively applied for modeling of the stock markets. To

improve the generalization ability of the model, GP have been

hybridized with its own variants (gene expression programming

(GEP), multi expression programming (MEP)) or with the other

methods such as neural networks and boosting. The

generalization ability of the GP model can also be improved by

an appropriate choice of model selection criterion. In the past,

several model selection criteria have been applied. In addition,

data transformations have significant impact on the performance

of the GP models. The literature reveals that few researchers

have paid attention to model selection criterion and data

transformation while modeling stock markets using GP. The

objective of this paper is to identify the most appropriate model

selection criterion and transformation that gives better

generalized GP models. Therefore, the present work will conduct

an empirical analysis to study the effect of three model selection

criteria across two data transformations on the performance of

GP while modeling the stock indexed in the New York Stock

Exchange (NYSE). It was found that FPE criteria have shown a

better fit for the GP model on both data transformations as

compared to other model selection criteria.

Keywords—genetic programming, model selection, stock market,

fitness function

I.

I

NTRODUCTION

Over the recent years, time series modeling have become an

active area of research. In time series modeling, stock market

prediction is of great challenge because it possesses higher

volatility, complexity and dynamics. The methods for

predicting stock market index can be divided into two

categories [1]. First category of methods includes five classical

methods such as exponential smoothing methods, regression

methods, autoregressive integrated moving average (ARIMA),

threshold methods and generalized autoregressive conditionally

hetroskedastic (GARCH) methods. The first three methods

assume the model in linear form and so may not be appropriate

for modeling highly non-linear chaotic stock markets. The last

two methods are able to capture the non-linearity in the stock

markets but rely strongly on assumption that the underlying

stock market must be stationary. However, in reality, the shift

in environmental conditions (sudden financial information)

could cause the stock market data to be change. Also, the

classical methods require human judgment to select a suitable

modeling method and then select an appropriate mathematical

model. Second category of methods includes modern heuristic

methods such as genetic programming (GP), artificial neural

network (ANN) and support vector regression (SVR) from the

field of machine learning. These methods have the ability to

handle non-static stock markets and build non-linear stock

market forecasting models. These methods does not require

model to be prescribed as in case of classical methods. Among

the heuristic methods, ANN cannot provide simple and elegant

representations of the model [33]. Therefore, the current paper

will focus on GP since it provides parsimonious models which

can explicitly described the changing trends in stock market

prices.

Several applications of GP for forecasting stock markets

have been reported in literature [2-7]. In these applications, the

GP method applied is either its variant (gene expression

programming (GEP) [8-10], multi expression programming

(MEP) [11]) or methods of ensembles [8, 12]. The variants of

GP mostly applied are based on the principle of linear genetic

programming (LGP). Since LGP is considered to be more

powerful than standard tree based GP [13], therefore it has

been preferred in modeling of stock markets. The hybrid

methods include the combination of either variants of GP or

GP and neural networks [14] or GP and genetic algorithms

(GA) [15] and GPboost [16]. These hybrid methods are applied

with an objective to improve the generalization ability of the

models. Another possible route to improve the generalization

ability of the models is the deployment of an appropriate model

selection criterion. The model selection criterion defines the

search efficiency of GP i.e. it controls the search to evolve

better fit and simpler models from the population of models.

The model selection criteria such as Akaike’s information

criteria (AIC) [17], Bayesian information criteria (BIC) [18],

final prediction error (FPE), predicted residual error sum of

squares (PRESS) [19], mean absolute percentage error

84

978-1-4673-5921-4/13/$31.00

c

2013 IEEE

(MAPE), root mean square error (RMSE) [20, 21] and

structural risk minimization (SRM) [22] takes into account the

number of data samples (sample size) and size of the model

(number of nodes of a GP tree) and penalizes the accuracy of

the model as the size of the model increases. In some cases, the

accuracy of the model may get compromised significantly with

an increase in size of the model which results in poor

generalization ability of the model. The exact amount of

penalization (complexity measure) of accuracy of the model is

never known and due to which, very often some models with

higher accuracy on training data and larger size perform poor

on testing data (unseen samples). There is a need of selecting

an appropriate balance between the complexity measure and

empirical error for the evolution of well generalized GP models

[1]. Every model selection criterion has different balance and

thus it becomes highly important to investigate the effect of

each criterion on the performance of GP models. In addition,

the pre-processing of data i.e. data transformation also have an

impact on the performance of GP models. Very few studies

exist in literature that compares the performance of GP in

respect to various model selection criteria [2, 23]. Iba and

Nikolaev [2] introduced various data transformations and

proposed enhanced STROGANOFF fitness function and

evaluated their effect on the performance of GP. It was found

that the GP with proposed fitness function produce better

predictive models.

The present work conducts an empirical analysis to study

the effect of various model selection criteria such as AIC, FPE

and PRESS across two data transformations on the

performance of GP while modeling the stock named ‘X’ listed

in New York stock exchange (NYSE). Currently, the authors’

have not named the company to avoid any sort of possible

conflict. The behavior of NYSE is a kind of random-walk and

is highly volatile and chaotic system. The purpose of the

empirical analysis is to identify the most appropriate model

selection criterion and transformation that could result in an

evolution of well generalized GP model. The rest of the paper

is organized as follows. Section 2 provides details on data

collection and methodology genetic programming (GP).

Section 3 shows the results and discussion. Finally, Section 4

concludes with the recommendations for the future work

II. M

ETHODOLOGY

The description on methodology comprise of two sub

sections: data preparation and genetic programming (GP).

A. Data Preparation

NYSE exchange market is considered in this study because

the return data have a large number of specific properties that

together makes the generalized forecasting model unusual. The

stock price of company ‘X’ for 290 days (July 1, 2011-

September 1, 2012) is noted. The data is divided into training,

validation and testing data in the ratio 4:1:1 respectively. Since

the actual raw data is considered to be noisy, the data is pre-

processed. The pre-processing step is essential to isolate the

significant information in the variables which is crucial for

successful learning of GP. The data is preprocessed through

two stages: (1) normalization (2) transformation. The

normalization is carried out using z-score so as to reduce the

effects due to magnitude of values. The data is then

transformed using two transformations. The two data

transformations are given as

t

1

= y

i+1

– y

i

(1)

t

2

= ln (y

i+1

/ y

i

) (2)

where y

i+1

and y

i

is the stock price on the following day and

present day respectively. Three model selection criteria to be

used are shown in Table I, where k is the number of nodes of

GP tree (size of model), SSE is the sum of square of error of

GP model on the training data and N is the number of training

samples.

T

ABLE

I

M

ODEL SELECTION CRITERION AND THEIR FORMULAE

Model selection criterion Mathematical formulae

AIC

N

log(SSE/

N

)+ k

FPE SSE/

N

(N+k/

N

-k)

PRESS SSE/

N

(1+2k/

N

)

B. Genetic Programming (GP)

GP has been applied to solve many real world problems.

One of the main applications is symbolic regression. GP is

based on the principle of Genetic algorithm (GA) which uses

Darwin’s theory of ‘survival of the fittest’. The principles of

GP and its terminology is formulated by Koza [24]. The aim of

the GP is to evolve an optimum program (model) that fits to a

given system. The models are represented in form of trees

structure and large population of these models is generated.

Each model is made of elements from the functional and

terminal set. The functional set elements can be basic

mathematical operators (+, - , ×, /) and/or Boolean algebraic

operators (eg. AND and OR) or any other user defined

mathematical symbols. The terminal set consists of input

variables of the problem. More details about this method can be

found in [25]. In present work, Multigene genetic programming

developed by Hinchliffe et al., [26] and Hiden [27] is used. The

main difference between the traditional GP and multigene GP

is that in multigene GP, number of trees makes a model. All of

the genes are combined by weights (different for each gene)

and a constant term added to it gives final formulae

(mathematical model). Multigene GP can be written as

y = a

0

+ a

1

gene

1

+a

2

gene

2

+a

3

gene

3

+…+a

n

gene

n

(3)

where a

0

is a bias term and a

i

is the weight of the ith gene.

Multigene genetic programming is implemented using the

software GPTIPS. This software is written on MATLAB.

Several applications of this software are reported in literature

[29-32]. This software tool has a inherited capacity to avoid

bloat problem by imposing restrictions on maximum value of

parameters such as the number of genes, depth of trees and

genes, number of nodes per tree, etc.

III. R

ESUTS AND DISCUSSION

GPTIPS is applied on each transformed data (stock price as

output and days as inputs (

x

1

)) for each model selection

criterion. Total of 30 runs are performed with the parameter

2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr) 85

settings shown in Table II. The best GP model is selected for

each model selection criterion across two transformations

based on minimum normalized RMSE on the validation data. A

total of 6 best GP models are selected. The performance of

each best GP model is evaluated on training, validation and

testing data. Figures 1-3 shows the performance of the best GP

model using transformation t

1

for the model selection criteria

AIC, FPE and PRESS respectively. Figures 4-6 shows the

performance of the best GP model using transformation t

2

for

the model selection criteria AIC, FPE and PRESS respectively.

Table III-IV shows the correlation coefficient of the best model

on training and testing data set for the model selection criteria

AIC, FPE and PRESS respectively. The graphs shown in

Figures 1-3 and Table III indicates that the best GP model

using FPE model selection criterion have shown better fit when

compared to the model selection criterion AIC and PRESS.

The best GP model using AIC and PRESS have shown sign of

overfitting because they have greater training, validation and

testing error. The graphs shown in Figures 4-6 and Table IV

indicates that the best GP model using FPE and PRESS model

selection criterion have shown better fit while AIC have given

the poorest performance. The possible reason for the poor

performance of AIC and PRESS for transformation t

1

and AIC

for transformation t

2

could be that these model selection criteria

have significantly penalized the accuracy of the model with an

increase in size of the model.

T

ABLE II

.

PARAMETER SETTINGS FOR GP

Figure 1. Fit of predicted values of GP model versus actual values for AIC

Figure 2. Fit of predicted values of GP model versus actual values for FPE

Figure 3. Fit of predicted values of GP model versus actual values for

PRESS

Figure 4. Fit of predicted values of GP model versus actual values for AIC

Figure 5. Fit of predicted values of GP model versus actual values for FPE

Figure 6. Fit of predicted values of GP model versus actual values for

PRESS

0

50

100

150

200

250

300

-4

-3

-2

-1

0

1

2

3

4

5

Days

Closing values

X - AIC

Actual values

Predicted values

0

50

100

150

200

250

30

0

-4

-3

-2

-1

0

1

2

3

4

5

Days

Closing values

X - FPE

Actual values

Predicted values

0

50

100

150

200

250

30

0

-4

-3

-2

-1

0

1

2

3

4

5

Days

Closing values

X - PRESS

Actual values

Predicted values

0

50

100

150

200

250

300

-0.15

-0.1

-0.05

0

0.05

0.1

Days

Closing Values

X - AIC

Actual values

Predicted values

0

50

100

150

200

250

300

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

Days

Closing Values

X - FPE

Actual values

Predicted Values

0

50

100

150

200

250

300

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

Days

Closing values

X - PRESS

Actual values

Predicted values

Parameters Values assigned

Population size 1000

Number of Generations 200

Tournament selection and

size

Lexicographic, 4

Termination criteria Maximum number of runs

Maximum number of genes 5

Functional set (multiply, plus, minus, divide, square,

cosine, sine, exp, tanh, sqrt, log, tan,

power)

Terminal set (x

1

, [-10 10])

Mutation rate 0.10

Crossover rate 0.85

Reproduction rate 0.05

86 2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr)

TABLE

III

CORRELATION COEFFICIENT FOR BEST GP MODELFOR EACH MODEL

SELECTION CRITERIA BASED ON TRANSFORM T

1

Criteria Training data set Testing data set

AIC 0.47 0.08

FPE 0.25 0.16

PRESS 0.27 0.04

TABLE

IV

CORRELATION COEFFICIENT FOR BEST GP MODELFOR EACH MODEL

SELECTION CRITERIA BASED ON TRANSFORM T

2

Criteria Training data set Testing data set

AIC 0.48 0.09

FPE 0.26 0.23

PRESS 0.35 0.23

IV. C

ONCLUSION AND FUTURE WORK

An empirical analysis to study the impact of various model

selection criteria on the performance of GP is conducted. The

results conclude that for both the transformation, the model

selection criterion FPE have shown better fit of the best GP

model when compared to other selection criterion. There is no

sign of overfitting of the model and therefore the best model

formed from the FPE model selection criteria have better

generalization ability. The transformation t

2

have shown better

fit of the best GP model for the two model selection criteria

FPE and PRESS while transformation t

1

have shown only for

FPE. This clearly indicates that the transformation t

2

have

performed better while modeling with GP. The current study is

limited to one stock prediction, three model selection criteria

and two transformations. Therefore in future, an indepth study

can be conducted by incorporating additional fitness functions

such as SRM, BIC, Allenby, etc and various other

transformations and measure their effect on the performance of

GP for multistocks prediction

REFERENCES:

[1] Wagner, N., et al., Time series forecasting for dynamic

environments: the DyFor genetic program model. Evolutionary

Computation, IEEE Transactions on, 2007. 11(4): p. 433-452.

[2] Iba, H. and N. Nikolaev. Genetic programming polynomial models

of financial data series. in Evolutionary Computation, 2000.

Proceedings of the 2000 Congress on. 2000. IEEE.

[3] El-Telbany, M.E. The Egyptian stock market return prediction: A

genetic programming approach. in International Conference on

Electrical, Electronic and Computer Engineering. 2005.

[4] Santini, M. and A. Tettamanzi, Genetic programming for financial

time series prediction. Genetic Programming, 2001: p. 361-370.

[5] Wagner, N. and Z. Michalewicz. Genetic programming with

efficient population control for financial time series prediction. in

2001 Genetic and Evolutionary Computation Conference Late

Breaking Papers. 2001.

[6] Kaboudan, M., Genetic programming prediction of stock prices.

Computational Economics, 2000. 16(3): p. 207-236.

[7] Iba, H. and T. Sasaki. Using genetic programming to predict

financial data. in Evolutionary Computation, 1999. CEC 99.

Proceedings of the 1999 Congress on. 1999. IEEE.

[8] Grosan, C. and A. Abraham, Stock market modeling using genetic

programming ensembles. Genetic Systems Programming, 2006: p.

131-146.

[9] Chen, F., Y.H. Chen, and J.Z. Zhang, Stock index modeling using

IP based gene expression programming. Jisuanji Gongcheng yu

Yingyong(Computer Engineering and Applications), 2007. 42(26):

p. 227-229.

[10] Qian, X. and C. Yang, Improved gene expression programming

algorithm tested by predicting stock indexes. CAAI Transactions

on Intelligent Systems, 2010. 4: p. 005.

[11] Grosan, C., et al. Stock market prediction using multi expression

programming. in Artificial intelligence, 2005. epia 2005.

portuguese conference on. 2005. IEEE.

[12] Kwon, Y.K. and B.R. Moon. Evolutionary ensemble for stock

prediction. in Genetic and Evolutionary Computation–GECCO

2004. 2004. Springer.

[13] Oltean, M. and C. Grosan, A comparison of several linear genetic

programming techniques. Complex Systems, 2003. 14(4): p. 285-

314.

[14] Hsu, C.M., A hybrid procedure for stock price prediction by

integrating self-organizing map and genetic programming. Expert

systems with applications, 2011. 38(11): p. 14026-14036.

[15] Chen, S.H., Genetic algorithms and genetic programming in

computational finance2002: Springer.

[16] de Souza, L.V., A.T.R. Pozo, and A.C. Neto. Using correlation to

improve boosting technique: an application for time series

forecasting. in Tools with Artificial Intelligence, 2006. ICTAI'06.

18th IEEE International Conference on. 2006. IEEE.

[17] Akaike, H., An information criterion (AIC). Math Sci, 1976.

14(153): p. 5-9.

[18] Schwarz, G., Estimating the dimension of a model. The annals of

statistics, 1978. 6(2): p. 461-464.

[19] Barron, A.R., Predicted squared error: a criterion for automatic

model selection. 1984.

[20] Miller, A., Subset selection in regression2002: Chapman &

Hall/CRC.

[21] Moody, J., Prediction risk and architecture selection for neural

networks. NATO ASI SERIES F COMPUTER AND SYSTEMS

SCIENCES, 1994. 136: p. 147-147.

[22] Vapnik, V.N., Statistical learning theory. 1998.

[23] Cherkassky, V. and Y. Ma, Comparison of model selection for

regression. Neural computation, 2003. 15(7): p. 1691-1714.

[24] Koza, J.R., Genetic programming II: automatic discovery of

reusable programs. 1994.

[25] Koza, J.R., Genetic programming II: automatic discovery of

reusable programs1994.

[26] Hinchliffe, M., et al. Modelling chemical process systems using a

multi-gene genetic programming algorithm

. in Genetic

Programming: Proceedings of the First Annual Conference (late

breaking papers). 1996.

2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr) 87

[27] Hiden, H.G., Data-based modelling using genetic programming,

1998, University of Newcastle upon Tyne.

[28] Searson, D.P., D.E. Leahy, and M.J. Willis. GPTIPS: an open

source genetic programming toolbox for multigene symbolic

regression. in Proceedings of the International MultiConference of

Engineers and Computer Scientists. 2010. Citeseer.

[29] Garg, A. & Tai, K. 2011. A Hybrid Genetic Programming–

Artificial Neural Network Approach For Modeling of Vibratory

Finishing Process. International Proceedings of Computer Science

and Information Technology (IPCSIT), Vol. 18, pp. 14-19.

[30] Garg, A. & Tai, K. 2012. Comparison of regression analysis,

Artificial Neural Network and genetic programming in Handling

the multicollinearity problem. Proceedings of International

Conference on Modelling, Identification & Control (ICMIC).

IEEE, pp. 353-358.

[31] Garg, A. and Tai, K. 2012. Review of Genetic Programming in

modeling of machining processes. International Conference on

Modelling, Identification and Control (ICMIC), IEEE, pp.653-658

[32] Baziar, M.H., et al., Prediction of strain energy-based liquefaction

resistance of sand–silt mixtures: An evolutionary approach.

Computers & Geosciences, 2011. 37(11): p. 1883-189

.

[33] Zhang, W., Z. Wu, and G. Yang, Genetic programming-based

chaotic time series modeling. Journal of Zhejiang University

Science A, 2004. 5(11): p. 1432-1439.

88 2013 IEEE Conference on Computational Intelligence for Financial Engineering &Economics (CIFEr)

## Comments 0

Log in to post a comment