Content

munchsistersΤεχνίτη Νοημοσύνη και Ρομποτική

17 Οκτ 2013 (πριν από 4 χρόνια και 2 μήνες)

110 εμφανίσεις







GIEE

ICS

R01943121
陳柏淳







2013 06

Content

1

Abstruct

................................
................................
................................
................

3

2

Introduction

................................
................................
................................
..........

4

2.1

Multi
-
media Content Analysis

................................
...............................

4

2.2

Machine Learning

and Design Challenges

................................
............

5

2.3

An Introduct
or
y Example

................................
................................
......

7

3

Learning
Pattern Recognition From Example

................................
................

11

4

Hyperplane Classifiers

................................
................................
......................

13

5

Optimal Margin Support Vector Classifiers

................................
...................

17

6

Kernels

................................
................................
................................
................

18

6
.1

Product Features
................................
................................
...................

18

6
.2

Polynomial Feature Spaces Induced by Kernels

....

Error! Bookmark not
defined.
8

6.3

Examples of Kernels

..............................

Error! Bookmark not defined.
9

7

Multi
-
class SV Classifiers

................................
................................
..................

20

7
.1

One
-
against
-
all Method

................................
................................
.......

20

7
.2

One
-
against
-
one

Method

................................
................................
......

25

7.3

Considering All Data At Once Method

................................
...............

25

8

Applications

................................
............................

Error! Bookmark not defined.

8
.1

LIBSVM

................................
................................
..............................

2
8

8
.2

Experiment

................................
................................
...........................

2
8

9

Conclusion

................................
..............................

Error! Bookmark not defined.

10

Reference

................................
................................

Error! Bookmark not defined.
















1

Abstract

Learning general functional dependencies is one of the main goals in machine
learning. Recent progress in kernel
-
based methods has focused on designing flexible
and powerful input representations. This tutorial addresses the complementary issue
of problems

involving complex outputs such as multiple dependent output variables
and structured output spaces.
Some

propose to generalize multiclass Support Vector

Machine learning in a formulation that involves features extracted jointly from inputs
and outputs. Th
e resulting optimization problem is solved efficiently by a cutting
plane algorithm that exploits the sparseness and structural decomposition of the
problem.
Some

demonstrate the versatility and effectiveness on problems ranging
from supervised grammar lea
rning and named
-
entity recognition, to taxonomic text
classification and sequence alignment.

The tutorial starts with an overview of the concepts of structural risk

minimization.
T
hen describe linear Support Vector Machines (SVMs) for separable
and non
-
sep
arable

data, working through a non
-
trivial example in detail. We describe
a mechanical analogy, and discuss

when SVM solutions are unique and when they are
global. We describe how support vector training can

be practically implemented, and
discuss in detai
l the kernel mapping technique which is used to construct

SVM
solutions which are nonlinear in the data. We show how Support Vector machines can
have very large

(even infinite) VC dimension by computing the VC dimension for
homogeneous polynomial and Gaussi
an

radial basis function kernels. While very high
VC dimension would normally bode ill for generalization

performance, and while at present there exists no theory which shows that good

generalization performance

is guaranteed for SVMs, there are several ar
guments
which support the observed high accuracy of SVMs,

which we review. Results of
some experiments which were inspired by these arguments are also presented.

We give numerous examples and proofs of most of the key theorems.











2

Introduction

2.1

Multi
-
media Content Analysis

With the growth of the semiconductor technology, hand
-
held devices nowadays

are equipped with more and more powerful VLSI architectures, like high
-
quality

CMOS senso
rs, large storage devices,.. ,
and so on. People can easily u
se this

kind of
product to take and store pictures, and to search for the related images in

the database.
Since the storage on the hand
-
held devices with cameras becomes

larger and larger, it
is quite often that the user took hundreds of thousands of

pictu
res and kept them in the
storage device without classifying them. It will be a

big burden to manually classify
and tag this huge amount of photos.

To effi
ciently manage such huge amount of
c
ollections, it is necessary to have

access to high
-
level informati
on about the contents
obtained in the image
.

Since it might be troublesome for people to manually
manipulate, index, sort,

fi
lter, summarize, or search
through the photos they take
, it is
necessary to

establish a system which can automatically analyze the
contents of
p
hotos. Organizing

this huge amount of photos into categories and providing effective
indexing

is imperative for

”real
-
time”

browsing and retrieval
.

The technique to
accomplish such tasks is called ”
Image Classifi
cation” or

”Image Categorizati
on”.
Early works only consider general purpose semantic

classes, such as outdoor

scenes
versus indoor scenes
, or city scenes versus

landscape scenes .
However, all these
previous works rely only on the low
-
level features and

the
y can handle very simple
classifi
cation problems as they only consider a few

classes. To classify such huge
amount of informati
ve image data, it is diffi
cult

to use only low
-
level features since
those features do not strongly correlate with

human perception. Therefore, a semantic
modeling step has to be employed to

bridge this semantic gap. Many software
solutions have been proposed to

provide an intelligent indexing and im
age content
analysis platform
.

One way to bridge this semantic gap is to map the low
-
level
features to the sem
antic

concepts. The generation of concept features usually involves
two stages

of processing: (1) concept learning and (2) concept detection and score
mapping

to produce the required feature. The concept detector may take any forms

includin
g Support Vector

Machines (SVM). Among this

machine learning

algorithms,
the supervised learning methods, SVM are the most commonly

used algorithms.






2.2

Machine Learning and Design Challenges

Machine learning algorithms, SVM, are gaining attention in many

fie
lds. To bri
dge the semantic gap encountered in the multimedia content analysis,

the supervised learning methods are adopted to project the low
-
level feature space

to higher
-
level of semantic concept space. In order to do the mapping, there are

a

lot of researchers who take the strategy to combine local region concept information

to get the rich information that are hidden in one image. In this scheme,

a two
-
stage
image featu
re extraction is required. The fi
rst stage contains image

sampling, such
as
image partitioning or key point extraction, and classifying those

samples into different
concept classes. Then in the next stage, a new image representation

can be obtained
by manipulating the local region concept labels. This

method is becoming more an
d
more important and achieves good performance in

many applications.

In contrast to
the generative model, like GMM, that models the distribution

of the data of a given
class, discriminative models, on the other hand, let the data

speak for themselves, and
SVM is the
one of the most popular one
. Support

Vector Machine (SVM) is one of
the most powerfu
l techniques for real
-
world pat
tern recogniti
on and data mining
problems
. In recent years, SVM proposed

by Cortes and Vapnik has become the
state
-
of
-
the
-
art cl
assi
fie
r for some supervised

classi
fication problems , especially in
the fi
eld of multimedia content

analysis. SVMs are famous for their strong

g
eneralization guarantees derived

from the max
-
margin property, and for their ability
to use very high dimension
al

feature spaces usi
ng the kernel functions
.

The SVM
classi
fier fi
nds a hyperplane which separates two
-
class data with

maximal margin.
Since the data sets are not always linearly separable, the SVM

takes the kernel
method to deal with this problem . Ther
e have been a variety

of SVM software
packages that provide ef
fi
cient SVM implementation, and

LIBSVM

is the most
commonly adopted one.

Ever since the SVM classi
fier was proposed
, there have been
many

researchers devoted themselves into this area. Most of
the researchers focus on

the algorithm part to make l
arge learning practical
, extend the original

binary
classi
fi
er into multi
-
class , or even generalize the SVM classi
fi
er to

h
andle arbitrary
output type
.

Since the computations involved in the SV
M algorithm are similar to that

involved

in the Arti
fi
cial Neural Network (ANN), some comparisons have also be

made between the two algorithms. The development of ANN followed a heuristic

path, with applications and extensive experimentation preceding the
ory. In contrast,

the developmen
t of SVM involved sound theory fi
rst, then implementation

and
experiments. A signi
fi
cant advantage of SVM is that while ANN can suffer

from
multiple local minima, the solution to an SVM is global and unique. The

reason that
SVM often outperforms ANN in practice is that SVM can deal with

bigger problems
than ANN,

and SVM is less prone to over fitting
.

When the problem to deal with is
getting more complex,

say, the dimension is quite large, then the SVM algorithm
often achieve
s

the state
-
of
-
the
-
art performance.
However, there is a signifi
cant lack of
hardware architecture implementations

for SVM classifi
er for solving the real
world
problems.




2.3

An Introductory Example

Suppose we are given empirical data



Here, the domain X

is some non
-
empty set that the patterns xi

are taken from .T
he



are called

labels or targets.

Unless stated otherwise, indices

i and j will always be
understood to run over the training set,

i.e.
i
;

j
,1;...;m.
Note that we have not made
any assumptio
ns on the domain X other than it being a set. In

order to study the
problem of learning, we need additional structure. In learning, we want to be

able to
generalize to unseen data points. In the case of pattern recognition, this means that
given

some new p
attern x

X; we want to predict the corresponding y

{
±
1
}
: By this
we mean,

loosely speaki
ng, that we choose y such that (
x ,
y
)

is in some sense similar
to the training

examples. To this end, we need similarity measures in X and in
{
±
1
}

The

latter is easy, as two

target values can only be identical or di

erent. For the
former, we require a similarity measure

i.e. a function that, given two examples x and




x

; returns a real number characterizing their

similarity. For reasons that will be
come
clear later, the function k is called a ke
rnel
.A type of similarity measure that is of
particular mathematical appeal are dot products. For

instance, given two vectors

x

;

x


𝑅
𝑁
the canonical dot product is defined as




Here,
(

)

denotes the ith entry of x.
The geometrical interpretation of this dot

p
roduct is that it computes the cosine of the angle

between the vectors x and x’,
provided they are normalized to length 1. Moreover, it allows

computation of the
length of a vector x
as






,

and of the distance between two vectors as

the length
of the di

erence vector. Therefore, being able to compute dot products mounts to

being able to carry out all geometrical constructions that can be formulated in terms of
angles,

lengths and d
istances.

Note, however, that we have not made the assumption
that the patterns live in a dot product

space. In order to be able to use a dot product as
a similarity measure, we therefore first need to

transform them into some dot product
space

H; which nee
d not be identical to

𝑅
𝑁
.

To this end

,

we use a map






The space

is

called a feature space. To summarize, there are three benefits to
transform the

data into

1.

It lets us define a similarity measure from the dot product in
,




2.

It allows us to deal

with the patterns geometrically, and thus lets us study learning

algorithm using linear algebra and analytic geometry.

3.

The freedom to choose the mapping

will enable us to design a large variety of
learning algorithms. For instance, consider a situation where the inputs already
live in a dot product space. In that case, we could directly define a similarity
measure as the dot product. However, we might stil
l choose to first apply a
non
-
linear map

to change the

representation into one that is more suitable for a
given problem and learning algorithm.

We are now in the position to describe a pattern recognition learning algorithm that is

arguable one of the si
mplest possible. The basic idea is to compute the means of the
two classes

in feature space,







where
𝑚
+

and

𝑚


are the number of examples with positive and negative labels,
respectively (see

Figure 1). We then assign a new point x to the class whose

mean is
closer to it. This geometrical

construction can be formulated in terms of dot products.
Half
-
way in between

+

and



lies the

point

We compute the class
of x by checking whether the vector connecting c

and x encloses an angle smaller

than
p=2

with the vector
connecting the class

means, in other words




Here, we have defined the o

set



It will be proved instructive to rewrite this expression in terms of the patterns



in the input

domain

To this end, note that we do not have a dot product in
,

all
we have is the similarity

measure k (cf. (5)). Therefore, we need to rewrite everything
in terms of the kernel k evaluated

on input patterns. To this end, substitute (6) and (7)
into (8) to get

the decision function







Similarly, the o

set becomes





Let us consider one well
-
known special case of this type of classifier. Assume that the
class

means have the same distance to the origin (hence b
=

0), and that k can be
viewed as a density,

i
.e. it is positive and has integral 1,




In order to state this assumption, we have to require that we can define an integral on
If the above holds true, then (10) corresponds to the so
-
called Bayes decision
boundary

separating the two classes, subject
to the assumption that the two classes
were generated from

two probability distributions that are correctly estimated by the
Parzen windows estimators of

the two classes







Given some point x; the label is then simply computed by checking which of the
two,
𝑝
1
(

)

or
𝑝
2
(

)

,
is larger, which directly leads to (10). Note that this decision is the
best we can do if we

have no prior information about the probabilities of the two
classes. For further details, see [1].

Classifier (10) is quite close to the
types of
learning machines that we will be interested in. It is

linear in the feature space, and
while in the input domain, it is represented by a kernel expansion

in terms of the
training points. It is example
-
based in the sense that the kernels are cent
e
red on

the
training examples, i.e. one of the two arguments of the kernels is always a training
example.

The main points that the more sophisticated techniques to be discussed later
will deviate from

(10) are in the selection of the examples that the kerne
ls are
cent
e
red on, and in the weights that

are put on the individual data in the decision
function. Namely, it will no longer be the case that

all training examples appear in the
kernel expansion, and the weights of the kernels in the

expansion will no lo
nger be
uniform. In the feature space representation, this statement

corresponds to saying that
we will study all normal vectors

w

of
decision hyperplanes that can be

represented as
linear combinations of the training examples. For instance, we might want
to

remove
the influence of patterns that are very far away from the decision boundary, either
since

we expect that they will not improve the generalization error of the decision
function, or since

we would like to reduce the computational cost of evaluating

the
decision function (cf. (10)).

The hyperplane will then only depend on a subset of
training examples, called support vectors.







3

Learning Pattern Recognition From Example

With the above example in mind, let us now consider the problem of pattern
r
ecognition in a

more formal setting [5, 6], following the introduction of Schoolkopf .

et al.

[7]. In two
-
class pattern

recognition, we seek to estimate a function



based on input

output training data (1). We assume that the data were generated
independen
tly

from some unknown (but fixed) probability distribution P
( x , y ).

Our
goal is to learn a function

that will correctly classify unseen examples
(x ,y ),

i.e. we
want

f(x) =
y for examples
(x , y)

that

were also generated from P
(x , y).
If we put no
restriction on the class of functions that we choose our estimate f from, however,

even
a function which does well on the training data, e.g. by satisfying f
(


) =



for all i
=
1
,...,m,

need not generalize well to unseen examples. To see this, note th
at for each
function f

and any test set

(

1
̅
̅
̅
,

1
̅
̅
̅
)
,
.
.
.
,
(

𝑚
̅
̅
̅
̅
,

𝑚
̅
̅
̅
̅
)
𝜖
𝑅
𝑁
×
{
±
1
}

,satisfying
{

̅
1
̅
,
.
.
.
,

̅
𝑚
̅
}

{

1
,

,

𝑚
}
=
{
}

,
there exists another function

𝑓


such that

𝑓

(


)
=




for all i = 1 , … ,m , yet
𝑓

(

̅

)


̅


for all
i

= 1 , …, m .
As we are
only giv
en the training data, we have no means of selecting which

of the two

f
unctions (and hence which of the completely di

erent sets of test label predictions) is

preferable. Hence, only minimizing the training error (or empirical risk),





does

not imply a small test error (called risk), averaged over test examples drawn
from the

underlying distribution P
(x ,
y
),



Statistical learning theory [5, 6, 8, 9], or VC (Vapnik

Chervonenkis) theory, shows
that it is

imperative to restrict the class of
functions that

f is chosen from to one which
has a capacity that

is suitable for the amount of available training data. VC theory
provides bounds on the test

error. The minimization of these bounds, which depend on
both the empirical risk and the

capacity
of the function class, leads to the principle of

structural risk minimization [5]. The best
-
known capacity concept of VC theory is the
VC dimension, defined as the largest number h of

points that can be separated in all
possible ways using functions of the
given class. An example

of a VC bound is the
following: if h
<
m is the VC dimension of the class of functions that the

learning
machine can implement, then for all functions of that class, with a probability of at

least 1
-
η
; the bound




holds, w
here the c
onfidence term f is defined as




Tighter bounds can be formulated in terms of other concepts, such as the annealed VC
entropy

or the Growth function. These are usually considered to be harder to evaluate,
but they play a

fundamental role in the conceptual
part of VC theory [6]. Alternative
capacity concepts that can

be used to formulate bounds include the fat shattering
d
imension [10].


The bound (18) deserves some further explanatory remarks. Suppose we wanted
to learn a

‘dependency’ where P(x ,
y
)

=

P
(x)

P(
y
)

,
i.e. where the pattern x contains no
information about

the label y; with uniform P
(y).

Given a training sample of fixed size,
we can then surely come up

with a learning machine which achieves zero training
error (provided we have no examples

contradic
ting each other).

H
owever, in order to
reproduce the random labelling, this machine

will necessarily require a large VC
dimension h: Thus, the confidence term (19), increasing

monotonically with h; will be
large, and bound (18) will not support possible hop
es that due to

the small training
error, we should expect a small test error. This makes it understandable how

(18) can
hold independent of assumptions about the

underlying distribution P(x , y)
:

it always

holds (provided that h<
m), but it does not alway
s
make a non
-
trivial prediction
-
a
bound on

an error rate becomes void if it is larger than the maximum error rate. In
order to get non
-
trivial

predictions from (18), the function space must be restricted
such that the capacity (e.g. VC

dimension) is small en
ough (in relation to the available
amount of data).



4

Hyperplane Classifiers

In the present section, we shall describe a hyperplane learning algorithm that can
be performed

in a dot product space (such as the feature space that we introduced
previously).
As described in

the previous section, to design learning algorithms, one
needs to come up with a class of

functions whose capacity can be computed.

Vapnik and Leaner [11] considered the class of hyperplanes



correspondin
g to decision
functions



and

proposed a learning algorithm for separable problems, termed the generalized
portrait, for

constructing f from empirical data. It is based on two facts. First, among
all hyperplanes

separating the data, there exists a unique one yielding the maximum
margi
n of separation

between the classes,

Second, the capacity decreases with increasing margin.

To construct this optimal hyperplane (cf. Figure 2), one solves the following
o
ptimization

problem:




A way to solve (23) is through its Lagrangian dual:



w
here



The Lagrangian L has to be minimized with respect to the primal variables w and b
and

maximized with
respect to the dual variables
𝛼

.

For a non
-
linear problem like
(23), called the

primal problem, there are several closely related problems of which
the Lagrangian dual is an

important one. Under certain conditions, the primal and dual
problems have the same optimal

objective values. Therefore, we can instead solve the
dual which may b
e an easier problem than

the primal.


In part
icular, we will see in Section 5

that when working in feature spaces, solving

the dual may be the only way to train SVM.


Let us try to get some intuition for this primal

dual relation. Assume
(

̅
,

̅
)

is an
op
timal

solution of the primal with the optimal objective value



Thus, no
(w , b)

satisfies



With (26), there is
α
>0 such that for all w ,
b



We do not provide a rigorous proof here but details can be found in, for example,
Reference

[13]. Note that for
general convex programming this result requires some
additional conditions

on constraints which are now satisfied by our simple linear
inequalities.

Therefore, (27) implies



On the other hand, for any
α

,



so



Therefore, with (28), the inequality in (29) becomes an equality. This property is
the strong

duality where the primal and dual have the same optimal objective value. In
addition, putting

(

̅
,

̅
)
into (27), with
𝛼
𝑖
̅

0 and



which is u
sually called the
complementarity condition.

To simplify the dual, as
L(w

,

b,
α
)

is convex when
α
is fixed, for any given
α
,




l
eads
to



and



As

α

is now given, we may wonder what (32) means. From the definition of the
L
agrangian, if


we can decrease

in
L(w,b,
α
)

as much as we want. Therefore, by

substituting (33) into (24), the dual problem can be written as

As



is definitely not the maximal objective value of the dual, the dual optimal
solution does

not happen when
Therefore, the dual problem is
simplified to

nding multipliers

𝛼


which

This is the dual SVM problem that we usually refer to. Note that (30), (32), and
𝛼


>
=0 for all i,

are called the Karush

Kuhn

Tucker (KKT) optimality conditions of the
primal problem.

Except an abnormal situation where all o
ptimal
𝛼


are zero, b can be
computed using (30).

The discussion from (31) to (33) implies that we can consider a
di

erent form of dual

problem:




Thi
s is the so
-
called Wolfe dual for convex optimization, which is a very early work in
duality

[14]. For

convex and di

erentiable problems, it is equivalent to the

L
agrangian dual though the

derivation of the Lagrangian dual more easily shows the
strong duality results. Some notes

about the two duals are

in, for example, [15,
Section 6
.4].

Following the
above discussion, the hyperplane decision function can be written
as




The solution vector

w has an expansion in terms of a subset of the training patterns,
namely

those patterns whose

𝛼


is non
-
zero, called support vectors. By (30), the
Support Vector
s lie on

the margin (cf. Figure 2). All remaining examples of the
training set are irrelevant: their

constraint (23) does not play a role in the
o
ptimization,
and they do not appear in expansion

(33). This nicely captures our intuition of the
problem: as t
he hyperplane (cf. Figure 2) is

completely determined by the patterns
closest to it, the solution should not depend on the other

examples.

The structure of the optimization problem closely resembles those that typically
arise in

Lagrange’s formulation of m
echanics. Also there, often only a subset of the
constraints become

active. For instance, if we keep a ball in a box, then it will
typically roll into one of the corners.

The constraints corresponding to the walls which
are not touched by the ball are irre
levant, the

walls could just as well be removed.

Seen in this light, it is not too surprising that it is possible to give a mechanical
interpretation

of optimal margin hyperplanes [16]: If we assume that each support
vector



exerts a

perpendicular forc
e of size
𝛼


and sign



on a solid plane sheet
lying along the hyperplane, then

the solution satisfies the requirements of mechanical
stability. Constraint (32) states that the

forces on the sheet sum to zero; and (33)
implies that the torques also sum

to zero, via




×
(
𝑦
𝑖
𝛼
𝑖
𝒘

𝒘

)

=
𝒘
×
(
𝒘

𝒘

)
=
0
.

There are theoretical arguments supporting the good generalization performance
of the

optimal hyperplane [5, 8, 17

19]. In addition, it is computationally attractive,
since it can be

constructed by solving a

quadratic programming problem.






5

Optimal Margin Support Vector Classifiers

We now have all the tools to describe support vector machines [1, 6]. Everything
in the last

section was formulated in a dot product space. We think of this space as the
feature space
described in Section 1. To express the formulas in terms of
the input
patterns living in X ,
we

thus need to employ (5), which expresses the dot product

of
bold face feature vectors x ,
x

in

terms of the kernel
k evaluated on input patterns x ,
x
’ ,



This can be done since all feature vectors only occurred in dot products. The weight
vector (cf.(33)) then becomes

an expansion in feature space,
and will

thus typically no
longer correspond

to the image of a single vector from input space. We thus obtain
decision functions of the more

general form (cf. (38))






and the following quadratic program (cf. (35)):

Working in the feature space somewhat forces

us to solve the dual problem
instead of the

primal. The dual problem has the same number of variables as the
number of training data.

However, the primal problem may have a lot more (even
infinite) variables depending on the

dimensionality of the feature s
pace (i.e. the length
of

Φ
(
x
)

). Though our derivation of the dual

problem in Section 3 considers
problems in finite
-
dimensional spaces, it can be directly extended

to problems in
Hilbert spaces [21].




6

Kernels

We now take a closer look at the issue of the

similarity measure, or kernel, k: In
this section,

we think of X as a subset of the vector space
𝑅
𝑁
,
(
𝑁𝜖
Ν
)

,
endowed with
the canonical dot

product (3).

6.1

Product features

Suppose we are given patterns x
ϵ

𝑅
𝑁

where most information is contained in
the
d

th order

products (monomials) of entries
[

]

of x,



where
𝑗
1

,

,
𝑗
𝑑
𝜖

{
1

,

,
𝑁
}
.

In that case, we might prefer to
extract

these product
features, and

work in the feature space


of all products of
d

entries. In visual
recognition problems, where

images are often represented as vectors, this would
amount to extracting features which are

products of individual pixels.

For instance, in
𝑅
2
,

we can collect all monomial feature extractors of degree 2 in
the nonlinear map




This approach works fine
for small toy examples, but it fails for realistically sized
problems: for

N
-
dimensional input patterns, there exist




di

erent monomials (43), comprising a feature space

of dimensionality

For instance,

already 16
x
16 pixel input images and a monomial
degree d
=

5 yield a
dimensionality of
10
10
.

In certain cases described below, there exists, however, a way of computing dot
products in

these high
-
dimensional feature spaces without explicitly mapping into
them: by means of kernels

non
-
linear in the input

space
𝑅
𝑁
.
Thus, if the subsequent
processing can be carried out using dot

products exclusively, we are able to deal with
the high dimensionality.

6.2

Polynomial f
eature spaces induced by kernels

In order to compute dot products of the form
(
Φ
(
x
)

Φ
(
x′
)
)

we employ kernel
representations of

the form


which allow us to compute the value of the dot product in

without having to
carry out the

map

Φ

.
This method was used by Boser et al. to extend the generalized
portrait hyperplane

classifier [8] to non
-
linea
r support vector machines [4]. Aizerman
et al. called


the linearization

space, and used in the context of the potential
function classification method to express the dot

product between elements of

in
terms of elements of the input space [3].


What does

k look like for the case of polynomial features? We start by giving an
example [6]

for N
=

d
=
2: For the map



dot
p
roducts in

take the form

i.e. the desired kernel k is simply the square of the dot product in input space. Note
that it is

possible to modify
(
x
,
x′
)
𝑑
such that it maps into the space of all monomials
up to degree d[6],

defining



6.3

Examples of
kernels

When considering feature maps, it is also possible to look at things the other way
around, and

start with the kernel. Given a ker
nel function satisfying a mathematical
condition termed

positive definiteness, it is possible to construct a feature space such
that the kernel computes the

dot product in that feature space. This has been brought
to the attention of the machine learning

co
mmunity by [3, 4, 6]. In functional analysis,
the issue has been studied under the heading of

reproducing kernel Hilbert space
(RKHS).

Besides (50), a popular choice of kernel is the Gaussian radial basis function
[3]



An illustration is in Figure 3. For
an overview of other kernels, see [1].

7

Multi
-
class SV Classifiers

Support vector machines were origin
ally designed for binary classifi
cation.

How
to effectively extend it for multi
-
class classi
fi
cation is still an ongoing research

issue.

C
urrently there

are mainly two types of approaches for multi
-
class

SVM. One is
one
-
against
-
all while the other is one
-
against
-
one as shown in Fig.4.1 and Fig. 4.2.
Based on the training process or solving the optimization problem

differently, it can
also be divided into
two kinds of approaches. One is “all
-
together”

methods that
consider all the training data at once, the other is by constructing

and
combining
several binary classifi
ers
.

7.1

One
-
against
-
all Method

In the case of
combining several binary classifi
ers, the
one
-
against
-
all method

constructs
K
SVM models where
K
is the number of classes
. The
i
th SVM is

trained with all of the examples in the
i
th class with positive labels, and all other

examples with negative labels. The problem fo
rmulation is as follows.
Given

l

training data (

1
,

1
), ..., (



,


), where





𝑅
𝑑
, j = 1, ...,
l

and





1, ...,K is
label of


. The mth SVM classifi
er solves the following problem:




(52)

where the training data



can be mapped to the higher
dimensional space the

using
the function


as shown in Fig. 4.3, and
C
is the penalty parameter to adjust

the

n
umber of training errors. Minimizing
1
2
(

𝑚
)


𝑚

is equivalent to maximize

the

m
argin between two groups of data as shown in Fig. 4.4 (a) an
d (b). From the

equations above, it can be observed that the main goal of SVM is to
fi
nd a balance

between the regularization term
1
2
(

𝑚
)


𝑚

and the training errors as illustrated

in
Fig. 4.4 (c).

After solving 4.1, the parameters can be obtained, and

thus there will be
K

decision functions:










The classifi
cation result, or the label of
x
can be obtained by
fi
nding the decision

function with the largest value:




(
53
)

(
54
)




(54)
can

be solved by using the Lagrange multiplier theory in its dual form
:






subject to





where




Only a few
𝛼


will be greater than zero. The corresponding xis are exactly the

support vectors, which lie on the margin.

Introducing the parameters obtained above into 4.3, the decision function for

the mth classifi
er can be expressed as the function of support vectors shown as

follows:





where
K
(


,
x
)
is the kernel function obtained from

:




(
55
)

(
56
)

(
57
)

(
58
)

(
59
)


And the label of
x
can be found
by:




The kernel function may take many forms, and the most commonly used are the

linear kernel, polynomial kernel, and the “Radial Basis Function” (RBF) kernel

shown as follows respectively:



(
60
)







7.2

One
-
against
-
one

Method

Likewise, the one
-
against
-
one method constructs

2


=
K
(
K

1
)
/
2 binary
classi
fi
ers where each one is trained on data from two classes. The most common

strategy to make the final classifi
cation decision is to vote for the
corresponding

class
in one of the
K
(
K

1
)
/
2 binary classifi
ers and classify the data to the class

with the
largest votes. This strategy is also called the “Max Wins”

strategy
.

That is, for the
classifi
er constructed from class
i
and class
j
, the following

process

is conducted to
vote for the candidate class: if
sign
(
(


)

Φ
(

)



)
says
x

is in the
i
th class, then
the vote for the
i
th class is added by one. Otherwise, the

j
th is increased by one.

7.3

Considering All Data At Once

Method

Different
from cons
tructing multi
-
class SVM classifi
er by combining binary

c
lassi
fi
ers, there also exist some works that handle the multi
-
class problems by

solving

one single optimization problem. That is, instead of iteratively sampling the

d
ata

with two differen
t labels, positive and negative, these kinds of methods consider

a
ll the training data at once by solving one problem. The concept is very similar to

the
one
-
against
-
all in that it also constructs
K
decision functions where each decision

function separates

vectors of the corresponding class from the other vectors.

Various
methods can be found in. Generally, the problem formulation

is as follows:








Then the decision function is

:





(
61
)

(
62
)

(
63
)

which is the same as
(54)

of the one
-
against
-
all method.


Hsu et al. implemented the multi
-
class SVM

classifi
er algorithms in a tool

called
“BSVM” using the one
-
against
-
one method, and two considering all

data at once

m
ethods with the one solving a single optimization problem,

and the other using,

respectiv
ely. The latter two methods are called “KBB”

and “SPOC” in the BSVM

tool.
Assume there are
L
support vectors, and the

decision function of “KBB” is:




where









From the above constraints, the term


𝑚
𝐴


𝛼

𝑚

can be expressed as a new
parameter
𝛼

𝑚


as follows:







The decision function of “SPOC” is as follows:




where



(
64
)

(
65
)

(
66
)

(
67
)

(
68
)

From the above constraints, it is clear that the lower and upper bound of
𝛼

𝑚

is:




By comparing the two decision functions,
(64)

and
(67)
, and their constraints,

it
is clear that the “SPOC” method is more hardware
-
friendly and the reasons are

in

t
hree
-
folds. One is the range of the weighting
𝛼

𝑚
is smaller and more consistent

compared with the weighting
𝛼

𝑚


, another is the decision function is of “SPOC” is

simpler that requires no bias term, and the other is the number of support vectors

obtained from the “SPOC” method is smaller in our experimental results.

I
t is
reported

that the considering all data at once

methods in general need fewer support
vectors although it might sacrifi
ce some

classifi
cation accuracy. Because the number
of support vectors affects the memory

size required to keep them and the computation
iteration
s required to compute

the decision function, the considering all data at once
method will be the candidate

in the proposed hardware architecture. Since the
considering all data at once

m
ethods also constructs K classifi
ers and differ from the
one
-
against
-
a
ll method

only in the functions that the objective function is subjected to.
The name of “one
-
against
-
all” method will stand for the considering all data at once
method
.



(
69
)

8

Applications

8.1

LIBSVM

Researchers have applied
SVM on di

erent applications. Some of
them feel that
it is easier

and more intuitive to deal with
v

ϵ

[
0 , 1
]
than C

ϵ

[0 ,

).

Here, we briefly
summarize some work

which use LIBSVM to solve SVM.

In Reference [39],

r
esearchers from HP Labs discuss the topics of personal email agent. Data

classi

cation is an important component for which the authors use n
-
SVM because
they think

‘the n parameter is more intuitive than the C parameter’.

Martin et al. [40]
applies machine learning methods to detect and localize boundaries of

natural images.

8.2

Experi
m
ent

(a)

System overview
: (
Cardiac dysrhythmia detector
)




SVM

Output:

Classified data

1:

abnormal

0: normal

Input:

ECG signal

(
heart beat
)

with 5 features

(b)

Experiment Result

Use LIBSVM to construct model file , and send to my .cpp

file .The following is the
related files
:
(for simplification , here only shows parts of the files)



Model file

Test file



The red
-
squared data in test file is the ground truth label of the heart beat , and
the red
-
squared data in

the result file is the result label classified by SVM . If the two
labels are the same , then match , or the error would increase by one.

The follow
ing is
the simulation result:


Result file

Performance measurement:

Precision = tp / tp+fp

Recall = tp / tp+fn



(
implemented by C/C++
)



9

Conclusion

One of the most appealing features of kernel algorithms is the solid foundation
provided by

both statistical learning theory and functional analysis. Kernel methods
let us interpret (and

design) learning algorithms geometrically in feature spaces
non
-
linearly related to the input

space, and combine statistics and geometry in a
promising way. Kernels provide an elegant

framework for studying three fundamental
issues of machine learning:


*
Similarity measures
-

the kernel can be viewed as a (non
-
linear) similarity
measure, and

should ideally incorporate prior knowledge about the problem at hand

* Data representation
-

as described above, kernels induce representations of the
data in a

linear

space

* Function class

-

due to the representer theorem, the kernel implicitly also
determines the

function class which is used for learning.

Support vector machines have been one of the major kernel methods for data
classification.

Its original form
requires a parameter C

[
0
,

)

which controls the
trade
-
o


between the

classifier capacity and
the training errors. Using the
v
-
parameterization, the parameter C is

replaced by a parameter
v

[
0
,
1
]
.

10

Reference

1.

Schoolkopf B, Smola

AJ. . Learning with Kernels. MIT Press: Cambridge, MA,
2002.

2.

Mercer J. Functions of positive and negative type and their connection with the
theory of integral equations.

Philosophical Transactions of the Royal Society of
London, Series A 1909; 209:41
5

446.

3.

Aizerman MA, Braverman EM, Rozono "

eer LI. Theoretical foundations of the
potential function method in pattern "

recognition learning. Automation and
Remote Control 1964; 25:821

837.

4.

Boser BE, Guyon IM, Vapnik V. A training algorithm for op
timal margin
classifiers. In Proceedings of the 5
th

Annual ACM Workshop on Computational
Learning Theory, Haussler D (ed.), Pittsburgh, PA, July 1992. ACM

Press: New
York, 1992; 144

152.

5.

Vapnik V. Estimation of Dependences Based on Empirical Data (in Ru
ssian).
Nauka: Moscow, 1979 (Englishtranslation: Springer: New York, 1982).

6.

Vapnik V. The Nature of Statistical Learning Theory. Springer: New York, 1995.

7.

Schoolkopf B, Burges CJC, Smola AJ. . Advances in Kernel Methods}Support
Vector Learning. MIT
Press:Cambridge, MA, 1999.

8.

Vapnik V, Chervonenkis A.Theory of Pattern Recognition (in Russian). Nauka:
Moscow, 1974 (German translation:Theorie der Zeichenerkennung, Wapnik W,
Tscherwonenkis A (eds). Akademie
-
Verlag: Berlin, 1979).

9.

Vapnik V. Statis
tical Learning Theory. Wiley: NY, 1998.

10.

Alon N, Ben
-
David S, Cesa
-
Bianchi N, Haussler D. Scale
-
sensitive dimensions,
uniform convergence, andlearnability. Journal of the ACM 1997; 44(4):615

631.

11.

Vapnik V, Lerner A. Pattern recognition using gener
alized portrait method.
Automation and Remote Control 1963;24:774

780.

12.

Schoolkopf B. . Support Vector Learning. R. Oldenbourg Verlag: Muunchen,
1997. Doktorarbeit, Technische .Universitaat Berlin. Available from
h
ttp://www.kyb.tuebingen.mpg.de/~bs .

1
3.

Bazaraa MS, Sherali HD, Shetty CM. Nonlinear Programming: Theory and
Algorithms(2nd edn). Wiley: New York,1993.

14.

Wolfe P. A duality theorem for non
-
linear programming. Quarterly of Applied
Mathematics 1961; 19:239

244.

15.

Avriel M. Nonlinear Prog
ramming. Prentice
-
Hall Inc.: Englewood Cli

s, NJ,
1976.

16.

Burges CJC, Schoolkopf B. Improving the accuracy and speed of support vector
learning machines. In . Advances inNeural Information Processing Systems, vol.
9. Mozer M, Jordan M, Petsche T (eds).
MIT Press: Cambridge, MA,1997;
375

381.

17.

Bartlett PL, Shawe
-
Taylor J. Generalization performance of support vector
m
achines and other pattern classifiers.In Advances in Kernel Methods}Support
Vector Learning, Schoolkopf B, Burges CJC, Smola AJ (eds). MI
T
Press: .Cambridge, MA, 1999; 43

54.

18.

Smola AJ, Bartlett PL, Schoolkopf B, Schuurmans D. . Advances in Large
Margin Classifiers. MIT Press: Cambridge,MA, 2000.

19.

Williamson RC, Smola AJ, Schoolkopf A. Generalization performance of
regularization net
works and support vector .machines via entropy numbers of
compact operators. IEEE Transactions on Information Theory 2001; 47(6):

2516

2532.

20.

Wahba G. Spline Models for Observational Data. CBMS
-
NSF Regional
Conference Series in Applied Mathematics,vol.

59. Society for Industrial and
Applied Mathematics: Philadelphia, PA, 1990.

21.

Lin C
-
J. Formulations of support vector machines: a note from an optimization
point of view. Neural Computation

2001; 13(2):307

317.

22.

Cortes C, Vapnik V. Support vector n
etworks. Machine Learning 1995;
20:273

297.

23.

Schoolkopf B, Smola AJ, Williamson RC, Bartlett PL. New support vector
a
lgorithms. . Neural Computation 2000;12:1207

1245.

24.

Crisp DJ, Burges CJC. A geometric interpretation of n
-
SVM classifiers. In
Advances in Neural Information

Processing Systems, vol. 12. Solla SA, Leen TK,
Muuller K
-
R (eds). MIT Press: Cambridge, MA, 2000.

25.

Bennett KP, Bredensteiner EJ. Duality and geometry in SVM classifiers. In
Proceedings of the 17th InternationalConference
on Machine Learning, Langley
P (ed.), San Francisco, CA. Morgan Kaufmann: Los Altos, CA, 2000;57

64.

26.

Chang C
-
C, Lin C
-
J. Training n
-
support vector classifiers: theory and algorithms.
Neural Computation 2001;13(9):2119

2147.

27.

Michie D, Spiegelhalter

DJ, Taylor CC. Machine Learning, Neural and Statistical
Classification. Prentice
-
Hall:Englewood Cli

s, NJ, 1994. Data available at
http://www.ncc.up.pt/liacc/ML/statlog/datasets.html

28.

Steinwart I. Support vector machines are universally consistent. Jou
rnal of
Complexity 2002; 18:768

791.

29.

Steinwart I. Sparseness of support vector machines. Technical Report, 2003.

30.

Steinwart I. On the optimal parameter choice for n
-
support vector achines.IEEE
Transactions on Pattern Analysis

and Machine Intellige
nce 2003;
25(10):1274

1284.

31.

Gretton A, Herbrich R, Chapelle O, Scho¨lkopf B, Rayner PJW. Estimating the
Leave
-
One
-
Out Error for

Classification Learning with SVMs. Technical Report
CUED/F
-
INFENG/TR.424, Cambridge University Engineering Department,
2001.

32.

Lin C
-
J. On the convergence of the decomposition method for support vector
machines. IEEE Transactions on

Neural Networks 2001; 12(6):1288

1298.

33.

Joachims T. Making large
-
scale SVM learning practical. In Advances in Kernel
Methods}Support Vector
Learning

,

Schoolkopf B, Burges CJC, Smola AJ (eds).
MIT Press: Cambridge, MA, 1999; 169

184. .

34.

Platt JC. Fast training of support vector machines using sequential minimal
optimization. In Advances in Kernel

Methods}Support Vector Learning,
Schoolkopf

B, Burges CJC, Smola AJ (eds). MIT Press: Cambridge, MA, 1998. .

35.

Hsu C
-
W, Lin C
-
J. A comparison of methods for multi
-
class support vector
machines.IEEE Transactions on Neural

Networks 2002; 13(2):415

425.

36.

Chung K
-
M, Kao W
-
C, Sun C
-
L, Lin C
-
J. De
composition methods for linear
support vector machines. Technical

Report, Department of Computer Science
and Information Engineering, National Taiwan University, 2002.

37.

Chang C
-
C, Lin C
-
J. LIBSVM: a library for support vector machines, 2001.
Software a
vailable at http://www.csie.ntu.edu.tw/ cjlin/libsvm

38.

Perez Cruz F, Weston J, Herrmann DJL, Schoolkopf B. Extension of the . n
-
SVM
range for classification. In Advancesin Learning Theory: Methods, Models and
Applications, vol. 190. Suykens J, Horvath G,

Basu S, Micchelli C,

Vandewalle J
(eds). IOS Press: Amsterdam, 2003; 179

196.

39.

Bergman R, Griss M, Staelin C. A personal email assistant.Technical Report
HPL
-
2002
-
236. HP Laboratories, PaloAlto, CA, 2002.

40.

Martin DR, Fowlkes CC, Malik J. Learning
to detect natural image boundaries
using brightness and texture. In

Advances in Neural Information Processing
Systems, vol. 14, 2002.

41.

Weston J, Chapelle O, Elissee


A, Schoolkopf B, Vapnik V. Kernel dependency
estimation. In . Advances in

Neural Infor
mation Processing Systems, Becker S,
Thrun S, Obermayer K (eds). MIT Press: Cambridge, MA, 2003:

873

880.