# Biobjective Approach for SVM-Based Binary Classication and Ordinal Regression

Τεχνίτη Νοημοσύνη και Ρομποτική

16 Οκτ 2013 (πριν από 4 χρόνια και 7 μήνες)

101 εμφανίσεις

Biobjective Approach for SVM-Based
Binary Classication and Ordinal Regression
Belen Martin-Barragan and Emilio Carrizosa
belen.martin@uc3m.es
September,2008
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Classication
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Classication
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Motivation
Example 1
predicted class
1 1
class 1
70% 5%
class 1
5% 20%
Example 2
predicted class
1 1
class 1
74% 1%
class 1
9% 16%
In both examples,misclassication is 10%
In Example 1,half of the misclassied objects belong to each
class.
In Example 2,most of the misclassied objects belong to class
1.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Well-dened costs
Example
Direct mailing for subscribing a magazine
Class 1:the person will subscribe
Class 1:the person will NOT subscribe
Costs
C cost of sending a mail
R reward of getting a new subscriber
Costs
predicted
1 1
1
0 C
1
R C 0
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Well-dened costs
Example
Direct mailing for subscribing a magazine
Class 1:the person will subscribe
Class 1:the person will NOT subscribe
Costs
C cost of sending a mail
R reward of getting a new subscriber
Costs
predicted
1 1
1
0 C
1
R C 0
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Well-dened costs
Example
Direct mailing for subscribing a magazine
Class 1:the person will subscribe
Class 1:the person will NOT subscribe
Costs
C cost of sending a mail
R reward of getting a new subscriber
Costs
predicted
1 1
1
0 C
1
R C 0
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Unknown costs
Example
Medical diagnosis of cancer:
class 1:the person has cancer
class 1:the person is healthy
Two errors
Error of misclassifying a person with
cancer
Error of misclassifying a healthy
person
Costs
predicted
1 1
1
??
1
??
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Unknown costs
Example
Medical diagnosis of cancer:
class 1:the person has cancer
class 1:the person is healthy
Two errors
Error of misclassifying a person with
cancer
Error of misclassifying a healthy
person
Costs
predicted
1 1
1
??
1
??
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Unknown costs
Example
Medical diagnosis of cancer:
class 1:the person has cancer
class 1:the person is healthy
Two errors
Error of misclassifying a person with
cancer
Error of misclassifying a healthy
person
Costs
predicted
1 1
1
??
1
??
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Aim
We present a model which is a modication of Support Vector
Machines and is able to take into account such situation
Outline
Introduction
Binary classication:hard-margin.
Binary classication:soft-margin.
Ordinal Regression Problem.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Aim
We present a model which is a modication of Support Vector
Machines and is able to take into account such situation
Outline
Introduction
Binary classication:hard-margin.
Binary classication:soft-margin.
Ordinal Regression Problem.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
The classication problem
Notation
Set of objects
,(population)
Two classes:1;1
u 2
;u = (x
u
;c
u
)
x
u
2 IR
N
predictor variable
c
u
2 f1;1g class membership
DATA:I 
;training sample
8u 2 I;(x
u
;c
u
) are known
Notation:I
c
= fu 2 I:c
u
= cg
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Support Vector Machines
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Support Vector Machines
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Support Vector Machines
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Maximal margin
Denition
We dene the margin of I
(!;) = min
u2I
c
u
(!
>
x
u
+)
k!k
2
;
where k  k
2
denotes the Euclidean norm
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Maximal margin
Denition
We dene the margin of I
(!;) = min
u2I
c
u
(!
>
x
u
+)
k!k
2
;
where k  k
2
denotes the Euclidean norm
The problem
max (!;)
st:c
u

!
>
x
u
+

> 0 8u 2 I:
!2 IR
N
; 2 IR
(P)
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Maximal margin
Denition
We dene the margin of I
(!;) = min
u2I
c
u
(!
>
x
u
+)
k!k
2
;
where k  k
2
denotes the Euclidean norm
Cortes-Vapnik
min!
>
!
st:c
u

!
>
x
u
+

 1 8u 2 I:
!2 IR
N
; 2 IR
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Biobjective approach
Denition
Margin of class c

c
(!;) = min
u2I
c
c
u
(!
>
x
u
+)
k!)k
2
;for c = 1;1
Biobjective margin maximization
max

1
(!;);
1
(!;)

st:c
u

!
>
x
u
+

> 0 8u 2 I:
!2 IR
N
; 2 IR
(BP)
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Biobjective approach
Denition
Margin of class c

c
(!;) = min
u2I
c
c
u
(!
>
x
u
+)
k!)k
2
;for c = 1;1
Biobjective margin maximization
max

1
(!;);
1
(!;)

st:c
u

!
>
x
u
+

> 0 8u 2 I:
!2 IR
N
; 2 IR
(BP)
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Main result
Theorem (Characterization of the set of Ecient solutions)
The set of ecient solutions of the Biobjective Problem (BP) is
given by
f(^!;):j 
^
j < 1; > 0g
for (^!;
^
) optimal solution of Problem (CV).
In particular,
all ecient solutions are parallel to the SVM solution.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Main result
Theorem (Characterization of the set of Ecient solutions)
The set of ecient solutions of the Biobjective Problem (BP) is
given by
f(^!;):j 
^
j < 1; > 0g
for (^!;
^
) optimal solution of Problem (CV).
In particular,
all ecient solutions are parallel to the SVM solution.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Parallel hyperplanes
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Parallel hyperplanes
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Parallel hyperplanes
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Parallel hyperplanes
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Parallel hyperplanes
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
What does the denition of eciency mean?
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Non-separable case
What can be done in the non-separable case?
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Main result.Soft-margin approach.
Theorem (Characterization of the set of Ecient solutions)
The set of ecient solutions of the biobjective Problem (s-BP) is
given by
f(^!;
^
;):j 
^
j < 1; > 0g
for (^!;
^
;
^
) optimal solution of Problem (s-CV).
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Ordinal Regression.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Ordinal Regression:the problem
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Ordinal Regression:the problem
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Ordinal Regression:the problem
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Ordinal Regression:the problem
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Classes:c
u
2 f1;2;:::;Rg
Classes has a known natural order
Example:slight,moderate,severe,extreme pain
Two types of misclassication
Upgrading error:Missclassifying an object of class c
u
to class
c > c
u
:
Downgrading error:Missclassifying an object of class c
u
to
class c < c
u
:
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Denition
Margin is dened as
(!;) = min
c=1;2;:::;R
min
u2I
c
min

(!
>
x
u
+
c
)
k!k
;
(!
>
x
u

c1
)
k!k

;
where k  k stands for an arbitrary norm,
0
= 1;and 
R
= +1:
Ordinal regression hard-margin problem (ORHP)
Finding the hyperplane with largest margin:
min k!k
s.t.:

!
>
x
u

c

 1 8u 2 I
c
;8c = 1;2;:::;R 1

!
>
x
u

c

 1 8u 2 I
c+1
;8c = 1;2;:::;R 1
!2 IR
p
; 2 IR
R1
;
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Denition

+
(!;) = min
c=1;2;:::;R1
min
u2I
c
(!
>
x
u
+
c
)
k!k

(!;) = min
c=2;:::;R
min
u2I
c
(!
>
x
u

c1
)
k!k
Note:general norm
We extend the result for the case in which k!k is a general norm.
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
Biobjective Ordinal Regression Problem (BORP)
max f
+
(!;);

(!;)g
s.t.:

!
>
x
u

c

< 0 8u 2 I
c
;c = 1;2;:::;R 1

!
>
x
u

c1

> 0 8u 2 I
c
;c = 2;:::;R
!2 IR
p
; 2 IR
R1
;!6= 0:
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Main result.Ordinal Regression
Theorem (Characterization of the set of Ecient solutions)
The set of ecient solutions of the biobjective Problem (BORP) is
given by
n
!
1
;(
^
 )

: 2 (1;1); > 0;
o
where (!
1
;
^
) is the optimal solution of Problem (ORHP).
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Ordinal Regression
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model
Future research
ROC curves using crossvalidation
Numerical results
Extensions to error per class
Extensions to multi-class problem
B.Martin-Barragan and E.Carrizosa
Biobjective margin maximization model