Definition of quadratic forms

nostrilswelderElectronics - Devices

Oct 10, 2013 (4 years and 29 days ago)

101 views


1

Quadratic forms

Definition of quadratic forms

Let {
x
1
,
x
2

, ...,
x
n
} be
n

(non random) variables. A
quadratic form

Q

is, by
definition, an expression such as :

Q

=
ij
a
ij
x
i
x
j


where the
a
ij

(the
coefficients

of the form) are real numbers.

So a quadratic form is a second degree, homogenous (no constant term) polynom
in the
x
i
.



Studying quadratic forms is made quite a bit easier by using matrix notation.


* The
ordered set of variables {
x
1
,
x
2

, ...,
x
n
} is considered as a vector
x

with
coordinates (
x
1
,
x
2

, ...,
x
n
). So we'll write :

x
'

= (
x
1
,
x
2

, ...,
x
n
)

with " ' " denoting transposition, which simply means that the
x
i

are written as a
row (as opposed to "a
s a column").


*
A

denotes the matrix whose general term is [
a
ij
].


* A quadratic form
Q

is then defined by the matrix equation :



Q =
x
'
Ax





A

is called the
matrix of the quadratic form
.

It is

easily shown that
A

may be assumed to be
symmetric

without loss of
generality.

-----

Quadratic forms are an important chapter of Linear Algebra. We will only retain
the aspects of quadratic forms that are useful to
the Statistician.

Expectation of a quadratic form

Let :


*
x

be a random vector with mean
µ

and
covariance matrix

.


*
Q =
x
'
Ax
. be a quadratic form.



We'll
show

that :



E
[
x
'
Ax
] =
tr
(
A
) +
µ'Aµ





This result makes no assumption about the nature of the distribution of
x
, with the
exception of the existence of the first two orders

moments.


2

Quadrati
c forms in a multivariate normal vector and the Chi
-
square distribution

Important parts of Statistics like

:


*
Multiple Linear Regression
,


*
Variance Analysis
,



extensively use the fact that if
x

is a
multivariate normal

v
ector :

x
~
N
(
µ
,
)

then

certain quadratic forms is
x

are Chi
-
square distributed.

Chi
-
square distribution

Let {
X
1
,
X
2

, ...,
X
n
} be
n

standard normal independent v
ariables :

X
i

~
N
(0, 1)

for all
i

Recall

that, by definition, the Chi
-
square distribution with
n

degrees of freedom is
the distribution of the variable
X
² defined a
s

:

X
² =
i

X
i
²


and

is usually denoted :

X
²

~
n


So the variable
X
² is
defined as a very special quadratic form in the
X
i

in which :


* There is no cross
-
term (
a
ij

= 0 for
i

j
),


* and all the
a
ii

are equal to 1.

The matrix

of the form is then just the identity matrix of order
n
, denoted
I
n
.

-----

Note that these
n

variables may conveniently

be represented by the single
multivariate normal standard variable :

x
~
N
(0,
I
n
)

Quadratic forms in a standard

multivariate normal varia
ble and the

Chi
-
square
distribution

It is then natural to ask whether there exist other quadratic forms in a
spherical

multinormal variable of unit variance that also have a Chi
-
square
distribution. In other words, do

matrices
P

exist such that :

x
'
Px

~
??


for
x
~
N
(0,
I
p
) ?



Recall that a symmetric matrix is said to be "idempotent" if
P
²

=
P
.

We'll show that a
necessary and sufficient

condition for
x
'
Px

to have a Ch
i
-
square distribution is that
P

be an

idempotent matrix. In addition, we'll show that
the rank of
P

and the number of degrees of freedom of the

distribution are the
n
equal.




3

In other words :



* If

x
~
N
(0,
I
n
)

*
Then
Q =
x
'
Px

~
r

if and only if

:


*
P
²

=
P


*
rank
(
P
) =
r




In fact, we'll show a bit more than that as we'll
consider variables
~
N
(
µ
,
I
)

that are not necessarily centered on the origin.

A symmetric idempotent matrix
P

or rank
r

is interpreted as the matrix of an
orthogonal

projection

operator on a
r
-
dimensional subspace. The projection of the
vector
x

is
Px
.

If

P
²

=
P
, then :

x
'
Px


=
x
'
P
²
x





=
x
'
PPx





=
x
'
P
'
Px

For
P

is symmetric



= (
Px
)'(
Px
)





and
x
'
Px

is therefore the
squared length

of the projection of

x

on the subspace.



The above result is then interpreted as follows :


* A quadratic form in a spherical, unit variance

multinormal vector is Chi
-
square distributed with
r

degrees of freedom if and only if it is the squared length
of the projection of

x

on some
r
-
dimensional subspace.








In the above illustration :


* The vector
x

has a spherical multinormal distribution with uni
t variance.


*
Px

is the projection of
x

on the
r
-
dimensional subspace
.


4


* The squared length of
Px

is distributed as
r
.

Quadratic forms in a general multinormal variable and the Chi
-
square
distribution

We then address the general issue of quadratic forms in a

multivariate normal
variable with an arbitrary covariance ma
trix

:

x
~
N
(
µ
,
)

We'll show that the quadratic form
Q

=
x'Ax

fol
lows a (non central) Chi
-
square
distribution with
r

degress of freedom
if and only if

both the following conditions
are satisfied

:



1)
A = A
A


2)
rank
(
A
) =
r





the value of the noncentrality parameter being

µ'Aµ
.

-----

So, in the general case, the interpretation in terms of the distribution of the
squared length of the projection of
x

on a subspace is lost, and
A

is
not

a
projection matrix.

-----

This situa
tion is encountered, for instance when studying the
Mahalanobis

distance.

Independence of two quadratic forms in a multivariate
normal vector, Craig's Theorem

Independe
nce of two random variables is a condition that makes life of a
Statistician a lot easier. In particular, many results pertaining to the
sum

and the
ratio

of two random variables explicitely assume that these variables are
independent. For example, the formal
definition

of Fisher's
F

distribution involves
the ratio of two independent
variables.

It is therefore n
atural to ask under what conditions two quadratic forms in a
multivariate normal vector
x

are independent.

Independence of two linear forms

We'll first
show

that two linear forms
A'x

et
B'x

in a multinormal vector
x

with
covariance matrix

are independent if and only if :



A'
B

= 0





This result is important by itself but, in addition, it will be needed later when we
address the issue of the independence of quadratic forms.


5

Independence of qua
dratic forms

We'll then
show

that :



* Let
x
~
N
p
(
µ
,
),

* And let
x'Ax

and
x'Bx

be two Chi
-
square distributed quadratic forms.

* Then these two forms are
independent

if and only if


A
B

= 0





I
n the special case where
x

is spherical, this condition becomes the simpler

AB

=
0.

-----

It is not assumed that the Chi
-
square distributions of the

quadratic forms are
central or have the same numbers of degrees of freedom.

Craig's Theorem

The above resul
t explicitely assumes that the quadratic forms are Chi
-
square
distributed. As it happens, this assumption is unnecessary and is introduced just as
to make the demonstration easier.

More generally :





* Let
x
~
N
p
(
µ
,
),

* And let
x'Ax

and
x'Bx

be two

quadratic forms.

* Then these two forms are
independent

if and only if


A
B

= 0





which is the same as before but with the assumption about the


distributions of
the quadratic forms

removed. The result is then known as
Craig's Theo
rem
,
whose demonstration lies beyond the bounds of this Glossary.

Cochran's Theorem

The results

about the nature of the distribution and the independence of quadratic
forms in multinormal vectors as described here are a key element of a most
important theo
rem in Statistics known as
Cochran's Theorem
, which is stated and
demonstrated
here
.

__________________________________________________________________




6



Tutorial 1



In

the first section of this Tutorial, we calculate the expectation of a quadratic
form with no assumption about the distribution of the variable. This section is
independent of the remainder of the Tutorial.

-----

We then establish a necessary and sufficien
t condition for a quadratic form
Q

:

Q =
x'Px

in a multivariate normal variable to be Chi
-
square distributed.




* We first address the special case of a spherical multinormal distribution with
unit variance (identity covariance matrix).


-

The "
sufficient" condition states that if
P

is a projection matrix with rank
r
,
then

Q

is Chi
-
square distributed with
r

degrees of freedom.


-

The "necessary" part states that if
Q

is Chi
-
square distributed with
s

degrees
of freedom, then
P

has to be a p
rojection matrix with rank
s
. The demonstration is
a bit more difficult, and the

moment generating function

(m.g.f.) of the quadratic
form will be of central importance.





* We then go over the general case (arbitrary covariance matrix) by identifying
a transformation that will turn the problem into the already solved

"special case"
problem, a common strategy when studying the multivariate normal distribution.
So the Tabl
e of Contents of the general case is quite short, as the brunt of the
work has already been done when solving the special case. Once the
transformation has been identified, only a small additional effort will be needed to
express the result as a function o
f
,

the covariance matrix of
x
.









QUADRATIC FORMS IN MULTINORMAL VECTORS

AND THE CHI
-
SQUARE DISTRIBUTION

Expectation of a quadratic form

-----------------
--------------

Multivariate normal distribution with identity

covariance matrix

A sufficient condition for a quadratic form in a multivariate normal vector

with identity covariance matrix to be Chi
-
square distributed



The condition is also necessary

First

form of the m.g.f. of the quadratic form


7

Second form of the m.g.f. of the quadratic form

Final result

Rank of
P

Eigenvalues of
P


P

is idempotent

Noncentrality parameter

General case : multivariate normal

distribution with arbitrary covariance matrix




TUTORIAL


_____________________________________________________________
_________
_____





Tutorial 2



We now address the issue

of the independence of two quadratic forms is a
multivariate normal vector.

We first solve the simpler

problem of the independence of two linear forms. This
result

will anyway be needed in the remainder of the Tutorial.

-----

We then move on to quadratic forms and establish a necessary and sufficient
condition for two such fo
rms to be independent. The reader won't be surprised to
see us first solve the case of quadratic forms in a spherically symmetric, unit
variance multinormal vector. The general case (arbitrary covariance matrix)

will
then be solved by identifying a transfo
rmation that turns the general case into the
already solved special case.

-----

Note that we always consider quadratic forms with Chi
-
square distributions. As it
turns out, this assumption is unnecessary and is introduced only to make the
demonstration eas
ier. The condition remains valid without it and then bears the
name of
Craig's Theorem
, which is very difficult to demonstrate.









INDEPEN
DENCE OF QUADRATIC FORMS

Independence of two linear forms in a multivariate normal vector

Independence of two Chi
-
square distributed

quadratic forms

in a standard multinormal vector

The condition is necessary

The condition is sufficient

The final result


8

In
dependence of two Chi
-
square distributed

quadratic forms

in a multinormal vector with arbitrary covariance matrix


TUTORIAL





___________________________________________________



Related readings :

Chi
-
square distribution


Multivariate normal distribution


Cochran's Theorem


Projection matrix