1
Quadratic forms
Definition of quadratic forms
Let {
x
1
,
x
2
, ...,
x
n
} be
n
(non random) variables. A
quadratic form
Q
is, by
definition, an expression such as :
Q
=
ij
a
ij
x
i
x
j
where the
a
ij
(the
coefficients
of the form) are real numbers.
So a quadratic form is a second degree, homogenous (no constant term) polynom
in the
x
i
.
Studying quadratic forms is made quite a bit easier by using matrix notation.
* The
ordered set of variables {
x
1
,
x
2
, ...,
x
n
} is considered as a vector
x
with
coordinates (
x
1
,
x
2
, ...,
x
n
). So we'll write :
x
'
= (
x
1
,
x
2
, ...,
x
n
)
with " ' " denoting transposition, which simply means that the
x
i
are written as a
row (as opposed to "a
s a column").
*
A
denotes the matrix whose general term is [
a
ij
].
* A quadratic form
Q
is then defined by the matrix equation :
Q =
x
'
Ax
A
is called the
matrix of the quadratic form
.
It is
easily shown that
A
may be assumed to be
symmetric
without loss of
generality.

Quadratic forms are an important chapter of Linear Algebra. We will only retain
the aspects of quadratic forms that are useful to
the Statistician.
Expectation of a quadratic form
Let :
*
x
be a random vector with mean
µ
and
covariance matrix
.
*
Q =
x
'
Ax
. be a quadratic form.
We'll
show
that :
E
[
x
'
Ax
] =
tr
(
A
) +
µ'Aµ
This result makes no assumption about the nature of the distribution of
x
, with the
exception of the existence of the first two orders
moments.
2
Quadrati
c forms in a multivariate normal vector and the Chi

square distribution
Important parts of Statistics like
:
*
Multiple Linear Regression
,
*
Variance Analysis
,
extensively use the fact that if
x
is a
multivariate normal
v
ector :
x
~
N
(
µ
,
)
then
certain quadratic forms is
x
are Chi

square distributed.
Chi

square distribution
Let {
X
1
,
X
2
, ...,
X
n
} be
n
standard normal independent v
ariables :
X
i
~
N
(0, 1)
for all
i
Recall
that, by definition, the Chi

square distribution with
n
degrees of freedom is
the distribution of the variable
X
² defined a
s
:
X
² =
i
X
i
²
and
is usually denoted :
X
²
~
n
So the variable
X
² is
defined as a very special quadratic form in the
X
i
in which :
* There is no cross

term (
a
ij
= 0 for
i
j
),
* and all the
a
ii
are equal to 1.
The matrix
of the form is then just the identity matrix of order
n
, denoted
I
n
.

Note that these
n
variables may conveniently
be represented by the single
multivariate normal standard variable :
x
~
N
(0,
I
n
)
Quadratic forms in a standard
multivariate normal varia
ble and the
Chi

square
distribution
It is then natural to ask whether there exist other quadratic forms in a
spherical
multinormal variable of unit variance that also have a Chi

square
distribution. In other words, do
matrices
P
exist such that :
x
'
Px
~
??
for
x
~
N
(0,
I
p
) ?
Recall that a symmetric matrix is said to be "idempotent" if
P
²
=
P
.
We'll show that a
necessary and sufficient
condition for
x
'
Px
to have a Ch
i

square distribution is that
P
be an
idempotent matrix. In addition, we'll show that
the rank of
P
and the number of degrees of freedom of the
distribution are the
n
equal.
3
In other words :
* If
x
~
N
(0,
I
n
)
*
Then
Q =
x
'
Px
~
r
if and only if
:
*
P
²
=
P
*
rank
(
P
) =
r
In fact, we'll show a bit more than that as we'll
consider variables
~
N
(
µ
,
I
)
that are not necessarily centered on the origin.
A symmetric idempotent matrix
P
or rank
r
is interpreted as the matrix of an
orthogonal
projection
operator on a
r

dimensional subspace. The projection of the
vector
x
is
Px
.
If
P
²
=
P
, then :
x
'
Px
=
x
'
P
²
x
=
x
'
PPx
=
x
'
P
'
Px
For
P
is symmetric
= (
Px
)'(
Px
)
and
x
'
Px
is therefore the
squared length
of the projection of
x
on the subspace.
The above result is then interpreted as follows :
* A quadratic form in a spherical, unit variance
multinormal vector is Chi

square distributed with
r
degrees of freedom if and only if it is the squared length
of the projection of
x
on some
r

dimensional subspace.
In the above illustration :
* The vector
x
has a spherical multinormal distribution with uni
t variance.
*
Px
is the projection of
x
on the
r

dimensional subspace
.
4
* The squared length of
Px
is distributed as
r
.
Quadratic forms in a general multinormal variable and the Chi

square
distribution
We then address the general issue of quadratic forms in a
multivariate normal
variable with an arbitrary covariance ma
trix
:
x
~
N
(
µ
,
)
We'll show that the quadratic form
Q
=
x'Ax
fol
lows a (non central) Chi

square
distribution with
r
degress of freedom
if and only if
both the following conditions
are satisfied
:
1)
A = A
A
2)
rank
(
A
) =
r
the value of the noncentrality parameter being
µ'Aµ
.

So, in the general case, the interpretation in terms of the distribution of the
squared length of the projection of
x
on a subspace is lost, and
A
is
not
a
projection matrix.

This situa
tion is encountered, for instance when studying the
Mahalanobis
distance.
Independence of two quadratic forms in a multivariate
normal vector, Craig's Theorem
Independe
nce of two random variables is a condition that makes life of a
Statistician a lot easier. In particular, many results pertaining to the
sum
and the
ratio
of two random variables explicitely assume that these variables are
independent. For example, the formal
definition
of Fisher's
F
distribution involves
the ratio of two independent
variables.
It is therefore n
atural to ask under what conditions two quadratic forms in a
multivariate normal vector
x
are independent.
Independence of two linear forms
We'll first
show
that two linear forms
A'x
et
B'x
in a multinormal vector
x
with
covariance matrix
are independent if and only if :
A'
B
= 0
This result is important by itself but, in addition, it will be needed later when we
address the issue of the independence of quadratic forms.
5
Independence of qua
dratic forms
We'll then
show
that :
* Let
x
~
N
p
(
µ
,
),
* And let
x'Ax
and
x'Bx
be two Chi

square distributed quadratic forms.
* Then these two forms are
independent
if and only if
A
B
= 0
I
n the special case where
x
is spherical, this condition becomes the simpler
AB
=
0.

It is not assumed that the Chi

square distributions of the
quadratic forms are
central or have the same numbers of degrees of freedom.
Craig's Theorem
The above resul
t explicitely assumes that the quadratic forms are Chi

square
distributed. As it happens, this assumption is unnecessary and is introduced just as
to make the demonstration easier.
More generally :
* Let
x
~
N
p
(
µ
,
),
* And let
x'Ax
and
x'Bx
be two
quadratic forms.
* Then these two forms are
independent
if and only if
A
B
= 0
which is the same as before but with the assumption about the
distributions of
the quadratic forms
removed. The result is then known as
Craig's Theo
rem
,
whose demonstration lies beyond the bounds of this Glossary.
Cochran's Theorem
The results
about the nature of the distribution and the independence of quadratic
forms in multinormal vectors as described here are a key element of a most
important theo
rem in Statistics known as
Cochran's Theorem
, which is stated and
demonstrated
here
.
__________________________________________________________________
6
Tutorial 1
In
the first section of this Tutorial, we calculate the expectation of a quadratic
form with no assumption about the distribution of the variable. This section is
independent of the remainder of the Tutorial.

We then establish a necessary and sufficien
t condition for a quadratic form
Q
:
Q =
x'Px
in a multivariate normal variable to be Chi

square distributed.
* We first address the special case of a spherical multinormal distribution with
unit variance (identity covariance matrix).

The "
sufficient" condition states that if
P
is a projection matrix with rank
r
,
then
Q
is Chi

square distributed with
r
degrees of freedom.

The "necessary" part states that if
Q
is Chi

square distributed with
s
degrees
of freedom, then
P
has to be a p
rojection matrix with rank
s
. The demonstration is
a bit more difficult, and the
moment generating function
(m.g.f.) of the quadratic
form will be of central importance.
* We then go over the general case (arbitrary covariance matrix) by identifying
a transformation that will turn the problem into the already solved
"special case"
problem, a common strategy when studying the multivariate normal distribution.
So the Tabl
e of Contents of the general case is quite short, as the brunt of the
work has already been done when solving the special case. Once the
transformation has been identified, only a small additional effort will be needed to
express the result as a function o
f
,
the covariance matrix of
x
.
QUADRATIC FORMS IN MULTINORMAL VECTORS
AND THE CHI

SQUARE DISTRIBUTION
Expectation of a quadratic form


Multivariate normal distribution with identity
covariance matrix
A sufficient condition for a quadratic form in a multivariate normal vector
with identity covariance matrix to be Chi

square distributed
The condition is also necessary
First
form of the m.g.f. of the quadratic form
7
Second form of the m.g.f. of the quadratic form
Final result
Rank of
P
Eigenvalues of
P
P
is idempotent
Noncentrality parameter
General case : multivariate normal
distribution with arbitrary covariance matrix
TUTORIAL
_____________________________________________________________
_________
_____
Tutorial 2
We now address the issue
of the independence of two quadratic forms is a
multivariate normal vector.
We first solve the simpler
problem of the independence of two linear forms. This
result
will anyway be needed in the remainder of the Tutorial.

We then move on to quadratic forms and establish a necessary and sufficient
condition for two such fo
rms to be independent. The reader won't be surprised to
see us first solve the case of quadratic forms in a spherically symmetric, unit
variance multinormal vector. The general case (arbitrary covariance matrix)
will
then be solved by identifying a transfo
rmation that turns the general case into the
already solved special case.

Note that we always consider quadratic forms with Chi

square distributions. As it
turns out, this assumption is unnecessary and is introduced only to make the
demonstration eas
ier. The condition remains valid without it and then bears the
name of
Craig's Theorem
, which is very difficult to demonstrate.
INDEPEN
DENCE OF QUADRATIC FORMS
Independence of two linear forms in a multivariate normal vector
Independence of two Chi

square distributed
quadratic forms
in a standard multinormal vector
The condition is necessary
The condition is sufficient
The final result
8
In
dependence of two Chi

square distributed
quadratic forms
in a multinormal vector with arbitrary covariance matrix
TUTORIAL
___________________________________________________
Related readings :
Chi

square distribution
Multivariate normal distribution
Cochran's Theorem
Projection matrix
Comments 0
Log in to post a comment