Project 1: Adaptive noise cancellation

streethicksvilleAI and Robotics

Nov 24, 2013 (3 years and 11 months ago)

95 views


1




EEL 6502 Adaptive Signal Processing



Project 1: Adaptive

noise cancellation











By

Meena Ramani












2

This report
analyzes the design and performance of an adap
tive noise canceller using nlms.


1)

Abstract

2)

Introduction

a.

Adaptive noise cancellation

b.

Wiener Filter

c.

LMS


d.


NLMS


3)

Analysis of
input

data

Primary and Secondary Inputs

a.

Time domain

b.

Frequency domain


4)

Results for a 2
-
tap filter

a.

Performance surface contours

b.

Weight Tracks

c.

Learning Curve

d.

Frequency Response

e.

Cleaned output

&
SNR improvement


5)

Analysis o
f

optimal step size and filter order

a.

Variation of step size (fixed order)

b.

Variation of order (fixed step size)

c.

Misadjustment vs. step size


6)

Results for a higher order
filters

a.

Weight Tracks

b.

Frequency Response

c.

Speed and Step S
ize

d.

Misadjustment

e.

Cleaned output &
SNR improvement


7)

Conclusion






3

Abstract

Adaptive systems are different fundamentally from other signal processing systems in that they require a
minimum
a priori

knowledge of the signals or systems under analysis. The
objective of any linear adaptive
system is to obtain, using some iterative process, an optimum linear estimator of a system.


There are four main types of adaptive filter configurations: Adaptive system identification, Adaptive noise
cancellation,
A
daptive

linear prediction and
A
daptive inverse systems. While they implement the same
algorithm
,

their system configuration differs.

Adaptive noise cancellation (ANC) is often used to extract the
desired speech from the given noisy speech. ANC plays an important
role in digital voice communication
systems For Eg: Cell phones require adaptive noise cancellation to reduce further degradation of vocoded
speech.


We will perform ANC using nlms in order to clean the noisy speech signal. The effects on stability,
conver
gence, speed and computation on choosing the different parameters for nlms is studied here and in
the end we will decide on a system which has the best tradeoffs.














4

2)

Introduction


2.
a)
Adaptive noise cancellation


ANC deals with the enhancement of

noise corrupted signals. Its biggest strength when compared to other
enhancement techniques is that no a priori information about either the signal or noise is required. This
advantage comes at the cost that we need in addition to the primary signal

d(n)

which is Signal corrupted by
noise a reference/

secondary signal

x(n)

which contains noise

1
( )
v n

correlated with the noise component

0
( )
v n

in the primary signal.

The principle here is to adaptively filter the ref
erence signal to get a replica of
the noise in the primary and subtract it from the
p
rimary.

Here the system output is the error signal.




Now we can see that:

0
( ) ( ) ( ) ( )
e n s n v n y n
  


Squaring and taking the expectation op
eration on both sides gives us:

2
2 2
0 0
{ ( )} { ( )} {( ( ) ( )) } 2 { ( )( ( ) ( ))}
E e n E s n E v n y n E s n v n y n
    


Assuming that s(n) is uncorrelated with
0
( )
v n
and
1
( )
v n

we get that
s(n)

is also uncorrelated with
y(n)

since
y(n)=
x(n)*w(n).

Thus the last term above b
ecomes zero and we get:


2 2 2
0
{ ( )} { ( )} {( ( ) ( )) }
E e n E s n E v n y n
  

Now to minimize the error we cannot reduce
s(n)

and hence we get the MSE minimized when
0
( ) ( )
y n v n


This implies
that
( ) ( )
e n s n

, the desired result.


Also if we assume
0 1
( ) ( )* ( )
v n v n h n

, where the transmission path between
0
( )
v n
and
1
( )
v n

has been
modeled by
h(n)
, we can conclude that the adaptive filter models the unknown transmission path
h(n)
.



0
( ) ( ) ( )
d n s n v n
 

1
( ) ( )
x n v n


( )
y n

( )
e n

w

+


5

Limitations of ANC:

We
assume the noise components are correlated if they are uncorrelated then the speech will further be
degraded as there is another additive noise component.


Also there might be leakage or cross
-
talk and the reference signal may contain a correlated componen
t of
Speech. This will result in the adaptive filter canceling speech and will result in distortion.


2.
b)
Wiener Filter

The solution of the
Normal equations
:


1
0
( ) ( ) ( )
L
k
P i R i k w k


 


i
=
0,1,…
L
-
1



Where
L

is the order of the filter yields the optimal
solution
.
The Wiener
-

Ho
p
f filter is an iterative solution
for
stationary normal equations
. It
is a linear filter which minimizes the mean
-
square estimation error. For an
FIR filter of order M the output
y(n)

is given in terms of its inputs and

weights (im
pulse response) as


1
0
( ) ( ) ( )
L
k
y n x n k w k


 

.



Let
d(n)

be the desired signal. The Wiener
-
Hopf equations minimize the MSE

J given by:





2
2
{ ( )} ( ) ( )
J E e n E d n y n
  



a
nd

provide
s

us with M optimum values of
w


which minimize the MSE
i.e.


1
*
w p


R


,
where



w

is the weight vector (
L

x 1)

the superscript * is to indicate that it is the optimal solution.

R

is the auto
-
correlation matrix (
L

x
L
)

p

is a vector of cross correlations between the desired signal & the input (
L

x 1)


For stationary signals we can use time average estimates (biased/unbiased) for the correlation coefficients
(
R

and
p
).


The error
vector J is a second order function of the weights. That is J is a ‘bowl shaped’ function of the
weights. J is also known as the error performance surface. For an L order filter the bowl is actually L
-
dimensional
-

a hyperparaboloid.


The minimum value
J
mi
n

occurs at the bottom of this bowl where
w
-
>w*



6

*
min
2
t t t
J J w w w p w p
   
R


Where

*
2
min
t
d
J w p

 
2 2
,{ ( )}
d
E d n




From the above we can observe that
w=w*

is a minimum of J since any change in
w

from
w*

c
an only
increase

J
. Also we have that
min
J
is a monotonically non
-
increasing function of filter order i.e.
1
min min
L L
J J




2.
c
)

Least Mean Squares
-

LMS


The problem
s

with the Weiner filter
are as follows:




Calculation of
R

and
p

is very computational

expensive

3
( )
O L



We have to use time average estimates to compute
R

and
p
and this creates problems for non
-
stationary signals.
We can assume local
-
statio
narity and then approach the problem using a short
term time average estimate but a better approach would be to use a filter which would track the
changes in the optimal solution.

The Least Mean Squares (LMS) filter does just this.


De
signed

by Bernard Wi
drow and M. Hoff in 1959, this simple algorithm now represents the cornerstone
for the design of adaptive filters
.

The LMS is based on the steepest descent approach where at each iteration
we move, in weight space, in a direction opposite to the gradient o
f the MSE

(
( )
J n

)

and by a distance
proportional to the magnitude of the gradient.


* ( )
( 1) ( )
2
J n
w n w n


  


, w
here


is a constant of proportionality which we call the
step size
.


In the LMS,

we estimat
e the gradient of the MSE using the gradient of the instantaneous value of squared
error

(
( )
J n

)
. The iterative update equation
thus
becomes
:


( 1) ( ) ( ) ( )
w n w n x n e n

  


,

where

( )
w n
is
the weights

vector (Lx1)

at time
n



This simple update equation reduces the computational complexity to O(L).
Now by using the
instantaneous estimate for the gradient
( )
J n


instead of
( )
J n

we cannot expect the filter to converge to
the

optimal solution

w*

obtained using the Wiener filter
.

This approximation leads to a “gradient noise”

which
makes the weights computed by LMS to have a random motion about
w*
.


Because of t
he feedback involved in the LMS,
stability and convergence are imp
ortant issues.



The algorithm is proven to converge if the
adaptation step

satisfies:


7


max
2
0


 


,
where
max

is the biggest eigenvalue of input correlation
matrix

. In practice we
use


2
0
*
L InputPower

 


When the value of


is in the range above, the
LMS behaves like a low pass filter and filters out the
gradient noise by computing the we
ights slowly.

Thus
for smaller

,
the MSE on an average is smaller.



The
time constant (/speed) for convergence

or the average time constant
is given by:

min
1
2





,
where
min

is the
smal
lest

eigenvalue of input correlation matrix
.


Misadjustment

is defined as t
he ratio of



min
min
( )
J J
M
J
 




, where
( ) ( )
J n J
 

as
n



For small


we can write
* [ ]
2
tr R
M




This is a measure of how far the LMS steady state solution is from the Wiener solution

and i
s the price we
pay by using LMS instead of the Wiener filt
er.

Smaller M is from unity the better.

Having a small


reduces Misadjustment as there is less deviation from
w*
.
Now the trace of R depends of the filter
order (
L)
and hence M increases as L increases
. It is also obvious that M

and

are inversely proportional.



We can make the following observations from

the

above

equations
:




The Convergence and stability of the LMS depend upon the input signal and the filter order.

If
( )
x n
is lar
ge then the LMS filter amplifies the gradient noise



T
he speed of convergence

gets very slow if there is a big spread among the eigenvalues of

(a small
eigenvalue
leads to a small correction in the filter weights so that there should be a lot of iterations
to reach the optimal point in that dimension).





8

2.
d)
Normalized

Least Mean Squares
-

N
LMS


As we saw

above
the Convergence, stability and the steady state prope
rties of the LMS filter depend upon
the input signal and the filter order. To remove this dependence the most obvious change would be to
normalize the step size by these factors:

'
2
* { ( )}
p
L E x n T
 

 

In practice we calculate the power
T
p

using a tim
e average
( ) ( )
t
x n x n


Hence we get:

( ) ( )
( 1) ( )
( ) ( )
t
x n e n
w n w n
x n x n


  


This is the Normalized LMS (NLMS) algorithm. This normalization carries a computational overhead but is
required especially in cases where the input signals fluctuate widely.
The



in the denominator is

a small
positive constant used to avoid dividing by zero.


A point to observe is that the NLMS has a higher and faster rate of convergence when compared to LMS



3)

Analysis of
input
data

T
he
Secondary/No
ise

Input
signal

time and frequency domain characteristics:





As the sampling frequency is 16

Khz the highest frequency signal recoverable is 8000Hz
.




9


The Spectrogram of the noise signal shows the distribution of energy over time and frequency. Ther
e is a
strong noise signal at frequency of 2.8Khz existing throughout the sentence duration.


T
he
Primary/Speech + Noise Input signal

time and frequency domain characteristics:


The frequency domain picture for speech+noise shows how noise has corrupted
the speech
. The spectrum
of primary and the secondary look almost identical!


The spectrogram below shows the presence of some speech signal among the noisy signal




10

4)

Results for a 2
-
tap filter


We are going to observe the performance of a 2
-
tap nlms fi
lter.


4.a)
Performance surface contour

As mentioned before, t
he quadratic MSE (
J
) is a
concave

upward hyperparaboloid. Contours of constant
J
or MSE are elliptical as can be seen below. The point at the
bottom of the bowl is the point of minimum
MSE and the weights which result in this are the wiener optimal weights

w*
.


Using the wiener filter the optimal weights got for a filter order 2 w0=
-
0.3465 and w1=1.2205. Also the
MSE here is:
0.0611
.

Note the pe
rformance of the Wiener filter

(
Excess
MSE)

for non
-
stationary signals is
worse than for NLMS.



2 dimensional Performance Surface








11

The following is a plot for a filter of order 2. Note that after about

=0.1 the MSE incr
eases with increase
in step size.


For values below 0.1
you can see that as the


is very small

and as the filter order is very low,

no of iterations required for the weights to converge is more than what we have provided (which is

the
length of the signal.) Thus the MSE computed for

<0.1 does not follow the trend of increasing with
increase in step size.


For a step size:

=0.
001


MSE
=

0.0160




For step size

=
0.01

MSE
=


0.0105








12


For a step size:


=0.
1

Using an NLMS filter of order 2 we get an MSE of

0.0105
. Note the weights w0=
-
0.35
and w1=1.2





W
e can
also
notice that larger the step size


less the stability

or more the movement
a
bout

the
optimal
weights.

At values of


>1 we can notice that the weights are very unstable.


For a step size:

=0.
5


MSE=
0.0106





For step size

=
2

MSE=

0.0113






13

4.b) Weight Tracks

We can notice that as


increases the
variation

of the weights

about the optimal

(w0=
-
0.35
and w1=1.2

)

increases.

Also notice that when

=
0.
001 the weights haven’t got any time to settle at.

For a step size:

=0.
001






For step size

=
0.
1




For a step size:

=0.
5




For step size

=
2









14


4.c)
Learning Curve

The learning curve which is normally plotted is for the ensemble averaged square estimation error. But here
since we have only one signal the plot is o
f

the time averaged

MSE. In a noise cancellation case the error
s
ignal is Speech so when we plot the learning curve what we get is as follows:





We can plot the residual noise as follows:




10
1
2
0
1
2
0
( ) 10log ( )/( )
( ) ( )
( ) ( )
E X
Q
E
j
Q
X
j
R k P k P k
P k e k j
P k x k j





 
 




, w
here Q=512


e(n) is the error signal (Speech)


x(n) is the Primary/de
sired signal (Speech+Noise)



The Residual error has been plotted below for both
Wiener filter and the nlms for step size 0.1 and order 2.
We can see that the Wiener Filter has worse performance which proves that nlms works better than wiener
for non
-
stati
onary data.







15








16


4.d)
Frequency Response








The
transfer function for the system with input as the primary
d(n)

and output the error
e(n)

is as follows:


( )
( )
( )
E z
H z
D z



Where H(Z) is the transfer fu
nction i.e. the response in Z domain for an unit impulse input at the secondary
x(n)


Thus
for an unit impulse
1
0
( ) ( ) ( )
L
j
j
y k w k x k j


 


reduces to:


1
1
1
1
.1
2 (.1)
..
2
2 ( )
(1 2 ) 2
k k k
k k k
k k k
k k k k
k k k
y w w
w w e
i e
y y e
y y d y
y y d



 




 
 
 
  
  


Now taking the Z transform we get:

0
( ) ( ) ( )
d n s n v n
 

1
( ) ( )
x n v n


( )
y n

( )
e n

w

+

Output

Input


17









1
( ) (1 2 ) ( ) 2 ( )
( ( ) ( )) (1 2 ) 2 ( )
( ) 1 2 2 ( ) (1 2 )
( ) ( 1)
,( )
( ) (1 2 )
Y z z Y z D z
D z E z z D z
D z z E z z
E z z
Thus H z
D Z z
 
 
  

  
   
     

 
 


Thus we ha
ve a zero on the unit circle and a pole at the (1
-
2

) value. For

=0.1 the pole is at 0.8 and
hence we have the following pole zero plot:




Also we have the transfer function estimate plot as: We can ob
serve that the filter appears to behave like a
HPF.








18

4.e)
SNR improvement


2
2
( )
10log
( )
d n
SNR
e n



d
(n)

is the desired and
e(n)

is the error signal. This is the reverse of the residual noise measure. A high value
of SNR is not really a good ind
ication of the cleaned speech quality as the speech might be attenuated too.


For order 2 step size=0.1 we have SNR=

3.6522
dB


The following plot shows the variation of SNR with step size. The last

10
00 samples of
d(n)

and
e(n)

were
used here. This is done

to give enough time for most of the step sizes to settle down.




SNR nlms:
7.0550
dB

SNR Wiener:
-
0.7545
dB


Let us now see the time, frequency and the spectrogram plot of the cleaned output. Here an order 2 step
size 0.1 nlms system was used.














19


5)

Analysis on optimal step size and filter order

The filter order and the step size affect the stability, the convergence and the speed of convergence of the
nlms filter. Hence we have to carefully select these and have a tradeoff between accuracy and c
omputation.


5.a)
Variation of step size (fixed order)


For a Fixed filter order of 2 variation in step size



Increasing the Filter order also increases the Misadjustment. This can be seen from the above plots which
show variation of step sizes for th
ree different filter orders.







In general as the Step size increases the
MSE increases because of increase in
misadjustment. The initial decrease in
MSE is for those small step sizes which
require longer iteration peri
ods than can
be provided.

As mentioned before the
step size of 0.1 gives the best results.


20


5.
b
) Variation of order (fixed step size)

For a Fixed step size of 0.1 effe
ct of variation of order on MSE is as follows:




J
is a monotonically non
-
increasing function of filter order i.e.
1
L L
J J


.

We can see that the error reduces and remains almost constant at about 80. But the value of MSE=0.006 is
also acceptable hence lets compare the performance of Order=2,25,80 and choose the best among the
three.


5.
c
) Misadjustmen
t vs. step size


Misadjustment

is defined as the ratio of



min
min
( )
J J
M
J
 




, where
( ) ( )
J n J
 

as
n



For small


we can write
* [ ]
2
tr R
M




This is a measure

of how far the LMS steady state solution is from the Wiener solution

and is the price we
pay by using LMS instead of the Wiener filter. Smaller M is from unity the better. Having a small


reduces Misadjustment as there is less
deviation from
w*
.
It is also obvious that M and

, the speed of
convergence , are inversely proportional.



21

Let’s first
calculate the autocorrelation of the Input R
XX

using the unbiased estimator for a window of 200.
Then
let’s

ca
lculate M using the above formula





6)

Results for a higher order filters


6.a)
Weight Tracks

25 order filter:






80 order filter:
















22


6.b
) Frequency Response



6.c
)
Step Size

The step size,

, determines th
e speed of convergence and the stability of the algorithm.

As

increases
the speed also increases. Thus is short we can say:


Small


implies



small misadjustment in steady
-
state



slow adaptation/tracking


Large


implies



large misadjustment in steady
-
state



fast adaptation/tracking



The
time constant (/speed) for convergence

or the average time constant is given by:

min
1
2





,
where
min

is

the
smallest

eigenvalue of input correlation matrix
.


The convergence rate determines the rate at which

the filter settles down at its optimal weights. A faster
convergence is a good feature for an adaptive filter. But Speed of convergence and misadjustment are
inversely proportional. Thus while a large step size will cause large misadjustment it will cause the speed of
convergence to increase but the stability wi
ll decrease if it exceeds

2
0
*
L InputPower

 

And then the system is more likely to diverge as was seen in the previous plots.



23



6.c
) SNR improvement


2
2
( )
10log
( )
d n
SNR
e n


The following plot shows the variation of order and SNR improvemen
t
. This is because increase in order
also increases Misadjustment as the Size of R increases



The following table has the values of SNR for the 3 orders under consideration

Order

SNR(dB)

2


7.0550

25


13.2881

80


13.0065



6.d
)
Misadjustment

vs Order

The following plot shows the variation in M with L.

The MSE decreases with order but the M increases
with order increase. T
his is because

min
J
is a monotonically non
-
increasing function of filter order i.e.
1
min min
L L
J J






24


The following table shows the value of M for the three orders under consideration.

Order

M

2


0.0042

25


0.0527

80



0.1687



6.
e
)
Output signal

ORDER 2



25


ORDER 25










MSE:
8.8032e
-
004

MSE:
0.0105


26

Order 80








We can notice that for a con
stant step size of 0.1 as the order increases the MSE decreases as shown. At
order 25 and 80 the speech is much clearer than before and though slightly blurred sounds like:


“He writes at home every other night”



Comments on Convergence:


NLMS performs be
tter for stationary data compared to non
-
stationary data as can be seen from the plots
and tables above. And it performs better than the Wiener filter with non
-
stationary data as shown below.
This is because the Wiener filter assumes stationarity which is
not always a valid assumption especially in the
case of speech.






MSE:

3.0102e
-
004


27





Conclusion


NLMS is recommended for use in noisy environments
. Increasing the filter order increased the performance
of the system. Also the step size was fixed at a point which gav
e reasonable tradeoff between the
Misadjustment and the convergence.

A point to be noted was that the system
worked even

when the inputs
were switched. This could be because the unknown system was IIR
.