Lecture 4. Paradigm #2 Recursion

geographertonguesΤεχνίτη Νοημοσύνη και Ρομποτική

30 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

140 εμφανίσεις

Lecture 4. Paradigm #2 Recursion


Last time we discussed Fibonacci numbers F(n), and Alg



fib(n)


if (n <= 1) then return(n)


else return(fib(n
-
1)+fib(n
-
2))


The problem with this algorithm is that it is woefully
inefficient. Let T(
n
) denote the number of steps needed
by fib(
n
). Then T(
0
) = 1, T(
1
) = 1, and T(
n
) = T(
n
-
1) +
T(
n
-
2) + 1. It is now easy to guess the solution


T(
n
) = 2 F(
n
+1)
-

1


Proof by induction: Clearly the claim is true for
n

= 0, 1.
Now assume it is true for all
n

<
N
; we prove it for
n

=
N
:

T(
N
) = T(
N
-
1) + T(
N
-
2) + 1

= (2 F(
N
)
-

1) + (2 F(
N
-
1)
-

1) + 1

= 2(F(
N
)+F(
N
-
1))
-

1

= 2 F(
N
+1)
-

1


We know F(n) =
Θ
(a
n
), a=(1+√5)/2

Memoization


A trick called "memoization“ can help recursion, store function values
as they

are computed. When the function is invoked, check the
argument to see if the function value is already known; that is, don't
recompute. Some programming languages even offer memoization as
a built
-
in option. Here are two Maple programs for computing Fibonacci
numbers; the first just uses the ordinary recursive method, and the
second uses memoization (through the command "option remember").


A recursive one:


f := proc(n)


if n ≤ 1 then n else f(n
-

1) + f(n
-

2) end if


end proc;


One that looks recursive, but that uses memoization:


g := proc(n)


option remember;


if n ≤ 1 then n else g(n
-

1) + g(n
-

2) end if


end proc;


When you run these and time them, you'll see the amazing difference in
the running times. The "memoized" version runs in linear time.

Printing permutations


Sometimes, recursion does provide simple & efficient
solutions. Consider the problem of printing out all
permutations of {1,2,...,
n
} in lexicographic order.


Why might you want to do this? Well, some
combinatorial problems have no known efficient
solution (such as traveling salesman), and if you want
to be absolutely sure you've covered all the
possibilities for some small case.


How can we do this? If we output a 1 followed by all
possible permutations of the elements other than 1;
then a 2 followed by all possible permutations of the
elements other than 2; then a 3 followed by .... etc.,
we'll have covered all the cases exactly once.


Printing all permutations …


So we might try a recursive algorithm. What should the input
parameter be? If we just say
n
, with the intention that this gives
all permutations of {1,2,...,
n
} that's not going to be good
enough, since later we will be permuting some arbitrary subset
of this. So you might think that the input parameters should be
an arbitrary set
S
. But even this is not quite enough, since we
will have to choose an arbitary element
i

out of
S
, and then print
i

followed by all the permutations of
S

-

{

i
}. But if we don't want
to
store

all the permutations of
S

-

{
i
} before we output them we
need some way to tell the program that when it goes and prints
all the permutations of
S

-

{
i
}, it should print
i

first, preceding
each one. This suggests making a program with
two

parameters: one will be the fixed "prefix" of numbers that is
printed out, and the second the set of remaining numbers to be
permuted.

Printing permutations


printperm(
P
,
S
)


/*
P

is a prefix,
S

is a nonempty set */


if
S={x}

then print
Px
;


else for each element
i

of
S

do


printperm( (
P
,
i
),
S

-

{

i
});



There are n! permutations. So any program must spend O(n*n!)
time to print all the permutations, each taking n steps.


Let me give a simple amortizing counting argument. We will be
printing n*n! symbols. Each time the "else" statement is
executed, we charge O(1) to that "i". This particular "i" in the
n*n! symbols (note i appears in different permutations many
times, but we are just referring to one instance of such "i") gets
charged only once. Summing up, the total time is O(
n


n
!).

Paradigm #3: Divide
-
and
-
conquer


Divide et impera

[Divide and rule]


--

Ancient political maxim cited by Machiavelli


--

Julius Caesar (102
-
44BC)


The paradigm of divide
-
and
-
conquer:


--

DIVIDE problem up into smaller problems


--

CONQUER by solving each subproblem


--

COMBINE results together to solve


the original problem

Divide & Conquer: MergeSort


Example: merge sort (an O(n log n) algorithm
for sorting) (See CLR, pp. 28
-
36.)


MERGE
-
SORT(A, p, r)


/* A is an array to be sorted. This algorithm
sorts the elements in the subarray A[p..r] */


if p < r then


q := floor( (p+r)/2 )


MERGE
-
SORT(A, p, q)


MERGE
-
SORT(A, q+1, r)


MERGE(A, p, q, r)

MergeSort continues ..


Let T(n) denote the number of comparisons
performed by algorithm MERGE
-
SORT on an
input of size n. Then we have


T(n) = 2T(n/2) + n


expanding …


= 2
k
T(n/2
k
) + kn


….


= O(nlogn)
---

when k=logn.

Another way:


Prove T(2
k
) = (k+1)2
k

by induction:


It is true for k = 0. Now assume it is true for k;
we will prove it for k+1.


We have

T(2
k+1
) = 2T(2
k
) + 2
k+1

(by recursion)

= 2(k+1)2
k

+ 2
k+1

(by induction)

= (k+2) 2
k+1
,

and this proves the result by induction.

Divide & conquer: multiply 2 numbers


Direct multiplication of 2 n
-
bit numbers takes
n
2
steps. Note, we assume n is very large,
and each register can hold only O(1) bits.


When do we need to multiply two very large
numbers?


In Cryptography and Network Security


message as numbers


encryption and decryption need to multiply
numbers


My comment: but really: none of above seems to be a good
enough reason. Even you wish to multiply a number of 1000
digits, an O(n
2
) alg. is good enough!

How to multiply 2 n
-
bit numbers



************


************


************


************


************


************


************


************


************


************


************


************


************


************


************************

operations
bit

)
(
2
n

History:


1960, AN Kolmogorov (we will meet him again later in
this class) organized a seminar on mathematical
problems in cybernetics at MSU.


He conjectured
Ω
(n
2
) lower bound for multiplication
and other problems.


Karatsuba, then 25 year old student, proposed the
n
log3

solution in a week, by divide and conquer.


Kolmogorov was upset, he discussed this result in
the next seminar, which was then terminated.


The paper was written up by Kolmogorov, but
authored by Karatsuba (who did not know before he
received reprints) and was published in Sov Phys.
Dol.

AN Kolmogorov

1903
-
1987

Can we multiply 2 numbers faster?


Karatsuba's 1962 algorithm for multiplying
two n bit numbers in O(n
1.59
) steps.


Suppose we wish to multiply two n
-
bit
numbers X Y. Let X = ab, Y = cd where a, b,
c, d are n/2 bit numbers.


Then XY = (a ∙ 2
n/2

+ b)(c ∙ 2
n/2

+ d)

= (ac)2
n

+ (ad + bc)2
n/2

+ bd


c

d

a

b

X =

Y =

Multiplying 2 numbers


So we have broken the problem up to 4
subproblems each of size n/2. Thus,


T(2
k
) = 4T(2
k
-
1
) + c 2
k



4T(2
k
-
1
) = 16 T(2
k
-
2
) + 4c2
k
-
1

= 4
2
T(2
k
-
2
)+c2
k+1


...


4
k
-
1

T(2) = 4
k

T(1) + 4
k
-
1

∙ c ∙ 2


Now T(1) = 1, so T(2
k
) = 4
k

+ c(2
k
+ 2
k+1

+ ... +
2
2k
-
1
) ≤ 4
k

+ c 4
k

= (4
k
)(c+1).


This gives T(n) = O(n
2
)! No improvement!

Multiplying 2 numbers


But Karatsuba did not give up. He observed:


XY = (2
n

+ 2
n/2
)∙ ac + 2
n/2
∙ (a
-
b) ∙ (d
-
c) + (2
n/2

+ 1)∙ bd



Now, we have broken the problem up into only 3
subproblems, each of size n/2, plus some linear
work. This time it should work!


K(n) ≤ 3K(n/2) + cn


≤ c(3
k+1

-

2
k+1
)


Putting n = 2k, we see that for n a power of 2, we get


K(n) ≤ c(3
lg n + 1
)


2
lg n + 1

)

= c(3 n
lg 3

-

2 n)


Here we have used the fact that a
log(b)

= b
log(a)
.



Since lg 3 is about 1.58496, this gives us a O(n
1.59
) algorithm


Note: Using FFT. Schonhage and Strassen: O(nlogn loglogn) in
1971. In 2007, this was slightly improved by Martin Furer

Divide & conquer: finding max
-
min


Problem: finding both the maximum and
minimum of a set of n numbers.


Obvious method: first compute the maximum,
using n
-
1 comparisons; then discard this
maximum, and compute the minimum of the
remaining numbers, using n
-
2 comparisons.
Total work: 2n
-
3 comparisons.


Maxmin by divide & conquer


MAXMIN(S)


/* find both the maximum and minimum elements of a set S */


if S = {a} then


min=a, max=a;


else if S = {a < b}


min=a, max=b;


else /* |S| > 2 */


divide S into 2 subsets, S1 and S2, such that S1 has floor(n/2)


elements and S2 has ceil(n/2) elements


(min1, max1) := MAXMIN(S1);


(min2, max2) := MAXMIN(S2);


min = min(min1, min2);


max = max(max1, max2);


return (max,min);

Time complexity


T(n) = 1 when n =2


T(n) = 2T(n/2) + 2, otherwise.


This gives T(n) = 3n/2


2, when n is a power
of 2.