CS 2223 Nov 06 2015

Lecture Path: 07
Back Next

Expected reading: 271-282
Daily Exercise:

Hmm, difficult. VERY difficult. Plenty of courage, I see. Not a bad mind, either. There’s talent, oh yes. And a thirst to prove yourself. But where to put you?
The Sorting Hat

1 Mergers and Acquisitions

Selection Sort and Insertion Sort both operate in linear fashion. That is, they each seek to reduce the size of the problem by one with each iteration.

Can we find someway to divide the problem in half with each pass? MergeSort offers the ability to do just this.

1.1 Merge Sort

Unfortunately, the first attempt at doing so typically leads to excessive wasted storage that slows everything down. Let’s show what the code would look like:

public static void sort(Comparable[] a) { sort (a, 0, a.length-1); } static void sort (Comparable[] a, int lo, int hi) { if (hi <= lo) return; int mid = lo + (hi - lo)/2; sort(a, lo, mid); sort(a, mid+1, hi); // merge a[lo..mid] with a[mid+1..hi] into new storage aux = new Comparable[hi-lo+1] // copy aux back into a }

The reason Mergesort works in the "real world" is that there is no need to worry about extra storage; as long as you have "room" on the table, then you will have place to put the merged results. But we don’t have that luxury in software. Fortunately, the fix is an easy one to make (p. 273). Instead of allocating the storage within each sort invocation, do so once when the function begins:

So we need to explain how merge works with an auxiliary array, which is accessible to the merge function:

static void merge (Comparable[] a, int lo, int mid, int hi) { int i = lo; // starting index into left sorted sub-array int j = mid+1; // starting index into right sorted sub-array // copy a[lo..hi] into aux[lo..hi] for (int k = lo; k <= hi; k++) { aux[k] = a[k]; } // now comes the merge. Something you might simulate with cards // drawn from two stack piles. This is the heart of mergesort. for (int k = lo; k <= hi; k++) { if (i > mid) { a[k] = aux[j++]; } else if (j > hi) { a[k] = aux[i++]; } else if (less(aux[j], aux[i])) { a[k] = aux[j++]; } else { a[k] = aux[i++]; } } }

The key is the final for loop. While it looks complicated at first, you can break it down into four cases as follows, which matches the above code

• (i > mid) Have you exhausted left elements?

• (j > hi) Have you exhausted right elements?

• (less(aux[j]], aux[i]) Take element from right if smaller

• Else we take element from left because it is larger or equal

1.2 Recursive in-class demonstration

Work through example on small array (from p. 273)

E M R G E S O R

Review in debugger in Eclipse (AnnotatedMerge)

1.3 Comparison analysis

Mergesort uses between ~1/2 N log N and N log N comparisons to sort any array of length N (p. 272).

How can we show this? Let’s develop a mathematical model that counts the number of comparisons, C(n), for sorting n elements. For starters, let’s assume that n is a power of 2. This simplifies the accounting of operations, and in the long run, this won’t affect the results since we will be able to show that the result holds even for n that are not powers of two.

First focus on the number of comparisons in merge (a, lo, mid, hi). Consider when a[lo..mid] has two elements and so does a[mid+1..hi]. With each comparison you advance either i or j, and it is possible that this could happen just 3 times, before one of them fall off the range. Check this out yourself:

i: [ 2, 4 ] merge j: [ 1, 3]

is 1 < 2? Yes, so advance j
is 3 < 2? No, so advance i
is 3 < 4? Yes, so advance j

To generalize, you could go as far as N-1 comparisons where N is the sum of the lengths of the two arrays being merged.

In the best case, you would only compare N/2 elements since one would become exhausted.

So you can’t do FEWER than N/2 and you can’t do more than N-1.

To clean this up, we’ll just say:

So you can’t do FEWER than N/2 and you can’t do more than N.

So now ready for the big claim:

C(N) <= C(N/2) + C(N/2) + N
C(N) >= C(N/2) + C(N/2) + N/2

This is a very accurate evaluation. To compute the upper bound for C(N) we use the first equation above and try to find equality:

Now we assume N is a power of 2, or 2n. Thus we have

C(2n) = 2*C(2n-1) + 2n.

If you recursively replace the right-most C(2n-1) you get:

C(2n) = 2*[2*C(2n-2) + 2n-1] + 2n.

Let’s do one more time to see the effect:

C(2n) = 2*[2*[2*C(2n-3) + 2n-2] + 2n-1] + 2n.

If you expand out the multiplications, you get the following:

C(2n) = 8*C(2n-3) + 2n + 2n + 2n.

As you look at this equation, it looks like the following:

C(2n) = 2k*C(2n-k) + k*2n.

Now replace back:

C(N) = 2k*C(2n-k) + k*N

So the final question to ask is how many times can you increase k? The answer is log(N), at which point C(1) means no comparisons for a single element. Thus the final equation we develop is:

C(N) = N * log N as an upper bound on the number of comparisons.

The analysis becomes more complicated when N is not a power of 2, but this suffices for lecture. Thus C(N) is ~N log N.

1.4 Array access analysis

Mergesort uses at most 6N log N array accesses. This works out in similar fashion and I won’t do it here (please see p. 275).

1.5 Comparison-based Sorting Proposition

So now let’s explain why the Merge Sort analysis interesting.

Remember the exercise to compute the fewest number of comparisons needed to sort five elements? If you use comparisons to sort elements, there are some fundamental limits that we can prove. First, consider the following decision tree that reflects the n! permutations of four elements:

Figure 1: Similar to p.281

First verify that all 4!=24 possible permutations exist as leaves in this decision tree. Now the question at hand is the maximum distance from the top decision (a1 <= a2) to any of the permutations.

There are some permutations that only require four comparisons (for example, the leftmost a1a2a3a4. But some require five comparisons, such as in the middle a4a3a1a2. Note that the maximum number of comparisons is 5.

Can we use this observation to make any claim about the maximum height of a decision tree for using comparisons to sort n elements? The answer is yes! First we need to state a properly of binary decisions trees, namely, that a tree with height h has no more than 2h leaves. And in fact, the tree of height h with the maximum number of leaves is perfectly balanced, or complete (see p. 281).

Second observe that the decision tree will have at least n! leaves; after all, you could have an algorithm that reaches the same sorted answer with a different ordering of comparison questions.

Thus N! <= number of leaves <= 2h

Take logarithm of both sides, and you see log(N!) <= h. So all you need is an approximation for log(N!). Try this:

log(N!) = log( N*(N-1)*(N-2)* ... *2*1 )
log(N!) > log( N*(N-1)*(N-2)* ... *N/2) Only go half way
log(N!) > log( (N/2)N/2) Simplify further
log(N!) > (N/2)*log(N/2)
log(N!) > (N/2)*(log N - 1)
log(N!) > (1/2)*N*log N - N/2

Now using Tilde approximation, N*log N will grow more rapidly than N/2, so we can say (finally):

log(N!) is ~ N log N

Because of this confirmation, you know that no sorting algorithm that uses comparisons can guarantee to use fewer than ~ N log N comparisons. This is a lower bound on the difficulty of the sorting problem.

No compare-based sorting algorithm can guarantee to sort N items with fewer than log(N!) comparisons. Note that log(N!) is ~ N log N.

This is a LOWER BOUND on the number of comparisons needed to sort elements.

Mergesort offers an UPPER BOUND on the number of comparisons that it uses.

With this in hand, we can assert the Mergesort is an asymptotically optimal compare-based sorting algorithm.

*whew*

1.6 Version : 2015/11/08

(c) 2015, George Heineman