Weighted Ranking (Pair Ranking)

Weighted ranking makes criteria and their relative importance explicit. This exercise introduces pair ranking, a component that compares options one pair at a time.

EXERCISE 22 Movies

In a column on a sheet of paper write fifteen movies you have seen and about which you can make reasoned judgments. The list below will give you some ideas.

Now rank these fifteen movies from best to worst according to your personal likes and dislikes.

BEFORE CONTINUING, LIST AND RANK 15 MOVIES.

On a second sheet of paper, copy the matrix in Figure 10-1. In the first column, write in your fifteen movies as you initially listed them. I have entered my own list of movies as an example. Enter your rankings in the next column, as I have done with mine, and write “Instinctive Ranking” at the head of that column.

BEFORE CONTINUING, CONSTRUCT A MATRIX AND ENTER FIFTEEN MOVIES AND THEIR RANKINGS.

FIGURE 10-1

For the Ranking Techniques Matrix, use columns for the movie, its instinctive rank, and its pair rank.

Now, on a separate sheet of paper, write down the reason(s) why you ranked your top choice at the top. Then write down the reason(s) why you ranked your bottom choice at the bottom.

BEFORE CONTINUING, WRITE THE REASONS FOR YOUR TOP AND BOTTOM CHOICES.

I would guess that the reason(s) you gave for making your top and bottom choices are distinctly different. For example, you may have put The Silence of the Lambs first because you love scary movies and put From Here to Eternity last because you dislike black-and-white films. “What’s wrong with that?” you ask. “After all, these are legitimate reasons: Silence is scary and Eternity is in black and white.” That’s true enough, but there are other factors—other criteria—to be considered. Scariness and film color are only two.

The Problem with Instinctive Ranking

The main defect in our instinctive method lies in our tendency to view problems one-dimensionally… to focus on the first solution that makes sense. Thus, the moment we think of a reason—any sound, persuasive reason—to like or dislike one of the movies on our list, we tend to latch on to that reason, make our ranking decision on that basis, and move on without considering our other likes and dislikes, which may actually be more important to us.

Imagine ten female dancers standing side by side at the rear of a stage waiting to audition for selection for a chorus line in a Broadway musical. The dance director walks onto the stage, glances at the line of dancers, instantly points at a lanky blonde, and announces, “You’re first!” As that dancer steps to the center of the stage to demonstrate her talents, the director quickly surveys the line again and, nodding to a short brunette, declares, “You may go! We won’t need you.” Crestfallen, the dancer leaves, while the director, retreating several paces, begins to study the remaining candidates with great care, striding up and down the line, first in front, then behind. She stops and crosses her arms, pondering her next decision.

What are all of the yet-to-be-chosen dancers thinking about at that moment? I’ll tell you. They’re trying to guess what the director saw in the dancer chosen first and the one sent home. They’re thinking, “Should I stand up straight to look taller? Is my goose cooked because I’m a brunette? Should I smile? Should I look her in the eye? Is my makeup too heavy?” Each is trying to guess what criteria the director used in making her first two selections.

The director picked the lanky blonde first because she was strikingly attractive and had a well-proportioned figure and slender legs, perfect for a chorus line. The director sent the brunette packing because her overly high forehead made her look bald. But whereas the criteria the director used for her first two decisions were clear-cut, the water quickly muddied when she considered the other candidates. Some had slender legs but chunky figures; some had chunky legs but slender figures; some faces were bland, some distinctive. It was difficult ranking one over another. So she was forced to come up with other rationales—other criteria—on which to make her decisions.

We all do the same thing when we rank things, such as when we select a place in which to reside. We survey several neighborhoods and dislike this one because the traffic is too heavy, like that one because it’s close to an elementary school, dislike this one because it lacks city sewers, like that one because the residences are attractive and well kept, and so on. We make our decision and, after moving in, are disappointed because the roof of the house leaks and the heating bills are too expensive. It is well and good to use stringent criteria in making decisions, but we should be sure we have considered all relevant criteria and have applied all of the criteria equitably to all of the decisions. As for the dance director, she ultimately dismissed the lanky blonde; although she was a talented dancer, she couldn’t remember the dance routines.

By applying different criteria to different ranking decisions, we distort our analysis, which can lead to gravely disappointing outcomes that later make us wonder what went wrong.

This phenomenon was brought home to me when my wife and I spent a couple of days in North Carolina looking at possible sites on which to build a home. As we went from one lot to another, we noted the acreage, whether city sewers were available, where the house would best be situated, whether the rear of the house would face south, and so on. As the day went on, we focused more and more on the landscape —the number, kinds, and placement of trees and bushes. When we returned to our hotel to mull over our findings, we discovered that we had eliminated several otherwise attractive lots simply because the landscape was not ideal. Suspecting that something was wrong with our methodology, we wrote down all of our major concerns (the major factors) about the suitability of a lot for our home. Then we ranked the concerns. It turned out that landscaping was far down on the list, so we reevaluated the lots we had excluded and moved a couple of them to the top of our list.

There are three inherent weaknesses in our instinctive method. First, we tend to apply different criteria to the different items being ranked. Second, we tend to regard all of our criteria as having equal importance to us, when one can easily demonstrate that this is not the case. Third, we don’t rank every item individually against every other item.

For these reasons, rankings generated with our instinctive method tend to be inconsistent and are thus unreliable. But the degree of error is rarely a problem for us, because most of the things we rank are of a mundane, commonplace nature. But when the decisions we make in ranking are crucial, as in, say, selecting one of several lawyers to represent us or our company in a damage suit or choosing among sites for building a home or a multimillion-dollar office complex, our instinctive method poses hazardous pitfalls.

Weighted Ranking and Pair Ranking

A systematic structure makes the comparisons easier to inspect and repeat. But how do we do that? What technique helps us make the criteria explicit and apply them consistently? The answer is a technique called “weighted ranking.”

The paramount feature of weighted ranking is its method of ranking each item against every other. It’s very simple. It’s called “pair ranking.” Here’s how it works.

Let’s say we are ranking four options: A, B, C, and D. To pair-rank these options, we list them in a column one beneath the other:

Option A

Option B

Option C

Option D

We then ask ourselves:

Which is better, A or B? We put a pencil mark after the one we decide is better, say, B.

Option A

Option B |

Option C

Option D

We then ask why B is better and record our reasoning separately. If we don’t ask why we favor B over A, the voting yields only our preference, not our rationale. Yet identifying the rationale is the most important part of the process.

Which is better, A or C? We put a pencil mark after, say, C, and record our reasoning.

Option A

Option B |

Option C |

Option D

Which is better, A or D? We put a mark after, say, D, and record our reasoning.

Option A

Option B |

Option C |

Option D |

Which is better, B or C? We mark B and record our reasoning.

Option A

Option B ‖

Option C |

Option D |

Which is better, B or D? We mark B and record our reasoning.

Option A

Option B‖|

Option C |

Option D |

Which is better, C or D? We mark C and record our reasoning.

Option A

Option B ‖|

Option C ‖

Option D |

We thus systematically compare each item with every other item. If every comparison has a winner and the preferences form a strict, transitive order, one item will have three marks (votes), one two, one one, and one none. The item with three votes is the most favored and thus is ranked first. The item with two is second; one vote—third; no votes—fourth. Equal tallies can arise from cyclic preferences as well as inconsistent judgments. Revisit the reasons and criteria behind the comparisons. A head-to-head result can serve as a declared tie-breaker, but it does not necessarily resolve a cycle involving other options. Longer lists make a systematic record more useful, but they also require more comparisons.

So let’s try it. Pair-rank your list of fifteen movies and enter your rankings in the next column of the Ranking Techniques Matrix and label that column “Pair Ranking.”

BEFORE CONTINUING, PAIR-RANK THE MOVIES, ENTER YOUR PAIR RANKINGS IN THE RANKING TECHNIQUES MATRIX, AND LABEL THE COLUMN “PAIR RANKING.”

See my pair rankings in Figure 10-2.

You may feel uncomfortable with pair ranking and think that it’s too mechanical and laborious. Well, your discomfort is only natural. It’s a manifestation of your mind sending you signals telling you, “I don’t like to work this way. I prefer my loosey-goosey, unfettered, mental-messing-around instinctive approach.” Practice can make the procedure more familiar. Its value still depends on whether the comparisons clarify this decision enough to justify the effort.

But is it really “mechanical and laborious”? Mechanical, yes. That’s its beauty. It’s systematic: 1 against 2, 1 against 3, 1 against 4, 2 against 3, 2 against 4, 3 against 4. But is it “laborious”? Yes again. One has to methodically, over and over again, reason out each pair to be ranked. But that’s what makes pair ranking so valuable analytically. It doesn’t permit us to take analytic shortcuts that shortchange our analysis. We must analyze and make a decision on each member of every pair.

FIGURE 10-2

Pair ranking, by comparing one thing against another, conforms to the one-track, ever-focusing nature of our minds. It narrows the mind’s field of attention first to a single pair and then separately to each of the items in the pair. Ranking things in pairs is something the mind can do easily, quickly, and effectively. The mind does love to focus.

Pair ranking offers a systematic way to record preferences across a set of items. By focusing on individual items, pair ranking separates the elements of the task—the problem—in an organized way that, like all effective structuring techniques, allows us to analyze each element separately, systematically, and sufficiently. The tally shows where each item ranks under the judgments supplied. It does not verify those judgments or ensure that the criteria reflect what matters. What happens is that the process delivers the rankings. We don’t have to decide which movie is fifth and which is sixth or which is eleventh and which is twelfth. Instead, we rank them in pairs and the process puts them into order. Compared with an unrecorded instinctive ranking, the procedure makes the reasons for the order easier to examine and challenge.

Now, take a look at the Ranking Techniques Matrix and compare the results of your instinctive ranking with the results of your pair ranking. Do you note any differences? Twelve of my rankings changed by an average of nearly four places. Two movies moved five places, and one (Star Wars) went from thirteenth to fifth.

This exercise establishes pair ranking. A full weighted ranking also needs explicit criteria, their relative importance, and a consistent way to evaluate options against them.