Rank reversals in decision-making


In decision-making, a rank reversal is a change in the rank ordering of the preferability of alternative possible decisions when, for example, the method of choosing changes or the set of other available alternatives changes. The issue of rank reversals lies at the heart of many debates in decision-making and multi-criteria decision-making, in particular.
Unlike most other computational procedures, it is hard to tell if a particular decision-making method has derived the correct answer or not. Such methods analyze a set of alternatives described in terms of some criteria. They determine which alternative is the best one, or they provide relative weights of how the alternatives perform, or just how the alternatives should be ranked when all the criteria are considered simultaneously. This is exactly where the challenge with decision making exists. Often it is hard, if not practically impossible, to determine whether a correct answer has been reached or not. With other computational methods, for instance with a job scheduling method, one can examine a set of different answers and then categorize the answers according to some metric of performance. But this may not be possible to do with the answers derived by most decision making methods. After all, determining the best decision making method leads to a decision making paradox.
Thus the following question emerges: How can one evaluate decision-making methods? This is a very difficult issue and may not be answered in a globally accepted manner.
A critical part in answering this fundamental question is played by what is known as rank reversals.

Rank reversal

One way to test the validity of decision-making methods is to construct special test problems and then study the solutions they derive. If the solutions exhibit some logic contradictions, then one may argue that something is wrong with the method that derived them.
To see the above point more clearly, suppose that three candidates are evaluated for some job opening. Let us designate these candidates as A, B, and C. Suppose that some decision making method has determined that the best candidate for that job is person A, followed by B, who is followed by C. This is the first ranking and it is indicated as follows: A > B > C. Next, suppose that candidate B is replaced by an even worse candidate, say person D. That is, now we have B > D, and candidate B is replaced by D while candidates A and C remain in the pool of candidates with exactly the same characteristics as before. When the new set of alternatives are ranked together and by assuming that the criteria have exactly the same weights as before, then should not candidate A still be the best one? It turns out that under some decision making methods the best alternative may be different now. This is known as a rank reversal and it is one of the types of rank reversals.
The first type of rank reversal in the above context was observed by Belton and Gear in 1983 as part of a study of the analytic hierarchy process. They first considered a simple decision problem comprised by 3 alternatives and 2 criteria. Next a copy of a non-optimal alternative was introduced. When the 4 alternatives were evaluated, and under the assumption that the criteria weights are exactly the same as before, it was observed that now the indication of the best alternative can change. That is, a rank reversal may occur with the AHP. A few years later it was observed that the AHP, as well as a new variant to it that was introduced by Professor Thomas Saaty in response to the previous observation by Belton and Gear, may exhibit rank reversals when a non-optimal alternative is replaced by a worse one.
The issue of rank reversals has captured the interest of many researchers and practitioners in the field of decision-making. It is something that continues to be considered controversial by many and is frequently debated.

Different types of rank reversals

There are many different types of rank reversals, depending on how the alternatives in a problem are defined and evaluated. These types are described next as Type 1, Type 2, Type 3, Type 4, and Type 5.

Rank reversals of Type 1

As stated earlier, one may introduce identical or near-identical copies of non-optimal alternatives and then check to see if the indication of the best alternative changes or not.

Rank reversals of Type 2

Another way is to replace a non-optimal alternative with a worse one and then see if the indication of the best alternative changes or not.

Rank reversals of Type 3

First consider a problem with all the alternatives together and get a ranking. Next, decompose the original problem into a set of smaller problems defined on two alternatives at a time and the same criteria as before. Get the rankings of these smaller problems and check to see if they are in conflict with the ranking of the alternatives of the original problem.

Rank reversals of Type 4

Type 4 is like Type 3, but ignores the ranking of the original problem. Instead, check to see if the rankings of the smaller problems are in conflict with each other. For instance, suppose that the following 3 alternatives A, B, and C are considered. Next, suppose that some 2-alternative problems are solved and the rankings A > B, B > C, and C > A, are derived from these 2-alternative problems. Obviously, the above situation indicates a case of non-transitivity as we get A > B > C > A.

Rank reversals of Type 5

All previous types of rank reversals are known to occur with the analytic hierarchy process and its additive variants, the TOPSIS and ELECTRE methods and their variants.
The weighted product model does not exhibit the previous types of rank reversals, due to the multiplication formula it uses. However, the WPM does cause rank reversals when it is compared with the weighted sum model and under the condition that all the criteria of a given decision problem can be measured in exactly the same unit. The same is true with all the previous methods as well. This is the Type 5 ranking reversal.
It is quite possible to define more types of rank reversals. One only needs to determine ways to alter a test problem and see how the ranking of the alternatives of the new problem differs from the original ranking of the alternatives of the original problem. Furthermore, the difference in rankings, somehow, should indicate the presence of undesirable effects.

Are rank reversals always undesirable?

Decision-making methods are used to make decisions in many aspects of human activity. This is especially true with decisions that involve large amounts of money or decisions that may have huge impact on large numbers of people. Given the well-established fact that different methods may yield different answers when they are fed with exactly the same problem, the question is how to evaluate them. Rank reversals are at the very heart of assessing the merits of such methods. At the same time, they are at the center of many heated debates in this area. Many authors use them as means to criticize decision making methods or to better explain rational behavior.
Consider a simple example of buying a car. Suppose that there are two cars available to the decision maker: Car A and Car B. Car A is much cheaper than Car B but its overall quality is much less when compared to that for Car B. On the other hand, Car B is more expensive than Car A but it is also of better quality. A decision maker who is concerned of the high price issue, may choose Car A over the better quality and more expensive Car B. Next suppose that the car dealer presents to the decision maker a third car, say Car C, which is way more expensive than Car B but now the overall quality of Car C is marginally higher than that of Car B. Under such a scenario, it is quite possible for a decision maker to alter his/her opinion and purchase Car B instead of Car A, even if he/she has not actually seen Car C.
Such events may take place with many rational decision makers. In other words, rank reversals may actually be possible in rational decision making. The issue of having rank reversals by rational decision makers has been studied extensively by Amos Tversky. In other words, having rank reversals in certain occasions and of certain types may not be indicative to faulty decision making. However, the key question is how to be able to distinguish when rank reversals indicate that something is wrong or when they do not conflict rational decision making. This is a highly debated issue in the decision making community.

Methods that have been verified to exhibit rank reversals

The following is just a partial list of multi-criteria decision making methods which have been confirmed to exhibit various types of rank reversals: