Potentially all pairwise rankings of all possible alternatives


Potentially All Pairwise RanKings of all possible Alternatives is a method for multi-criteria decision making or conjoint analysis, as implemented by decision-making software and conjoint analysis products 1000minds and MeenyMo.
The PAPRIKA method is based on users expressing their preferences with respect to the relative importance of the criteria or attributes of interest for the decision or choice at hand by pairwise comparing alternatives.
In MCDM applications, PAPRIKA is used by decision-makers to determine weights on the criteria for the decision being made, representing their relative importance. Depending on the application, these weights are used to rank, prioritize or choose between alternatives.
In conjoint analysis applications, PAPRIKA is used with consumers or other stakeholders to estimate ‘part-worth utilities’ representing the relative importance of the attributes characterizing products or other objects of interest.

Applications

The PAPRIKA method is implemented by decision-making software and conjoint analysis products 1000minds and MeenyMo.
Examples of areas in which the method is used for multi-criteria decision making or conjoint analysis include :
The PAPRIKA method specifically applies to additive multi-attribute value models with performance categories – also known as ‘points’, ‘scoring’, ‘point-count’ or ‘linear’ systems or models. The following explanations are mostly couched in terms of multi-criteria decision making. Analogous explanations in terms of conjoint analysis are possible but not presented here.
As the name implies, additive multi-attribute value models with performance categories – hereinafter referred to simply as ‘value models’ – consist of multiple criteria, with two or more performance categories within each criterion, that are combined additively.
Each category within each criterion is worth a certain number of points that is intended to reflect both the relative importance of the criterion and its degree of achievement. For each alternative being considered, the point values are summed across the criteria to get a total score – hence, these are additive value models – by which the alternatives are prioritized or ranked relative to each other.
Thus, a value model is simply a schedule of criteria and point values for the decision problem at hand; for an example, see Table 1 in the sub-section below. This ‘points system’ representation is equivalent to a more traditional approach involving normalized criterion weights and ‘single-criterion value functions’ to represent the relative importance of the criteria and to combine values overall. The unweighted points system representation is easier to use and helps inform the explanation of the PAPRIKA method below.

An example application of a points system

An example application of a points system is ranking candidates applying for a job.
Imagine that ‘Tom’, ‘Dick’ and ‘Harry’ are three job candidates to be ranked using the value model in Table 1 below. Suppose they are assessed on the five criteria like this:
Table 1: Example of a value model for ranking job candidates
CriterionCategoryPoints
Educationpoor0
good8
very good20
excellent40
Experience< 2 years0
2–5 years3
> 5 years10
Referencespoor0
good27
Social skillspoor0
good10
Enthusiasmpoor0
good13

Summing the point values in Table 1 corresponding to the descriptions for Tom, Dick and Harry gives their total scores:
Clearly, Harry has the highest total score. Therefore, according to the value model Harry is the best candidate for the job.
In general terms, having specified the criteria and categories for a given value model, the challenge is to derive point values that accurately reflect the relative importance of the criteria and categories to the decision-maker. Deriving valid and reliable point values is arguably the most difficult task when creating a value model. The PAPRIKA method does this based on decision-makers’ preferences as expressed using pairwise rankings of alternatives.

Overview of the PAPRIKA method

As mentioned at the start of the article, PAPRIKA is a acronym for ‘Potentially All Pairwise RanKings of all possible Alternatives’. The following explanation should make clear this name's derivation.
The PAPRIKA method pertains to value models for ranking particular alternatives that are known to decision-makers and also to models for ranking potentially all hypothetically possible alternatives in a pool that is changing over time. The following explanation is centered on this second type of application because it is more general.
PAPRIKA is based on the fundamental principle that an overall ranking of all possible alternatives representable by a given value model – i.e. all possible combinations of the categories on the criteria – is defined when all pairwise rankings of the alternatives vis-à-vis each other are known.
However, depending on the number of criteria and categories included in the value model, the number of pairwise rankings of all possible alternatives is potentially in the millions or even billions. Of course, though, many of these pairwise rankings are automatically resolved due to one alternative in the pair having a higher category for at least one criterion and none lower for the other criteria than for the other alternative – known as ‘dominated pairs’.
But this still leaves potentially millions or billions of ‘undominated pairs’ – pairs of alternatives where one has a higher ranked category for at least one criterion and a lower ranked category for at least one other criterion than the other alternative – and hence a judgment is required for the alternatives to be pairwise ranked. With reference to the example of ranking job candidates in the previous section, an example of an undominated pair would be where one person in the pair is, say, highly educated but inexperienced whereas the other person is uneducated but highly experienced, and so a judgement is required to pairwise rank this pair.
For n possible alternatives, there are n/2 pairwise rankings. For example, for a value model with eight criteria and four categories within each criterion, and hence 48 = 65,536 possible alternatives, there are 65,536 x 65,535 / 2 = 2,147,450,880 pairwise rankings. Even after eliminating the 99,934,464 dominated pairs, there are still 2,047,516,416 undominated pairs to be ranked. Clearly, performing anywhere near this number of pairwise rankings – more than two billion! – is humanly impossible without a special method.
The PAPRIKA method resolves this ‘impossibility’ problem by ensuring that the number of pairwise rankings that decision-makers need to perform is kept to a minimum – i.e. only a small fraction of the potentially millions or billions of undominated pairrs – so that the burden on decision-makers is minimized and the method is practicable. PAPRIKA keeps the number of pairwise rankings performed by decision-makers to a minimum by, for each undominated pair explicitly ranked by decision-makers, identifying all undominated pairs implicitly ranked as corollaries of this and other explicitly ranked pairs. Fundamental to the efficiency of the method is application of the transitivity property of additive value models, as illustrated in the simple demonstration later below.
The PAPRIKA method begins with the decision-maker pairwise ranking undominated pairs defined on just two criteria at-a-time. Again with reference to the example of ranking job candidates, an example of such a pairwise-ranking question is: “Who would you prefer to hire, someone whose education is poor but he or she has 5 years or more experience or another person whose education is excellent but he or she has less than 2 years experience, all else being equal?”.
Figure 1: Example of a pairwise-ranking question
Each time the decision-maker ranks a pair, all undominated pairs implicitly ranked as corollaries are identified and discarded. After having completed ranking undominated pairs defined on just two criteria at-a-time, this is followed, if the decision-maker chooses to continue, by pairs with successively more criteria, until potentially all undominated pairs are ranked.
Thus, Potentially All Pairwise RanKings of all possible Alternatives are identified as either: dominated pairs, or undominated pairs explicitly ranked by the decision-maker, or undominated pairs implicitly ranked as corollaries. From the explicitly ranked pairs, point values are obtained via linear programming; although multiple solutions to the linear program are possible, the resulting point values all reproduce the same overall ranking of alternatives.
Simulations of PAPRIKA’s use reveal that if the decision-maker stops after having ranked undominated pairs defined on just two criteria at-a-time, the resulting overall ranking of all possible alternatives is very highly correlated with the decision-maker’s ‘true’ overall ranking obtained if all undominated pairs were ranked.
Therefore, for most practical purposes decision-makers are unlikely to need to rank pairs defined on more than two criteria, thereby reducing the burden on decision-makers. For example, approximately 95 explicit pairwise rankings are required for the value model referred to above with eight criteria and four categories each ; 25 pairwise rankings for a model with five criteria and three categories each; and so on. The real-world applications of PAPRIKA referred to earlier suggest that decision-makers are able to rank comfortably more than 50 and up to at least 100 pairs, and relatively quickly, and that this is sufficient for most applications.

Theoretical antecedents

The PAPRIKA method’s closest theoretical antecedent is Pairwise Trade-off Analysis, a precursor to Adaptive Conjoint Analysis in marketing research. Like the PAPRIKA method, Pairwise Trade-off Analysis is based on the idea that undominated pairs that are explicitly ranked by the decision-maker can be used to implicitly rank other undominated pairs. Pairwise Trade-off Analysis was abandoned in the late 1970s, however, because it lacked a method for systematically identifying implicitly ranked pairs.
The ZAPROS method was also proposed; however, with respect to pairwise ranking all undominated pairs defined on two criteria “it is not efficient to try to obtain full information”. As explained in the present article, the PAPRIKA method overcomes this efficiency problem.

A simple demonstration of the PAPRIKA method

The PAPRIKA method can be easily demonstrated via the simple example of determining the point values on the criteria for a value model with just three criteria – denoted by ‘a’, ‘b’ and ‘c’ – and two categories within each criterion – ‘1’ and ‘2’, where 2 is the higher ranked category.
This value model’s six point values can be represented by the variables a1, a2, b1, b2, c1, c2, and the eight possible alternatives as ordered triples of the categories on the criteria : 222, 221, 212, 122, 211, 121, 112, 111. These eight alternatives and their total score equations – derived by simply adding up the variables corresponding to the point values – are listed in Table 2.
Undominated pairs are represented as ‘221 vs 212’ or, in terms of the total score equations, as ‘a2 + b2 + c1 vs a2 + b1 + c2’, etc.
‘Scoring’ this model involves determining the values of the six point value variables so that the decision-maker’s preferred ranking of the eight alternatives is realized.
For many readers, this simple value model can perhaps be made more concrete by considering an example to which most people can probably relate: a model for ranking job candidates consisting of the three criteria education, experience, and references, each with two ‘performance’ categories, poor or good.
Accordingly, each of this model’s eight possible alternatives can be thought of as being a ‘type’ of candidate who might ever, hypothetically, apply. For example, ‘222’ denotes a candidate who is good on all three criteria; ‘221’ is a candidate who is good on education and experience but poor on references; ‘212’ a third who is good on education, poor on experience, and good on references; etc.
Finally, with respect to undominated pairs, 221 vs 212, for example, represents candidate 221 who has good experience and poor references whereas 212 has the opposite characteristics. Thus, which is the better candidate ultimately depends on the decision-maker’s preferences with respect to the relative importance of experience vis-à-vis references.
Table 2: The eight possible alternatives and their total-score equations
AlternativeTotal-score equation
222a2 + b2 + c2
221a2 + b2 + c1
212a2 + b1 + c2
122a1 + b2 + c2
211a2 + b1 + c1
121a1 + b2 + c1
112a1 + b1 + c2
111a1 + b1 + c1

Identifying undominated pairs

The PAPRIKA method's first step is to identify the undominated pairs. With just eight alternatives this can be done by pairwise comparing all of them vis-à-vis each other and discarding dominated pairs.
This simple approach can be represented by the matrix in Figure 2, where the eight possible alternatives are listed down the left-hand side and also along the top. Each alternative on the left-hand side is pairwise compared with each alternative along the top with respect to which of the two alternatives is higher ranked. The cells with hats denote dominated pairs and the empty cells are either the central diagonal or the inverse of the non-empty cells containing the undominated pairs.
Figure 2: Undominated pairs identified by pairwise comparing the eight possible alternatives
vs222221212122112121211111
222^^^^^^^
221 b2 + c1 vs b1 + c2 a2 + c1 vs a1 + c2 a2 + b2 + c1 vs a1 + b1 + c2^^^
212 a2 + b1 vs a1 + b2^ a2 + b1 + c2 vs a1 + b2 + c1^^
122^^ a1 + b2 + c2 vs a2 + b1 + c1^
112 b1 + c2 vs b2 + c1 a1 + c2 vs a2 + c1^
121 a1 + b2 vs a2 + b1^
211^
111

Figure 2 notes: ^ denotes dominated pairs. The undominated pairs are labelled with Roman numerals; the three with asterisks are duplicates of pairs -.
As summarized in Figure 2, there are nine undominated pairs. However, three pairs are duplicates after any variables common to a pair are ‘cancelled’. Thus, there are six unique undominated pairs.
The cancellation of variables common to undominated pairs can be illustrated as follows. When comparing alternatives 121 and 112, for example, a1 can be subtracted from both sides of a1 + b2 + c1 vs a1 + b1 + c2. Similarly, when comparing 221 and 212, a2 can be subtracted from both sides of a2 + b2 + c1 vs a2 + b1 + c2. For both pairs this leaves the same ‘cancelled’ form: b2 + c1 vs b1 + c2.
Formally, these subtractions reflect the ‘joint-factor’ independence property of additive value models: the ranking of undominated pairs is independent of their tied rankings on one or more criteria. Notationally, undominated pairs in their cancelled forms, like b2 + c1 vs b1 + c2, are also representable as _21 ‘‘vs’’ _12 – i.e. where ‘_’ signifies identical categories for the identified criterion.
In summary, here are the six undominated pairs for the value model:
The task is to pairwise rank these six undominated pairs, with the objective that the decision-maker is required to perform the fewest pairwise rankings possible.

Ranking undominated pairs and identifying implicitly ranked pairs

Undominated pairs with just two criteria are intrinsically the least cognitively difficult for the decision-maker to pairwise rank relative to pairs with more criteria. Thus, arbitrarily beginning here with pair b2 + c1 vs b1 + c2, the decision-maker is asked: “Which alternative do you prefer, _21 or _12, or are you indifferent between them?” This choice, in other words, is between a candidate with good experience and poor references and another with poor experience and good references, all else the same.
Suppose the decision-maker answers: “I prefer _21 to _12”. This preference can be represented by ‘_21 ≻_12’, which corresponds, in terms of total score equations, to b2 + c1 > b1 + c2 .
Central to the PAPRIKA method is the identification of all undominated pairs implicitly ranked as corollaries of the explicitly ranked pairs. Thus, given a2 > a1, it is clear that b2 + c1 > b1 + c2 implies pair is ranked as a2 + b2 + c1 > a1 + b1 + c2. This result reflects the transitivity property of value models. Specifically, 221≻121 and 121≻112 implies 221≻112; equivalently, 212≻112 and 221≻212 implies 221≻112.
Next, corresponding to pair a2 + c1 vs a1 + c2, suppose the decision-maker is asked: “Which alternative do you prefer, 1_2 or 2_1, or are you indifferent between them?” This choice, in other words, is between a candidate with poor education and good references and another with good education and poor references, all else the same.
Suppose the decision-maker answers: “I prefer 1_2 to 2_1”. This preference corresponds to a1 + c2 > a2 + c1. Also, given b2 > b1, this preference/inequality implies pair is ranked as a1 + b2 + c2 > a2 + b1 + c1.
Furthermore, the two explicitly ranked pairs b2 + c1 > b1 + c2 and a1 + c2 > a2 + c1 imply pair is ranked as a1 + b2 > a2 + b1. This result can easily be seen by adding the corresponding sides of the inequalities for pairs and and cancelling common variables. Again, this result reflects the transitivity property: 121≻112 and 112≻211 implies 121≻211; equivalently, 122≻221 and 221≻212 implies 122≻212.
As a result of two explicit pairwise comparisons – i.e. explicitly performed by the decision-maker – five of the six undominated pairs have been ranked. The decision-maker may cease ranking whenever she likes, but let's suppose she continues and ranks the remaining pair as a2 + b1 + c2 > a1 + b2 + c1.
Thus, all six undominated pairs have been ranked as a result of the decision-maker explicitly ranking just three:

The overall ranking of alternatives and point values

Because the three pairwise rankings above are consistent – and all n /2 = 28 pairwise rankings for this simple value model are known – a complete overall ranking of all eight possible alternatives is defined : 222, 122, 221, 212, 121, 112, 211, 111.
Simultaneously solving the three inequalities above, subject to a2 > a1, b2 > b1 and c2 > c1, gives the point values, reflecting the relative importance of the criteria to the decision-maker. For example, one solution is: a1 = 0, a2 = 2, b1 = 0, b2 = 4, c1 = 0 and c2 = 3.
Thus, in the context of the example of a value model for ranking job candidates, the most important criterion is revealed to be experience followed by references and, least important, education. Although multiple solutions to the three inequalities are possible, the resulting point values all reproduce the same overall ranking of alternatives as listed above and reproduced here with their total scores:

Additional considerations

First, the decision-maker may decline to explicitly rank any given undominated pair on the grounds that at least one of the alternatives considered corresponds to an impossible combination of the categories on the criteria. Also, if the decision-maker cannot decide how to explicitly rank a given pair, she may skip it – and the pair may eventually be implicitly ranked as a corollary of other explicitly ranked pairs.
Second, in order for all undominated pairs to be ranked, the decision-maker will usually be required to perform fewer pairwise rankings if some indicate indifference rather than strict preference. For example, if the decision-maker had ranked pair above as _21~_12 instead of _21≻_12, then she would have needed to rank only one more pair rather than two. On the whole, indifferently ranked pairs generate more corollaries with respect to implicitly ranked pairs than pairs that are strictly ranked.
Finally, the order in which the decision-maker ranks the undominated pairs affects the number of rankings required. For example, if the decision-maker had ranked pair before pairs and then it is easy to show that all three would have had to be explicitly ranked, as well as pair . However, determining the optimal order is problematical as it depends on the rankings themselves, which are unknown beforehand.

Applying PAPRIKA to 'larger' value models

Of course, most real-world value models have more criteria and categories than the simple example above, which means they have many more undominated pairs. For example, the value model referred to earlier with eight criteria and four categories within each criterion has 2,047,516,416 undominated pairs in total, of which, excluding replicas, 402,100,560 are unique.
For such real-world value models, the simple pairwise-comparisons approach to identifying undominated pairs used in the previous sub-section is highly impractical. Likewise, identifying all pairs implicitly ranked as corollaries of the explicitly ranked pairs becomes increasingly intractable as the numbers of criteria and categories increase. The PAPRIKA method, therefore, relies on computationally efficient processes for identifying unique undominated pairs and implicitly ranked pairs respectively. The details of these processes are beyond the scope of this article, but are available elsewhere and, as mentioned earlier, the PAPRIKA method is implemented by decision-making software products 1000minds and MeenyMo.

Comparison with traditional scoring methods

PAPRIKA entails a greater number of judgments than most ‘traditional’ scoring methods, such as direct rating, SMART, SMARTER and the Analytic Hierarchy Process. Clearly, though, different types of judgments are involved. For PAPRIKA, the judgements entail pairwise comparisons of undominated pairs, whereas most traditional methods involve interval scale or ratio scale measurements of the decision-maker's preferences with respect to the relative importance of criteria and categories respectively. Arguably, the judgments for PAPRIKA are simpler and more natural, and therefore they might reasonably be expected to reflect decision-makers’ preferences more accurately.