The ‘PAPRIKA’ method.

1000Minds applies our patented ‘PAPRIKA’ (Potentially All Pairwise RanKings of all possible Alternatives) method for Multi-Criteria Decision-Making and Conjoint Analysis.

The PAPRIKA method is based on the fundamental principle that an overall ranking of all possible alternatives that might ever be considered by decision-makers is defined when all pairwise rankings of the alternatives vis-à-vis each other are known (provided they’re consistent).

As an analogy, suppose you wanted to rank 5000 people from the tallest to smallest. If you pairwise ranked each individual relative to everyone else – i.e. for each possible pair of individuals, by identifying the taller person or that they’re the same height – an overall ranking of all 5000 can be produced.

In general, however, depending on the number of possible alternatives, the number of pairwise rankings is potentially in the millions (even billions!). Without a special method, considering them all is impossible. Exploiting the properties of additive Points Systems, PAPRIKA overcomes this problem.

Thus, using 1000Minds, decision-makers need to perform only a tiny fraction (e.g. < 100) of the potentially millions of pairwise rankings in order to identify ‘Potentially All Pairwise RanKings of all possible Alternatives’ (i.e. ‘PAPRIKA’) – thereby producing an overall ranking of alternatives.

As well as an overall ranking of alternatives – of all possible alternatives and/or particular ones considered by decision-makers – 1000Minds calculates the weights on the decision criteria, representing the relative importance of the criteria to decision-makers.

This and more is explained below, and the mathematical and technical details are in our article:

As well as being technically advanced, 1000Minds is very intuitive and user-friendly, as illustrated next.

What do you want to do?

Depending on the activity mode selected (at the ‘start / mode’ step in 1000Minds), 1000Minds supports these three main (related) activities:

• Discover decision-makers’ preference values – representing the relative importance (‘weights’) of the criteria entered into 1000Minds. For some applications this includes creating a Points System (also known as a ‘Scoring’, ‘Point-Count’ or ‘Linear’ System or Model).

• Rank or prioritise alternatives entered into 1000Minds (and all hypothetically possible alternatives, if desired) – including finding the top-ranked alternative, etc.

• Select alternatives, subject perhaps to a budget constraint – including, when appropriate, considering their ‘value for money’ and allocating resources.

Having selected your preferred activity mode, all you have to do is perform the following simple steps – where the first four steps correspond to the first activity above, the first five steps to the second activity, and all six to the third activity.

In addition, customisable processes to include potentially 10s or 100s (even 1000s!) of participants in a variety of decision-making activities are available.

For completeness, the third activity above is explained in detail here, as it, in effect, encompasses the first two activities (the first four or five steps).

Step 1: Your criteria

The first thing to do is enter your criteria for your particular decision-making application. The criteria can be in qualitative or quantitative terms. Within reason, you can enter as many as criteria as appropriate, and as many levels within each criterion (it’s up to you!).

As an illustration, let’s consider the example of prioritising projects for an organisation – an application most people can probably relate to (or think of an application of your own that involves ranking alternatives or individuals – e.g. see solutions).

For this example the following criteria might be appropriate (all illustrations here are 1000Minds screenshots):

Other considerations that are relevant for prioritising projects, such as 'cost’ and ‘confidence in cost estimates’, etc, are introduced later below. For now, the criteria have been chosen exclusively with respect to possible benefits associated with the alternatives (projects) being considered.

Instead of guessing the relative importance of the criteria – i.e. their ‘weights’ or ‘point values’ – 1000Minds determines them in a systematic, transparent and repeatable way, as explained at the Your decisions step later below.

Step 2: Alternatives

You can describe the alternatives you’re considering in terms of your criteria either before or after answering the tradeoff questions at the next step. If you don’t have any particular alternatives in mind yet, that’s fine – you can enter them whenever you like.

Step 3: Your decisions

In order to derive your preference values (i.e. ‘weights’ or ‘point values’) – representing the relative importance of the criteria – all you have to do is answer a series of simple pairwise ranking questions.

As in the example below, each question involves two hypothetical alternatives (‘projects’) at a time, described according to just two criteria and involving a tradeoff between the criteria.

Such simple questions are repeated with different pairs of hypothetical alternatives, all involving tradeoffs between the criteria, until enough information about your preferences has been collected to accurately rank the alternatives being considered.

As mentioned earlier, this ‘scoring’ method – which is patented and has won innovation awards – is known by the acronym ‘PAPRIKA’ (Potentially All Pairwise RanKings of all possible Alternatives).

The number of questions to be answered is minimised because each time you answer a question like the one above, PAPRIKA eliminates all other potential questions that are implicitly answered as corollaries of those already answered.

In essence, PAPRIKA does this by exploiting the ‘transitivity’ property – e.g. if alternative A is ranked above alternative B, and B is ranked above alternative C, then, logically (‘by transitivity’), A is ranked above C.

Approximately 30 decisions would be required for the present example. The number of decisions depends on the number of criteria and levels entered: the more criteria and/or levels, the more decisions.

PAPRIKA / 1000Minds keeps track of all of the potentially millions of pairwise rankings of the hypothetical alternatives implied by decision-makers’ answers (as a result of transitivity). This is the remarkable feat achieved by 1000Minds – and all in ‘real time’ (and very efficiently).

Thus PAPRIKA identifies ‘Potentially All Pairwise RanKings of all possible Alternatives’ representable by the criteria and their levels. Because the pairwise rankings are consistent, a complete overall ranking of alternatives is defined.

The advantage of this method (‘pairwise ranking’) relative to alternative methods, which usually rely on scaling or ratio measurements of participants’ preferences, is that choosing one alternative from two is a natural type of decision activity that everyone has experience of in their daily lives.

In contrast, other methods such as the Analytic Hierarchy Process (AHP) ask: e.g. On a scale from 1 to 9, how many times more or less important is “Urgency / importance ...” than “Likely effectiveness ...” for prioritising projects?

Clearly, 1000Minds questions are more natural – because picking one alternative from just two possibilities is the easiest choice in the world. This means you can have greater confidence in your answers and ultimately your final decisions.

Step 4: Preference values

Based on your decisions, your preference values (i.e. ‘weights’ or ‘point values’) – representing the relative importance of the criteria – are derived by 1000Minds. Depending on the application, your ‘Points System’ can be used repeatedly as new alternatives enter the frame and old ones exit.

Step 5: Ranked alternatives

As noted earlier, you can describe the alternatives you’re considering in terms of your criteria whenever you like.

After you’ve answered the pairwise ranking questions (at the Your decisions step above), 1000Minds automatically scores each alternative according to your preference values (as above) and, based on the alternatives’ total scores, ranks them from first to last.

As well as entered alternatives, 1000Minds automatically scores and ranks all hypothetically possible alternatives (all combinations of the criteria and their levels). Sometimes it is useful to consider how such possible alternatives – e.g. that might be considered in the future – will be ranked.

You can also include ‘other considerations’ that are relevant – i.e. in addition to your original criteria entered earlier. As in the table above, examples include ‘cost’ and ‘confidence in cost estimates’. Other examples (depending on the application) include ‘strength of evidence’, etc.

Such other considerations (e.g. ‘cost’) that have numerical data can be graphed against alternatives’ total scores – representing their benefits – on the axes in the chart below (in the context of the present example representing Value for Money).

The size and colour of the bubbles in the chart can be used to represent additional ‘other considerations – e.g. ‘strength of evidence’, etc (or to highlight a particular criterion of special interest, such as ‘fit with organisation’s strategic priorities’, as below).

Thus, depending on what you want to show, 2 - 4 variables of interest can be represented in the chart, to help you make your selections.

Step 6: Your selections

Assisted by the chart below – in the present context, the ‘Value for Money Chart’ – you can select your preferred alternatives (projects). Depending on the application, this may involve choosing the ‘best’ alternative or multiple alternatives.

In many cases, such selections will be subject to a budget constraint (or resource constraint, more generally). This can also be entered into 1000Minds (see the bottom of the table below – e.g. $8 million), so that you are continually apprised of how much you’ve spent and what’s left respectively.

To appreciate the analytical power of the Value for Money Chart, ask yourself (with reference to the chart): With a budget of, say, $8 million, which projects would you select?

Clearly, the focus of attention in the chart is the top-left quadrant – i.e. high Total Scores (Benefits) and low Costs. Possible acceptable tradeoffs are in a nor-easterly direction, along the blue line representing the ‘Pareto (efficiency) frontier’.

(This is ignoring for now – for simplicity – the size and colour of the bubbles, reflecting other potentially important considerations, such as ‘confidence in cost estimates’, etc.)

In addition, the projects can also be ranked in the table according to their Benefit-Cost Ratios (BCR) (i.e. “y/x ratio” in the table) – or any other variables of interest.

As projects are selected (or rejected) in the table – by clicking the ‘ticks’ (or ‘crosses’) in the table – the chart and the totals at the bottom of the table (i.e. totals spent, rejected, not yet allocated) update automatically.

Reports & accessories

• Everything done with 1000Minds is recorded in a report, which lends transparency and focus to discussions with stakeholders.

• If priorities change, 1000Minds can be fine-tuned easily, further earning the confidence of stakeholders.

• After decision-making you can test yourself (or your group of decision-makers) for consistency.

• You can aggregate the preferences of multiple decision-makers and pin-point areas of disagreement right down to individual decisions.

• You can let other users (in other locations) see the same questions as you – so they can be involved in the decision-making too (via phone or email) or simply observe you.

• You can test how your preferences compare with an existing Points System (perhaps that you’ve used in the past).

• You can compare the rankings of alternatives produced by different Points Systems.

1000Minds decision processes

In addition – depending on what you want to do – you can also use 1000Minds to create your own customised processes involving a variety of decision-making activities.

Each process can include as many participants as you like – e.g. potentially 10s or 100s (even 1000s!) of participants.

For more information, please see 1000Minds decision processes.