Points systems.

Additive Points Systems are also known variously as ‘Scoring’, ‘Linear’ and ‘Point-Count’ Systems. In the MCDM literature they’re sometimes known as ‘additive multiple criteria value models’.

Additive Points Systems (hereinafter simply ‘Points Systems’) are a common solution to the pervasive problem in a wide variety of applications of how to combine alternatives’ characteristics on multiple criteria (or attributes) for the purpose of obtaining an overall ranking of alternatives.

A Points System is simply a schedule of ‘point values’ for each criterion, where each criterion is demarcated into mutually-exclusive levels and the point values for each level represent both the relative importance and the degree of achievement of a particular performance dimension. An example of a Points System appears below.

The criteria for a Points System may be quantitative (e.g. Experience, as below) or qualitative in nature (the four other criteria). For criteria that are not naturally categorical it is usually possible to represent them in terms of levels that are appropriate to the application (which must be listed within each criterion from lowest ranked to highest ranked).

A Points System works by each alternative (or individual) that's being considered being graded on the system’s criteria and the corresponding point values summed to obtain a total score for each alternative, by which they are ranked or classified relative to each other (where the total scores usually have no meaning other than for determining this ranking or classification). Clearly, the additive nature of (Additive) Points Systems means that, by construction, interaction effects between the criteria are ruled out (i.e. the criteria are independent).

It should be emphasised in particular that a Points System is not intended as a replacement for ‘human’ decision-making. Rather, a Points System is simply a tool to support decision-makers by increasing the validity and reliability of the decision-making process, and to make such processes more explicit and transparent (i.e. accountable).

Here is a simple example of a Points System for deciding which job applicants to hire, with five criteria and two, three or four levels on each criterion respectively:

Thus, in the simple example of the two job applicants, Peter and Fran, mentioned in the Introduction to MCDM, they would be scored as in the table below, where it can be seen that Fran is ranked ahead of Peter in terms of their total scores.

Normalised weights & single criterion scores

As well as the (unweighted) additive form above, Points Systems may also be implemented, equivalently, in terms of normalised criterion weights (summing to 1) and single criterion scores (with a common scale, such as from 0 to 100 points).

Thus the Points System above can be expressed, equivalently (demonstrated below), as:

The equivalence of these two representations (unweighted and weighted) can be easily confirmed by calculating Peter’s and Fran’s total scores using both approaches.

(1) Unweighted Points System

Peter’s total score = 20 + 9 + 0 + 10 + 0 = 39 points

Fran’s total score = 0 + 9 + 20 + 10 + 12 = 51 points (both as in the 2nd table above)

(2) ‘Normalised criterion weights and single criterion scores’

Peter’s total score = (0.4 x 50) + (0.18 x 50) + (0.20 x 0) + (0.10 x 100) + (0.12 x 0) = 39 points

Fran’s total score = (0.4 x 0) + (0.18 x 50) + (0.20 x 100) + (0.10 x 100) + (0.12 x 100) = 51 points

Although they are equivalent, many users consider the unweighted representation to be simpler to use.

Simplicity & accuracy

As mentioned earlier, Points Systems (in either of the representations above) are used in a wide variety of applications. As well as being simple to use, Points Systems have been found to be more accurate than the unaided ‘expert’ judgements of decision-makers.

This is the near-universal finding of a large number of studies, as surveyed by Hastie & Dawes (2010 – references appear at the end of this page), who concluded that (p. 52):

“... the simple linear model [i.e. Points System] is surprisingly successful in many applications. We say ‘surprisingly’, because many judges claim that their mental processes are much more complex than the linear summary equations would suggest – although empirically, the equation does a remarkably good job of ‘capturing’ their judgment habits.”

This is because (p. 60):

“The mind is in many essential respects a linear weighting and adding device. In fact, much of what we know about the neural networks in the physical brain suggests that a natural computation for such a ‘machine’ is weighting and adding, exactly the fundamental processes that are well described by linear equations [i.e. Points Systems].”

Similarly, according to Hadorn et al. (2003, p. 49):

“[Points Systems] are, as a rule, more accurate than human predictors. This is not surprising, as it has been known for decades that human cognitive performance is limited with respect to complex multi-variable judgment and prediction tasks (Meehl 1954).”

Creating a points system

When creating a Points System, two things are essential in order for it to be accurate (in terms of helping users make the ‘right’ decisions):

• The decision criteria included must be relevant to the decision being made.

• Their relative importance must be accurately represented by the Points System’s point values – so that the decision maker’s preferred ranking of individuals/alternatives is reproduced by the ranking of total scores.

For example, with respect to (2), are the point values in the Points System shown above (in the tables) right? How would you know? Certainly, they will result in a particular ranking of job applicants. Clearly, though, by changing the point values, the ranking will change (likely affecting which applicants are selected – and which are not). Should the top score for “Qualifications”, for example, really be 40 points? How about 36, 38, or 32? (Or even 20 or 60?!)

A variety of methods is available for determining the point values for a Points System (sometimes known as ‘scoring’ the Points System). The method used by the 1000Minds’ technology is known by the acronym PAPRIKA (‘Potentially All Pairwise RanKings of all possible Alternatives’).

References

DC Hadorn & the Steering Committee of the Western Canada Waiting List Project (2003) “Setting priorities on waiting lists: Point-count systems as linear models”. Journal of Health Services Research & Policy 8, 48-54.

R Hastie & RM Dawes (2010) Rational Choice in an Uncertain World. The Psychology of Judgement & Decision Making. (2nd edition) Sage Publications.

PE Meehl (1954) Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. University of Minnesota Press.