Use 1000minds’ to create a consensus-based tool or system for prioritizing patients for access to health care, e.g. elective surgery.

The process outlined below makes use of three types of 1000minds survey: ‘Ranking’, ‘Categorization’ and ‘Preferences’. A Voting option is also available to supplement a Preferences Survey.

The steps in the process outlined below can be adapted, depending on your needs. For example, if you’re in a hurry, steps 1-5 can be dispensed with (i.e. assuming you already know you need a new tool and the criteria you want to use – but do you really?)

1. Show the need for a new tool
Collate 10-15 patient vignettes (case studies) and run a Ranking Survey – a “noise audit” – to capture individual clinicians’ intuitive rankings. Show them how variable they are (e.g. see ‘Noisy’ expert judgments), and that, therefore, a new prioritization tool is needed!
2. Elicit prioritization criteria
As the group of clinicians discusses the Ranking Survey results – i.e. which patients should be prioritized and why – start specifying the prioritization criteria. Run the same Ranking Survey but this time by consensus to get a pseudo-gold standard for step #8 later
3. Refine the criteria (and their levels)
Refine the criteria and levels – e.g. based on the literature and available evidence/experience. Get the number of levels and the wording right!
4. Test the criteria / levels
Run a Categorization Survey for the clinicians to rate the patient vignettes on the criteria. Show them how consistent they are (inter-rater reliability). Further refine the criteria (and their levels), and finalise them
5. Average / consensus rating
Resolve disagreements in the ratings of the patients on the criteria using the rating-averaging feature in 1000minds or by reaching consensus in preparation for step #8 later
6. Individual weightings
Run a Preferences Survey for clinicians to experience making trade-offs between hypothetical patients rated on the criteria. Show them similarities and differences in their weights
7. Consensus weightings
Weight the criteria / levels by group consensus – using 1000minds’ Voting option. Alternatively, use the mean of their individual weights (previous step)
8. Validate the tool
Compare the ranking of the patient vignettes from the initial consensus (the pseudo-gold standard) at step #2 with the ranking produced by the tool. Perform other tests of (face) validity and reliability
9. Implement the tool
Implement the tool in information systems or a prioritization web service (e.g. see NZ’s Ministry of Health) and socialise the tool with clinicians who will use it


The process summarised above is expanded on and illustrated in this article:

P Hansen, A Hendry, R Naden, F Ombler & R Stewart (2012), “A new process for creating points systems for prioritizing patients for elective health services”, Clinical Governance: An International Journal 17, 200-9