The process outlined below shows how you can use 1000minds to create a consensus-based tool or system for prioritizing patients for access to health care, e.g. elective surgery. It makes use of three types of 1000minds survey: ‘ranking’, ‘categorization’ and ‘preferences’. A voting option is also available to supplement a preferences survey.
The steps in the process can be adapted to create your own patient prioritization tool, depending on your needs. For example, if you’re in a hurry, steps 1-5 could be dispensed with – assuming you already know you need a new tool and the criteria you want to use – but do you really?
1. Show the need for a new tool
- 1.1 Collate 10-15 patient vignettes (case studies) and run a ranking survey – a “noise audit” – to capture individual clinicians’ intuitive rankings.
- 1.2 Show them how variable they are (e.g. see ‘Noisy’ expert judgments), and that, therefore, a new prioritization tool is needed!
2. Elicit prioritization criteria
- 2.1 As the group of clinicians discusses the ranking survey results – i.e. which patients should be prioritized and why – start specifying the prioritization criteria.
- 2.2 Run the same ranking survey but this time by consensus to get a pseudo-gold standard for step #8 later.
3. Refine the criteria (and their levels)
- 3.1 Refine the criteria and levels – e.g. based on the literature and available evidence/experience.
- 3.2 Get the number of levels and the wording right!
4. Test the criteria / levels
- 4.1 Run a categorization survey for the clinicians to rate the patient vignettes on the criteria.
- 4.2 Show them how consistent they are (inter-rater reliability).
- 4.3 Further refine the criteria (and their levels), and finalise them.
5. Average / consensus rating
Resolve disagreements in the ratings of the patients on the criteria using the rating-averaging feature in 1000minds or by reaching consensus in preparation for step #8 later.
6. Individual judgments
- 6.1 Run a preferences survey for clinicians to experience making trade-offs between hypothetical patients rated on the criteria.
- 6.2 Show them similarities and differences in their weights.
7. Consensus weighting
- 7.1 Weight the criteria / levels by group consensus – using 1000minds’ voting option.
- 7.2 Alternatively, use the mean of their individual weights (previous step).
8. Validate the tool
- 8.1 Compare the ranking of the patient vignettes from the initial consensus (the pseudo-gold standard) at step #2 with the ranking produced by the tool.
- 8.2 Perform other tests of (face) validity and reliability.
9. Implement the tool
- 9.1 Implement the tool in information systems or a prioritization web service (e.g. see NZ’s Ministry of Health).
- 9.2 Socialize the tool with clinicians who will use it.
The process summarized above is illustrated in this article:
P Hansen, A Hendry, R Naden, F Ombler & R Stewart (2012), “A new process for creating points systems for prioritising patients for elective health services”, Clinical Governance: An International Journal 17, 200-9