WCWL Rheumatology Priority Referral Score reliability & validity testing

ACR/ARHP Annual Scientific Meeting, Philadelphia, 2009

Avril Fitzgerald (1), Barbara Conner Spady (1), Carolyn DeCoster (1), Ray Naden (2), Gillian A. Hawker (3), Thomas Noseworthy (1), and the WCWL Investigators (4)

(1) University of Calgary, Calgary, AB, (2) New Zealand Ministry of Health, Auckland, New Zealand, (3) Women’s College Hospital, Toronto, ON, (4) Calgary, AB

Purpose: Rheumatology consultation demand exceeds supply of rheumatologists in many countries. Western Canada Waiting List (WCWL) has developed a diagnosis independent, priority-referral score (PRS) for rating relative urgency for referral to rheumatologists by primary care providers (PCPs). This paper reports the reliability and validity of the PRS.

Method: To develop WCWL PRS, a clinical panel of 5 rheumatologists and 5 PCPs, using anonymous case scenarios, engaged in a deliberative process leading to development of criteria and levels. Relative weights were generated using discrete choice experiment software (1000 Minds®). Reliability and validity of PRS were tested independently with another 14 rheumatologists and 10 PCPs. These physicians clinically ranked, and then PRS scored, 16 case scenarios representative of the range of case-mix and urgency of referrals from PCPs to rheumatologists. Participants also rated relative urgency using visual analogue scale (VAS) and maximum acceptable waiting time (MAWT) for each case. Retesting was performed 6-12 weeks later, using altered case identifiers.

Results: The inter- and intra-rater reliability coefficients (ICC) for PRS and clinical ranking are shown for rheumatologists, and inter-rater reliability coefficients for PCPs (retesting in progress).

Inter-rater Correlation Coefficients (ICC) for Urgency Scores

Rheumatologists PCPs
Method Time 1 Time 2 Time 1
PRS 0.80 0.80 0.81
Clinical Rank 0.81 0.78 0.60
VAS 0.70 0.75 0.69
MAWT 0.46 0.43 0.38

Intra-rater Correlation Coefficients (ICC) for Urgency Scores

Method Rheumatologists
Average Range
PRS 0.83 0.57-0.97
Rank 0.94 0.74-0.97
VAS 0.82 0.74-0.95
MAWT 0.85 0.65-0.96

Three emergent cases were correctly identified by 14, 12 and 10 rheumatolgists initially, and 14, 13 and 13 on retesting. These emergent cases were correctly identified by 10, 8 and 7 of the 10 PCPs.

Conclusion: Average correlation between rheumatologists’ clinical rankings and PRS-derived rankings is moderately strong at 0.71. Inter-rater and intra-rater ICCs for clinical ranking and PRS for rheumatologists are strong and consistent over six weeks. PCPs demonstrate higher inter-rater reliability with PRS than clinical ranking. Using PRS, PCPs have similarly high inter-rater reliability as rheumatologists. Emergent cases are identified by rheumatologists correctly, or score high on urgency. These results show acceptable performance of the PRS, albeit further testing of specific case mix groups is required. Implementation and pilot testing is in progress.

Keywords: access to care and rheumatologic practice

Disclosure: A. Fitzgerald, None; B. Conner Spady, None; C. DeCoster, 3 ; R. Naden, None; G. A. Hawker, None; T. Noseworthy, None.