SEARCHRCTcardiology, family medicine

RCT·cardiology, family medicine, research methods, public health, clinical trial·PMC10436639

Rating versus ranking in a Delphi survey: a randomized controlled trial

Trials · 2 authors, 3 centres

AI SUMMARY

FIDELITY 100%

POPULATIONPrimary care patients, trained patient partners, and primary care clinicians from seven primary care practices

INTERVENTIONRanking assessment approach for prioritizing organizational items in a Delphi survey

COMPARISONRating assessment approach for prioritizing organizational items in a Delphi survey

This summary was generated by AI from a single paper. It has not been reviewed by a clinician and is not clinical advice. Verify against the source before acting on it.

AGENT 01 · STRUCTURED SUMMARY

TRACE TO SOURCE

“

TL;DR

A randomized controlled trial embedded in a three-round online Delphi survey compared rating versus ranking approaches for item prioritization among primary care patients, patient partners, and clinicians. Both groups identified the same highest priorities from 41 items, but ranking was perceived as significantly more difficult than rating.

Full summary

687 CHARS

Forty-one organizational items across nine themes were elicited in round 1. The main outcome was agreement on top item prioritization between groups after round 3, measured by Krippendorff's alpha. Both groups identified the same highest priorities, but significant discrepancies appeared from the seventh top item onward. Krippendorff's alpha was 0.811 (95% CI 0.669–0.920), indicating moderately strong agreement on common items. Ranking was perceived as significantly more difficult than rating (p < 0.001), and a 9-minute mean difference in third-round completion time favoring rating narrowly missed statistical significance (p = 0.053). Thirty panelists (83%) completed all rounds.

PICO

PPOPULATION

Primary care patients, trained patient partners, and primary care clinicians from seven primary care practices

IINTERVENTION

Ranking assessment approach for prioritizing organizational items in a Delphi survey

OOUTCOME

Agreement on item prioritization (Krippendorff's alpha), questionnaire completion time, and self-reported task difficulty

STUDY TYPE

RCT

SPECIALTY

cardiology

SUMMARISED BY

AI pipeline

FIDELITY CHECK

100% · A

ORIGINAL PAPER

PMC10436639

Rating versus ranking in a Delphi survey: a randomized controlled trial

Trials

READ THE FULL PAPER

ABSTRACT

Background The Delphi technique has steeply grown in popularity in health research as a structured approach to group communication process. Rating and ranking are two different procedures commonly used to quantify participants’ opinions in Delphi surveys. We explored the influence of using a rating or ranking approach on item prioritization (main outcome), questionnaire completion time, and evaluation of task difficulty in a Delphi survey aimed at identifying priorities for the organization of primary cardiovascular care. Methods A randomized controlled parallel group trial was embedded in a three-round online Delphi survey. After an “open” first round, primary care patients, trained patient partners, and primary care clinicians from seven primary care practices were allocated 1:1 to a rating or ranking assessment group for the remainder of the study by stratified permuted block randomization, with strata based on participants’ gender and status. Agreement on item prioritization between the experimental groups was measured by calculating Krippendorff’s alpha reliability coefficient on the aggregate rank order of items in each group after the final round. Self-reported ease or difficulty with the assessment task was measured with the Single Ease Question. Results Thirty-six panelists (13 clinic patients, 7 patient partners, 16 clinicians; 60% females) were randomized to the rating ( n = 18) or ranking ( n = 18) group, with 30 (83%) completing all rounds. Both groups identified the same highest priorities from a set of 41 items, but significant discrepancies were found as early as the seventh top item. There was moderately strong agreement between the priority ordering of top items common to both groups (Krippendorff’s alpha = 0.811, 95% CI = 0.669–0.920). A 9-min mean difference to complete the third-round questionnaire in favor of the rating group failed to achieve statistical significance ( p = 0.053). Ranking was perceived as more difficult ( p < 0.001). Conclusions A rating or ranking procedure led to modestly similar item prioritization in a Delphi survey, but ranking was more difficult. This study should be replicated with a larger number of participants and with variations in the ranking and rating procedures. Trial registration Not applicable. Supplementary Information The online version contains supplementary material available at 10.1186/s13063-023-07442-6.

CITATION EXPORT

Claudio Del Grande, Janusz Kaczorowski. Rating versus ranking in a Delphi survey: a randomized controlled trial. Trials. (2023). https://doi.org/10.1186/s13063-023-07442-6

AUTHORS

2 · 3 CENTRES

Claudio Del Grande, Janusz Kaczorowski

1https://ror.org/0161xgx34grid.14848.310000 0001 2104 2136Health Innovation and Evaluation Hub, University of Montreal Hospital Research Centre, Saint-Antoine Tower, 850 Saint-Denis Street, Montreal, QC H2X 0A9 Canada
2https://ror.org/0161xgx34grid.14848.310000 0001 2104 2136School of Public Health, Université de Montréal, Montreal, QC Canada
3https://ror.org/0161xgx34grid.14848.310000 0001 2104 2136Department of Family Medicine and Emergency Medicine, Université de Montréal, Montreal, QC Canada