Agreement Between Large Language Models and Faculty Assessment in the Evaluation of Clinical Reasoning Case Examinations in Undergraduate Physiotherapy Education: A Comparative Reliability Study
Agreement Between Large Language Models and Faculty Assessment in the Evaluation of Clinical Reasoning Case Examinations in Undergraduate Physiotherapy Education: A Comparative Reliability Study
This study evaluates whether large language models (LLMs) can reliably assess written clinical-reasoning case examinations completed by undergraduate physiotherapy students, compared with faculty assessment. In the course "Specific Methods in Physiotherapy" (third year of the Physiotherapy Degree), students solve complex clinical cases that require clinical reasoning, technical knowledge, and therapeutic decision-making. These cases are traditionally graded by faculty, a time-consuming process that may show inter-rater variability.
A set of de-identified student case examinations will be assessed using the rubric currently applied in the course, which covers clarity and structure of clinical reasoning, integration of the biopsychosocial model (ICF and APTA frameworks), accuracy in identifying pain mechanisms, coherence between diagnosis, hypotheses, and treatment, originality and depth of analysis, and professional writing. Each examination will be scored independently by three LLMs (for example, Claude, ChatGPT, and Gemini), each receiving an identical standardized prompt that embeds the same rubric, and by faculty serving as the reference standard.
To avoid overloading faculty, full double human grading may not be feasible; the human reference will therefore consist of expert faculty grading by one independent rater or, when resources allow, two independent raters. In contrast, paired assessment is fully implemented across the AI models: each examination is scored by several LLMs, and each model is queried in duplicate, allowing the study to estimate agreement between models and the test-retest stability of each model.
The primary aim is to quantify agreement between LLM-generated scores and the faculty reference score. Secondary aims include agreement among the LLMs, test-retest reliability of each model, criterion-level agreement, the quality and usefulness of the qualitative feedback generated, the time and cost associated with each approach, and students' perceptions of the usefulness of human versus AI feedback.
The findings will clarify the strengths and limitations of LLMs as supportive tools for formative assessment in health-professions education and will inform criteria for their responsible and effective use. No LLM output will affect students' official grades, which remain the sole responsibility of faculty.
BACKGROUND AND RATIONALE The assessment of clinical case examinations in physiotherapy requires the appraisal of multiple dimensions, including clinical reasoning, selection of appropriate techniques, treatment dosage, ethical considerations, and patient communication. This grading process demands substantial faculty time and may be affected by evaluator fatigue and inter-rater variability. Large language models (LLMs) have shown notable capabilities in text comprehension, reasoning, and the generation of structured feedback, and preliminary evidence suggests they may provide consistent evaluations in medical education contexts. However, questions remain regarding their reliability, potential bias, and their ability to capture the complexity of clinical reasoning. There is currently limited empirical evidence on whether LLMs can complement human grading while maintaining quality standards and offering immediate formative feedback. This study addresses that gap through a systematic comparison between human assessment (reference standard) and LLM-assisted assessment using identical materials and criteria.
OBJECTIVES Primary objective: To quantify the agreement between the scores generated by LLMs and the faculty reference score in the assessment of physiotherapy clinical-reasoning case examinations.
Secondary objectives:
STUDY DESIGN Cross-sectional inter-rater agreement and reliability study with repeated measures, in which the same set of de-identified student clinical case examinations is assessed independently by human and artificial-intelligence raters using a shared, predefined rubric. The study is observational and educational in nature; it does not modify the teaching or the official assessment received by students.
SETTING, PARTICIPANTS, AND MATERIALS The study is conducted within the course "Specific Methods in Physiotherapy" (third year, Physiotherapy Degree), in which clinical cases are used as a central learning and assessment tool. The units of analysis are the written case examinations produced by enrolled students (approximately 60 to 80 examinations). All examinations are anonymized prior to assessment so that no rater can identify the author. The grading instrument is the rubric already used in the course, comprising the following criteria: (1) clarity and structure of clinical reasoning; (2) integration of the biopsychosocial model (ICF and APTA frameworks); (3) accuracy in identifying pain mechanisms; (4) coherence between diagnosis, hypotheses, and treatment; (5) originality and depth of analysis; and (6) appropriate professional writing. Each criterion yields a partial score, and the criteria sum to a global score.
RATERS AND ASSESSMENT PROCEDURE Human assessment (reference standard): Each examination is scored by faculty with expertise in the course, applying the established rubric. The protocol is designed to accommodate two scenarios depending on faculty workload. In the preferred scenario, two faculty members score each examination independently (paired human correction), enabling estimation of human-human reliability and the use of the mean or consensus score as the reference. In the contingency scenario, to avoid overloading faculty, a single expert faculty rating (or, alternatively, the official course grade already assigned) is used as the reference standard; in that case, human-human reliability is not estimated within this study and is acknowledged as a limitation.
Artificial-intelligence assessment (paired AI correction): The same anonymized examinations are scored independently by three LLMs (for example, Claude, ChatGPT, and Gemini, in the versions available during the data-collection period). Each model receives an identical standardized prompt that embeds the same rubric and requests, for every examination, a partial score per criterion, a global score, and structured qualitative feedback. To assess intra-model (test-retest) reliability, each model is queried in duplicate in independent sessions under fixed generation parameters. This design implements "peer" correction across models: outputs are cross-compared between models (inter-model agreement) and against repeated runs of the same model (intra-model stability), mirroring the logic of paired review while remaining feasible without additional faculty burden. Grading order is randomized, and raters (human and AI) are blinded to one another's scores. The time required for each evaluation and the operating cost of each LLM are recorded.
OUTCOME MEASURES Primary outcome: agreement between each LLM's global score and the faculty reference global score.
Secondary outcomes: inter-model agreement among LLMs; intra-model test-retest reliability; criterion-level agreement; feedback quality (number of specific and actionable comments, coverage of case dimensions, and rated formative usefulness); efficiency (mean evaluation time and cost per evaluation); and student-perceived usefulness of human versus AI feedback (Likert scales).
STATISTICAL ANALYSIS General approach: Analyses will be performed in R. Continuous variables will be summarized as mean and standard deviation or median and interquartile range, according to distribution (assessed with the Shapiro-Wilk test and graphical inspection); categorical variables as absolute and relative frequencies. All tests will be two-sided with an alpha of 0.05, and 95% confidence intervals (CI) will be reported for all reliability and agreement estimates.
Primary analysis (LLM vs faculty agreement): For the global score (continuous), agreement between each LLM and the faculty reference will be quantified with the intraclass correlation coefficient (ICC), two-way random-effects model, absolute-agreement definition, single-rater and average-rater forms [ICC(2,1) and ICC(2,k)], following the conventions of McGraw and Wong and the reporting guidance of Koo and Li. ICC values will be interpreted as poor (<0.50), moderate (0.50-0.75), good (0.75-0.90), and excellent (>0.90). Systematic bias will be examined with Bland-Altman analysis, reporting the mean difference and 95% limits of agreement, with inspection for proportional bias. For criterion-level (ordinal) scores, agreement will be quantified with Cohen's weighted kappa using quadratic weights. Because kappa is sensitive to prevalence and marginal imbalance (the "kappa paradox"), Gwet's AC1/AC2 and the prevalence-adjusted bias-adjusted kappa (PABAK) will be reported as robust complements. Categorical agreement coefficients will be interpreted using the Landis and Koch benchmarks. Pre-specified targets, consistent with the project objectives, are ICC >= 0.75 for the global score and weighted kappa >= 0.60 at the criterion level.
Inter-model reliability: Agreement among the three LLMs on the global score will be estimated with a two-way random-effects ICC for multiple raters; for criterion-level categorical scores, Fleiss' kappa and Gwet's AC will be used. Pairwise model comparisons will also be reported.
Intra-model (test-retest) reliability: For each model, agreement between duplicate runs will be quantified with the ICC and the percentage of exact agreement for categorical criteria, complemented by Gwet's AC. The standard error of measurement (SEM) and the minimal detectable change (MDC95) will be derived from the ICC to express measurement precision in score units.
Feedback quality (qualitative and quantitative): The qualitative feedback will be analyzed through structured content analysis with predefined categories aligned to the rubric dimensions. The number of specific, actionable comments and the coverage of case dimensions (reasoning, technique, communication, ethics) will be counted for each rater type and compared using chi-square or Fisher's exact tests for proportions and Kruskal-Wallis tests for counts, with appropriate post-hoc comparisons. Inter-coder reliability for the content-analysis coding will itself be reported (Cohen's or Gwet's coefficient) to ensure the trustworthiness of the categorization.
Efficiency: Mean evaluation time will be compared between humans and each LLM using paired t-tests or Wilcoxon signed-rank tests depending on distribution. Cost per evaluation (faculty time valued at standard institutional rates and LLM usage fees) will be summarized descriptively.
Student perception: Likert responses will be summarized as medians and IQRs and as the proportion of favorable responses. Differences in perceived usefulness between human and AI feedback will be tested with the Wilcoxon signed-rank test, and the Friedman test will be used when more than two feedback sources are compared, with appropriate post-hoc analysis.
Handling of the reference-standard contingency: If two independent faculty ratings are obtained, human-human reliability will be reported and the mean or consensus score used as the reference for all LLM comparisons; if only a single faculty rating (or the official course grade) is available, that single value serves as the reference, the absence of an in-study human-human reliability estimate is reported as a limitation, and, where available, historical course data on faculty agreement are cited as external context. The AI-centered analyses (inter-model and intra-model reliability) are unaffected by this contingency and are performed in all cases.
Sensitivity analyses and missing data: Sensitivity analyses will explore the influence of score distribution and of any examinations with extreme scores. The pattern of any missing or non-evaluable outputs (for example, an LLM failing to return a parsable score) will be described, and complete-case analysis will be the primary approach, with the proportion of missing data reported.
SAMPLE SIZE JUSTIFICATION This is a reliability/agreement study, so the sample-size rationale is based on the precision of the reliability estimates rather than on hypothesis testing of a between-group difference. For an expected ICC of approximately 0.75 estimated with several raters and a target 95% CI half-width of about 0.10-0.12, established approximations for ICC precision (Bonett) indicate that on the order of 40-60 examinations are required; for kappa-based criterion agreement, the approximations of Donner and Rotnitzky yield comparable figures. The approximately 60-80 available examinations therefore provide adequate precision for the planned estimates. These figures are indicative and will be refined once the final number of evaluable examinations and raters is confirmed.
ETHICS AND DATA PROTECTION The study will be submitted to the Research Ethics Committee of Universidad Rey Juan Carlos for review and approval. Informed consent will be requested from participating students. All examinations are anonymized before assessment, data are processed in accordance with the EU General Data Protection Regulation (GDPR) and applicable national law, and participation does not affect students' final grades, which are determined exclusively by faculty. Students are informed that their de-identified case examinations will be assessed by both faculty and LLMs for educational-research purposes.
Inclusion Criteria:
Exclusion Criteria:
alfredo.lerin@lasallecampus.es+34620187457