Abstract
This exploratory study investigates the predictive validity and inter-rater reliability of automated essay scoring using GPT-5 against certified human examiner ratings in the high-stakes Malaysian University English Test (MUET). Employing a purposive sample of 23 student essays, the writing scripts were evaluated by GPT-5 utilizing a zero-shot prompting pipeline embedded with the official 90-mark MUET analytic rubric (Task Fulfilment, Language, and Organisation). The resulting automated scores were benchmarked against the candidates' official examiner marks. Statistical analyses revealed a severe upward scoring bias, with GPT-5 awarding significantly higher marks than trained human examiners (mean difference = 23.52, t (22) = 6.55, p < .001, d = 1.37), overpredicting outcomes in the vast majority of cases. Furthermore, absolute agreement between the two rating sets was negligible (ICC = -.07), and linear regression confirmed that raw GPT-5 scores failed to reliably predict official MUET outcomes (F (1, 21) = 1.45, p = .243, R2 = .06). These findings demonstrate that, in its default and uncalibrated configuration, GPT-5 applies the shared rubric with extreme leniency and lacks the centralized calibration of certified human raters. The study cautions against deploying general-purpose Large Language Models as predictive or summative tools in high-stakes writing assessments, though it leaves open their potential utility for moderated, formative pedagogical feedback.