269: Comparative Evaluation of Machine Learning and Large Language Models for Headache Disorders

Topic: Clinical Science
Subtopic: Pain category (ICD-11), Chronic primary headache, Developing methodology/technology, Artificial Intelligence/Machine Learning, Digital Health Technologies

Background and Aims: Precise classification of headache disorders is a cornerstone of both clinical management and research. Supervised machine-learning approaches trained on structured clinical data have demonstrated robust diagnostic performance in this domain. In contrast, the diagnostic capabilities of large language models (LLMs), which are increasingly explored in medical applications, remain largely unvalidated for headache classification. To date, no study has directly compared an established machine-learning classifier with an LLM using identical, structured headache questionnaire data and specialist-confirmed diagnoses as the reference standard. This study aimed to address this gap by systematically evaluating and comparing the diagnostic performance of these two modeling approaches.

Methods: We analyzed data from an independent hold-out test cohort of 1,818 patients who completed a standardized 22-item headache questionnaire and subsequently received diagnoses confirmed by headache specialists. A previously validated supervised machine-learning classifier and a general-purpose LLM (GPT-4.1), applied in a zero-shot prompting setting, independently generated five-class diagnostic predictions based on the International Classification of Headache Disorders, 3rd edition (ICHD-3). The diagnostic categories included migraine and/or medication-overuse headache, tension-type headache, trigeminal autonomic cephalalgias, other primary headache disorders, and secondary headaches. Overall agreement and class-wise sensitivity and specificity were assessed and statistically compared using McNemar’s test.

Results: The machine-learning classifier demonstrated significantly superior diagnostic performance compared with the LLM. Agreement with specialist-confirmed diagnoses was higher for the machine-learning model (Cohen’s κ = 0.46) than for the LLM (κ = 0.26). In class-wise analyses, the machine-learning model showed higher sensitivity for migraine and/or medication-overuse headache and for secondary headaches. Conversely, the LLM exhibited greater sensitivity for tension-type headache. Regarding specificity, the machine-learning model outperformed the LLM for tension-type headache, trigeminal autonomic cephalalgias, and other primary headache disorders, while the LLM showed higher specificity only for migraine and/or medication-overuse headache.

Conclusions: In this head-to-head comparison using identical structured questionnaire data, a supervised machine-learning model trained on real-world clinical data outperformed a general-purpose LLM applied in a zero-shot configuration for ICHD-3–based headache classification. These findings suggest that, despite their versatility, current general-purpose LLMs may have limited reliability for diagnostic classification tasks in headache medicine when used without domain-specific training. Caution is therefore warranted when considering LLMs for clinical or research applications involving headache diagnosis, particularly in settings requiring precise adherence to established diagnostic criteria.

References: Katsuki M, Shimazu T, Kikui S, et al. Developing an artificial intelligence-based headache diagnostic model and its utility for non-specialists’ diagnostic accuracy. Cephalalgia 2023; 43: 033310242311569.
Katsuki M, Matsumori Y, Kawamura S, et al. Developing an artificial intelligence-based diagnostic model of headaches from a dataset of clinic patients’ records. Headache 2023; 63: 1097–1108.
Katsuki M, Narita N, Matsumori Y, et al. Preliminary development of a deep learning-based automated primary headache diagnosis model using Japanese natural language processing of medical questionnaire. Surg Neurol Int 2020; 11: 475.

Presenting Author: Masahito Katsuki
Submitting Author: Masahito Katsuki

Senior Author: Tomás Ward

IASP 2026 Global Year theme of Neuropathic Pain: No

• Kieran Moran: PhD - Department of Sport Science and Nutrition and Insight Research Ireland Centre for Data Analytics, Maynooth University, Kildare, Ireland
• Siobhán O’Connor: PhD CAT - School of Health and Human Performance, Dublin City University, Dublin, Ireland
• Tomás Ward: PhD - Insight Research Ireland Centre for Data Analytics, Dublin City University, Dublin, Ireland
• Yutaro Fuse: MD PhD - Department of Artificial Intelligence Medicine, Graduate School of Medicine, Chiba University, Chiba, Japan
• Omid Kohandel Gargari: MD - Headache Department, Iranian Center of Neurological Research, Neuroscience Institute, Tehran University of Medical Sciences
• Marina Romozzi: MD PhD - Dipartimento Universitario di Neuroscienze, Università Cattolica del Sacro Cuore, Rome, Italy.
• Alicia Gonzalez-Martinez: MD PhD - Universidad Autónoma de Madrid, Madrid, Spain
• Miguel Ángel Huerta: PharmD - Department of Pharmacology, University of Granada, Granada, Spain
• Woo-Seok Ha: MD PhD - Department of Neurology, Severance Hospital, Yonsei University College of Medicine, Seoul, Korea.
• Yasuhiko Matsumori: MD PhD - Sendai Headache and Neurology Clinic, Sendai, Japan