Abstract

This work explores the use of a Large Language Models (LLM) for the automatic identification of laterality spans in mammography reports. Specifically, we assess the performance of three GPT models accessible via OpenAI's API in categorizing sentences according to their entailment towards either the right or left breast. Our methodology involves employing a framework that takes natural language reports authored by radiologists as input, subsequently generating labels which are refined through straightforward rule-based post-processing methods. We evaluate three prompt types (zero-shot, one-shot and few-shot with 3 examples) across the GPT models. Our findings demonstrate that GPT-4o achieves the highest F1 score of 0.77 in the few-shot scenario. Furthermore, we observe notable performance differences and error rates between GPT-3.5-turbo and GPT-4 models, highlighting the superiority of the latter models. Additionally, we find that incorporating one or more task examples in the prompt reduces errors, enhances adherence to instructions and boosts overall performance of the models.