NLP/LLM Interest Group

A Prompt Library for Efficient Clinical Entity Recognition Using Large Language Models by Yang Ren, PhD

Abstract: Large Language Models (LLMs) hold strong potential for clinical information extraction (IE), but their evaluation is often limited by manually crafted prompts and the need for annotated data. We developed an automated framework that extracts entity-level schema information from published clinical IE studies to construct structured prompts. Using literature covering 44 diseases and over 100 entities, we generated prompts to evaluate multiple LLMs under few-shot and fine-tuned settings. Compared to baselines using generic prompts, models prompted with schema-derived information consistently outperformed across tasks. Our results demonstrate the value of structured prompting for robust and reproducible LLM evaluation in diverse clinical IE applications.