NLP/LLM Interest Group

This session will feature two exciting talks:

1. Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights by Hyunjae Kim, PhD

Abstract: Retrieval-augmented generation (RAG) is widely adopted to keep medical LLMs current and verifiable, yet its effectiveness remains unclear. We present the first end-to-end, expert annotated evaluation of RAG in medicine, systematically assessing the full pipeline across three stages: evidence retrieval, evidence selection, and response generation. Eighteen medical experts provided 80,502 annotations across 800 model outputs on 200 clinical queries.

Contrary to expectations, conventional RAG often degraded performance—only 22% of retrieved passages were relevant, evidence selection was weak, and factuality dropped up to 6%. However, simple strategies like evidence filtering and query reformulation improved performance by up to 12%.

Our findings challenge current RAG assumptions and highlight the need for deliberate system design in medical AI applications.

2. TopicForest: Embedding-Driven Hierarchical Clustering and Labeling for Biomedical Literature by Chia-Hsuan Chang, PhD

Abstract: The vast and complex landscape of biomedical literature presents significant challenges for organization and interpretation. Current embedding-based topic models like BERTopic are limited to flat, single-granularity clusters, failing to capture the inherently nested, hierarchical structure of scientific subjects. We introduce TopicForest, a novel framework that captures this natural hierarchy by building a "forest of topic trees" directly from text embeddings.

TopicForest delivers high-quality topic clustering comparable to state-of-the-art flat models while providing the essential multi-scale resolution they lack. Through recursive topic labeling, the framework achieves efficient token usage and practical scalability for large corpora. This design provides researchers with an effective tool for exploring and visualizing hierarchical biomedical knowledge landscapes.