NLP/LLM Interest Group

Title: HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation

Abstract: Embedding geometry fundamentally affects retrieval quality, yet dense retrievers for retrieval-augmented generation (RAG) remain confined to Euclidean space. Natural language has hierarchical structure from broad topics to specific entities that Euclidean embeddings fail to preserve, causing semantically distant documents to appear spuriously similar and increasing hallucination risk. We introduce hyperbolic dense retrieval with two model variants in the Lorentz model: HyTE-FH, a fully hyperbolic transformer, and HyTE-H, a hybrid architecture projecting pre-trained Euclidean embeddings into hyperbolic space. To prevent representational collapse during sequence aggregation, we introduce the Outward Einstein Midpoint, a geometry-aware pooling operator that provably preserves hierarchical structure. On MTEB, HyTE-FH outperforms equivalent Euclidean baselines. On RAGBench, HyTE-H achieves up to 29% gains over Euclidean baselines in context and answer relevance using substantially smaller models than current state-of-the-art retrievers. Hyperbolic representations encode document specificity through norm-based separation, with over 20% radial increase from general to specific concepts, a property absent in Euclidean embeddings.

Hiren Madhu a second year PhD student at Yale CS, advised by Professor Smita Krishnaswamy and Professor Rex Ying.

Earlier, he was an ECE pre-doctoral research fellow in Electrical Communication Engineering Department at the Indian Institute of Science, Bengaluru. He worked with Sundeep Prabhakar Chepuri on unsupervised representation learning for simplicial complexes.

He began his research journey at LDRP Institute of Technology and Research (BE in Computer Engineering) under the guidance of Sandip Modha and Thomas Mandl, where he focused on the detection of hate speech in public conversational threads on social media.

His primary research interest is encoding the geometric inductive biases into foundation models. Previously, he worked on simplicial representation learning without labels.