Clinical Natural Language Processing (NLP) Lab

"Learning to Reason with Multimodal Large Language Models" by Jingyi Zhang, PhD, postdoctoral associate in biomedical informatics & data science

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities across a wide range of vision and language tasks. However, developing MLLMs with strong human-like reasoning abilities remains a key challenge, especially in complex, real-world domains such as healthcare. In this talk, I will share my recent exploration of enhancing the reasoning capabilities of MLLMs. I will first introduce our efforts to improve the general reasoning ability of MLLMs through supervised fine-tuning on high-quality multimodal chain-of-thought (CoT) data, which are searched and generated using a novel tree search algorithm across a wide range of application domains. Moving further, I will introduce our study on exploiting online reinforcement learning techniques (e.g., GRPO) that incentivize the model to actively explore alternative reasoning paths, unlocking deeper reasoning capabilities through self-improvement. Finally, I will discuss whether synthetic data is ready to address data scarcity and the high cost of data annotation in MLLMs, with a focus on developing effective data synthesis methods that can automatically generate multimodal training data to improve MLLMs’ ability to solve complex real-world tasks.

Abstract: Medical informatics is the critical bridge connecting clinical research to practice within a Learning Healthcare System (LHS). This presentation highlights the core bioinformatics capabilities and activities of the VA CSP-CERC center in the context of establishing a quality management system using large observational databases and causal inference research to evaluate the effectiveness and safety of treatment strategies, aiding medical professionals in patient care decisions. This involves leveraging high-quality VA data with scalable advanced statistical methods, especially when randomized clinical trials are impractical.

Speaker Bio: Mihaela Aslan, PhD, is a Senior Research Scientist at the Yale School of Medicine and the Director of the Veterans Affairs Cooperative Studies Program Clinical Epidemiology Research Center (CSP-CERC). As a mathematical statistician, she conducts methodological research using high-dimensional data on a wide range of clinical topics, such as causal inference methods leveraging electronic medical records to emulate randomized clinical trials.

CME accredited seminar. Information for claiming credit will be provided at the start of the session.

Abstract: Real-world data (RWD) studies and target trial emulation (TTE) are becoming increasingly important for comparative effectiveness, safety evaluation, and regulatory-grade real-world evidence. Yet in practice, turning a clinical question into a credible TTE still requires substantial manual effort: investigators must define cohorts, compose statistical analysis plans (SAPs), align time zero and follow-up, execute analyses, and document design decisions in a form that is both review-ready and interpretable. Recent agentic systems automate parts of this workflow, but they often remain fragmented and rarely treat debiasing—especially negative-control-based adjustment for residual and potentially unmeasured confounding—as a core design objective.We present MATTE (Multi-agent Target Trial Emulation), an end-to-end agentic TTE framework that links hypothesis interpretation, protocol and SAP generation, cohort construction, effect estimation, negative control outcome (NCO)-guided debiasing, and FDA-aware compliance auditing within a single traceable workflow. By integrating study design, bias assessment, and compliance auditing into one human-reviewable pipeline, MATTE aims to make RWD-based evidence generation more scalable, interpretable, and audit-ready for translational and regulatory use. This talk describes the current MATTE system snapshot and its expected evaluation behavior while full benchmarking is ongoing.

Howard Chan Tsai Hor, PhD, is a Postdoctoral Researcher in the Department of Biostatistics, Epidemiology and Informatics at the Perelman School of Medicine, University of Pennsylvania, working with Dr. Yong Chen. His research focuses on agentic AI for biomedical evidence generation, real-world data analysis, and target trial emulation using large-scale electronic health records. He develops methods that combine causal inference, trustworthy machine learning, negative control outcome calibration, and automated statistical analysis planning to support scalable, interpretable, and audit-ready real-world evidence workflows. Howard has authored more than 16 peer-reviewed papers at leading AI/ML venues, including the International Conference on Learning Representations (ICLR), and in journals including IEEE Transactions on Medical Imaging (TMI) and IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

Abstract: Deep learning models have achieved expert-level performance in diagnosing various ophthalmic conditions using imaging modalities like color fundus photography (CFP). However, these models operate under the assumption that the training and test data are drawn from an identical distribution1. When this assumption is violated by covariate shifts (e.g., varying imaging protocols, camera hardware, field-of-view differences, patient demographics), performance degrades substantially. Unsupervised Domain Adaptation (UDA) addresses this problem by adapting models using unlabeled target data. Existing UDA approaches typically align feature distributions using adversarial learning or entropy-based objectives driven by softmax probabilities. However, softmax normalizes logit magnitudes, which may obscure distributional shifts and cause falsely overconfident predictions. In this study, we propose Class-Conditional Energy Alignment, which adapts source-trained classifiers by matching energy computed directly from unnormalized logits across source and target domains.

Younjoon Chung is a Ph.D. student in Computational Biology and Biomedical Informatics (CBB) at Yale University, advised by Prof. Qingyu Chen and Prof. Lucila Ohno-Machado. His research interests lie in the intersection of machine learning, computer vision and healthcare. Specifically, focusing on developing robust domain adaptation techniques to ensure medical AI models can generalize across diverse clinical environments, including variations in patient populations, imaging hardware, etc.