Skip to Main Content

Can ChatGPT Replace Peer Review? MD Students Explore the Big Data Issue

March 19, 2024
by Akio Tamura-Ho

From ChatGPT to groundbreaking advancements in genomics, researchers and clinicians today are experimenting with the ever-increasing volumes of data at their fingertips. As Conrad Safranek and Ryland Mortlock put it: “The trajectory of big data promises a transformative reshaping of medicine’s landscape.”

Safranek is an MD student at Yale School of Medicine and co-deputy editor of the September issue of the Yale Journal of Biology and Medicine (YJBM). Along with Mortlock, a Yale MD-PhD student in genetics, he spent several months putting together the issue, which is themed around big data.

Though he isn't enrolled in one of the degree programs affiliated with the Section of Biomedical Informatics and Data Science, Safranek is a member of the AI in Medicine Student Interest Group and can often be found attending the Section's lunchtime seminars as an active and enthusiastic participant. “I’ve always been into student-run initiatives and wanted to continue that during medical school,” Safranek says. The Big Data issue also intersected with his research interests: his undergraduate thesis at Stanford examined how machine learning algorithms could be used with big clinical data in the field of anesthesiology. “I’ve been interested in this stuff for a while.”

When the opportunity came up to select themed issues for the journal, many YJBM team members were excited about big data. “And that was before ChatGPT dropped,” Safranek adds. “It blew things up, at least in my world, for the potential of AI in medicine. So I got even more excited as we went through the issue.”

The journal issue includes four original contributions in addition to eight case reports, reviews and perspective pieces. Now that the issue has finally been published, Safranek says he’s learned a lot.

“It gave me a lot of insight into the academic research review process. I’ve been on the side of submitting manuscripts and it’s not always the most intuitive—like a black box. It was exciting to be on the other side of that and get a real window into how the process runs,” Safranek says. “It’s very insightful seeing how an article goes from submission to publication—including how many invited reviewers you have to have before someone accepts.”

Recent discussions in the publishing world have examined how overworked and underpaid reviewers might be resorting to ChatGPT, further complicating the strained economy of traditional peer review. When I mention this, Safranek lights up. “One of the articles published in the issue addressed exactly that,” Safranek says. “It was a case study where they compared a review that ChatGPT provided against the reviews from three invited reviewers.”

The case study in question, written by Som Biswas, Dushyant Dobaria, and Harris L. Cohen, argues that ChatGPT’s critical analyses aligned with those of human reviewers, demonstrating a proof-of-concept for more streamlined peer review. Biswas recently gained widespread media attention for using ChatGPT to write at least 16 papers over four months, several of which were published in peer-reviewed journals. It’s the kind of research practice that is bold or terrifying, depending on who you ask. It also highlights the urgent systemic problems in academic publishing that Safranek and his peers hope to address. “The bar is low,” Safranek admits, noting the nationwide shortage of experts who are willing and able to review a constant stream of manuscripts. That said, he also isn’t impressed with ChatGPT’s efforts: “I wasn’t blown away."

Biswas, Dobaria and Cohen are careful to note that biases in models like ChatGPT, for example, should be addressed, along with transparency and explainability. In fact, bias and health equity remains front-of-mind for several other scientists who use big data approaches in their research. One article in the Big Data issue used Electronic Health Records data to uncover racial disparities in critical care settings, while another found that patients in minority groups were less likely to present for care for substance-related diagnoses, suggesting possible implications for stigma and health disparities.

The Future of Big Biomedical Data

Running the editorial process for a journal like YJBM came with a surprising amount of responsibility. Safranek mentions the importance of mentors in supporting student efforts. “We do have faculty support and a copyeditor, which makes a world of difference,” he says. Safranek is a member of Dr. Andrew Taylor’s lab, and also meets regularly with his research mentor, David Chartash, PhD, who is a lecturer in Biomedical Informatics and Data Science. “[David] is incredible and really invested in mentoring my research as well as my development as a whole. He’s gotten me excited about going to the Biomedical Informatics lunch talks.” As if on cue, Chartash pops into our interview Zoom, ready for his weekly meeting with Safranek.

Co-editing the YJBM has only made Safranek more excited about the opportunities that big data offers. “The biggest learning point for me came to fruition as I wrote the introduction,” he says. “This is such a diverse field. I’ve been honed into clinical data, machine learning and now, large language models. But genomics is a whole other side of it. Insurance claims data is another angle. Automating imaging analysis is huge. I think the future of medicine is integrating those pieces, and they still have some ways to come together.”

The Yale Journal of Biology and Medicine was founded in 1928.

Submitted by Akio Tamura-Ho on October 24, 2023