Blended or augmented reality arrives in the classroom

Across Yale campus, medical researchers interact with scientists, engineers, artists, and computer coders. “There is value in bringing diverse perspectives to process,” says Justin Berry, a critic at the School of Art and core faculty member at Yale’s Center for Collaborative Arts and Media (CCAM). At CCAM, researchers are encouraged to seek collaboration outside their fields, and shed some scientific constraints to access new perspectives. Some have the potential to transform medical education and health care.

“We are not a well-oiled, super-fixed, hyper-predictable machine,” Berry says. As principal investigator of CCAM’s Blended Reality grant from Hewlett Packard, he and his colleagues introduce researchers to Virtual Reality and Augmented Reality technology. “The unique opportunity that we offer to the School of Medicine is the ability to do experimental work,” he says, because Blended Reality “is not an outcome-based grant. It’s a process-oriented grant that is about bringing creative approaches to these emerging technologies.”

Previously known as the Digital Media Center for the Arts, CCAM was reimagined as more than a resource center, Berry says, but “a bridge on campus that connects departments, schools or organizations, where they can come together and find ways to connect and share their insights. The hope is that ultimately, the creative process will yield insights that will fold back into that clinical setting.”

Philip Corlett, PhD, Associate Professor of Psychiatry, is one of the researchers who found saw possibility in CCAM’s resources to expand his own work. He was unaware of the center until a performance artist, Brody Condon, approached him. Condon, who had been invited to CCAM to conduct a workshop and is internationally-known, uses cultural materials such video games to create art installations, and chose Corlett’s lab as inspiration for a new video project. “I thought that was really cool,” Corlett says.

In his lab, Corlett focuses on the brain and psychological mechanisms of hallucinations and delusions, two symptoms that occur in serious mental illnesses, particularly schizophrenia. “We’ve been working on ways to model, understand, and experience hallucinations and delusions,” he says. Virtual reality, he thought, “would be an interesting way to explore the psychological, brain, and body mechanisms of hallucination.”

With the guidance of Justin Berry and Bobby Berry, a recent Yale graduate and “Researcher in Residence” at CCAM, Corlett explored ways in which virtual reality (VR), as well as augmented reality (AR) could be used to capture the experience of hallucination, and measure it. “It was interesting to be working with someone who had clearly been thinking about the same sorts of ideas but in a completely different direction,” he says about his interaction with people at CCAM who use VR in other, mainly artistic, contexts. “I think that encapsulates CCAM for me. It’s a space in which people with shared interests from lots of different backgrounds and fields can work together.”

Along with a colleague at the University of Sussex, Corlett is developing content for an AR headset that will recreate the experience of hallucination. Corlett will use CCAM’s 20-camera motion capture studio to record how people experiencing this altered reality react with their bodies, and how they respond physically to objects that are actually in the room, versus objects planted in the mind using AR. Further down the road, a potential therapeutic use for this technology, Corlett says, is to use it to create empathy for people whose have family members who experience hallucinations.

For Michael Schwartz, PhD, Associate Professor of Neuroscience and Director of Medical Studies in Neuroscience, Berry’s artistic perspective opened up new ways to explore his own goal to bring VR and AR into the classroom. An abiding challenge for medical educators is how best to teach medical students about spatial relationships in systems of the body, particularly in the brain. Though he had the idea, “I don’t think we had the plan of how to do it, or even how to engage with capable programmers,” Schwartz recalls. The idea brought him to CCAM, where he and his colleagues formed one of the first teams on the Blended Reality grant, and there he met Justin Berry.

Berry connected Schwartz with students at CCAM who helped his team create a pilot which gives students, using AR, a 3D view of the brain, and how its systems interact and fit together in space.

“When you are focused on your particular area, you are focused and constrained by what you know,” says Schwartz. “The beauty of the people of CCAM is that they were not constrained by our vision. Justin is able to be able to absorb our vision in a way he can advise us.” Further, without CCAM, “we would be doing what several other medical schools are doing, which is hooking up with commercial vendors, and hope that the commercial vendor is, at great expense, able to create the types of things we want to create. And maybe not with quite the flexibility that we can put into ours.”

Schwartz will add the AR brain to the curriculum for the coming year, and test its effectiveness with students. According to Schwartz, designing effective pedagogy means meeting students where they are, and increasingly, that means employing technology that is increasingly common in daily life. “Our learners, by the time they get to us in five years, are going to be so conversant in this, that if we can’t incorporate this into their learning environment, we will be doing them a disservice.”

Back at CCAM, Berry is working to move beyond providing access to VR and AR. He hopes to teach professionals in a variety of fields to become proficient in using it themselves. Through a project he is working on with Bobby Berry called Verb Collective, he is spreading the word about the power of blended reality by creating a toolkit for using VR and AR. Doing so, he says, “can put the power of working with it in the hands of the people that really know what they are doing” and use it, unconstrained, to further knowledge in their own fields.