12-14 MEDG: ChatGPT Has Arrived: Where it Belongs in Medical Education
December 14, 2023ID11092
To CiteDCA Citation Guide
- 00:04So good to see you everybody.
- 00:06Welcome to our ChatGPT has arrived and
- 00:11I really have the honor of introducing 2
- 00:14faculty and two students who have really
- 00:17worked hard for the presentation today.
- 00:20Conrad Sefranek is an MD student
- 00:22interested in pursuing internal
- 00:24medicine and research regarding clinical
- 00:27applications of artificial intelligence.
- 00:29He graduated with honors from Stanford
- 00:32with ABS in Computational Biology
- 00:34and a minor in Management Science
- 00:36and Engineering at Stanford's.
- 00:39His research focused on applying
- 00:42machine learning algorithms to
- 00:44clinical decision support systems.
- 00:46And now at Yale,
- 00:48his research has focused on
- 00:50how large language models,
- 00:52such as what he's going to talk about today,
- 00:54ChatGPT can be applied both in the clinical
- 00:57practice and to augment medical education.
- 01:00And Elizabeth Sidiman Aristof is an MDPHD
- 01:04student intending to study how early life
- 01:08stress exposure alters neurodevelopment.
- 01:11She graduated summa *** laude from Princeton.
- 01:14She had an AB in Spanish and Portuguese
- 01:17and a certificate in neuroscience.
- 01:20She worked during her Princeton years,
- 01:23studying early life stress.
- 01:24And then she went to Boston Children's
- 01:27and conducted research on how
- 01:30children's environments influence
- 01:32their mental health outcomes.
- 01:34I'm so thrilled that they're
- 01:36both with us today.
- 01:37On this presentation,
- 01:38David Cartash is a lecturer in the
- 01:41section of Biomedical Informatics
- 01:43and Data Science at Yale University.
- 01:47His undergrad degree in Engineering
- 01:49Science in Electrical Engineering from
- 01:51the University of Western Ontario was
- 01:54specializing in biomedical signals and
- 01:57systems at the University of Toronto.
- 01:59He went on to earn a master's of Health
- 02:02Science in Clinical Engineering,
- 02:04focusing on the evaluation of the
- 02:07prediction of hospital decompensation events.
- 02:10He then obtained his PhD in Medical
- 02:15Informatics and Complex Systems
- 02:18Science from Indiana University,
- 02:21and he's been here working with
- 02:24us on many of these issues,
- 02:26Talan Wejesakara,
- 02:28an assistant professor on the Clinician,
- 02:31Educator, Scholar Track.
- 02:32I have to brag a little.
- 02:34He completed his master's in Health Science,
- 02:38medical education.
- 02:39He was part of our first cohort.
- 02:42He did his undergrad at Duke,
- 02:44and then he went to University of
- 02:47Rochester for his MD.
- 02:48He then came to Yale for his
- 02:51internship and residency and did a
- 02:54fellowship in the general internal
- 02:56medicine that's under Donna Windish.
- 02:58And I see that she's with us today.
- 03:02He's had a number of leadership positions.
- 03:06He's been the Course Director of
- 03:08Clinical Reasoning since 2017 and he's
- 03:11also an associate in our Center for
- 03:15Medical Education providing educator
- 03:18development on clinical reasoning.
- 03:21He's also an advisor to the clinical,
- 03:24the clinician educator distinction path for
- 03:28residents in the Department of Medicine
- 03:31and he is the Director of Performance
- 03:34Improvement for our medical students.
- 03:36And they've all put a lot of time
- 03:38and I'm really looking forward
- 03:40to hearing this presentation.
- 03:42We have so much to learn.
- 03:43So thank you very much for
- 03:45putting this together.
- 03:46I'll pass it over to you, thalan,
- 03:50and I will kick it off to
- 03:53Conrad and Anna Elizabeth.
- 03:56Hello, everybody.
- 03:57Thanks so much for coming.
- 03:59I'm sure many, if not all of
- 04:01you have heard of Chatcha BT,
- 04:03probably with a range of excitement to
- 04:06skepticism and maybe some apprehension.
- 04:08Today, we'd like to share with you our
- 04:11perspective as medical students on how
- 04:13Chatcha BT could and we argue should,
- 04:15be used to augment medical education.
- 04:19This is the accreditation
- 04:21and disclosure slide.
- 04:22I think there's information in the chat,
- 04:24but make sure you don't forget that overall,
- 04:27in this presentation we'll
- 04:29cover some context and recent
- 04:30research on large language models,
- 04:32followed by an exploration of its
- 04:34integration into medical education.
- 04:36And the second-half will focus on some
- 04:38ways that faculty may use these models
- 04:40to augment their teaching practices.
- 04:44So this is a very basic neural network.
- 04:47The idea is that there's an
- 04:49input layer and an output layer,
- 04:51and then a hidden layer that
- 04:53does computation that gets a.
- 04:57The idea is that you can strengthen or
- 04:59weaken the connections between particular
- 05:01nodes based on prior experience,
- 05:03and this would be represented by
- 05:05changing the thickness of the arrows.
- 05:07And what we're doing then is altering
- 05:10the probability of a particular
- 05:12output being generated given an input.
- 05:15A deep neural network is a much more
- 05:18complicated version of this with
- 05:20more layers and non linear layers,
- 05:22and the computation of this goes way
- 05:24beyond the scope of our presentation.
- 05:26But it's important to understand
- 05:28the basics because ChatGPT and
- 05:30large language models broadly are
- 05:33a type of deeper neural network.
- 05:36So we're creating a probability
- 05:38distribution to understand not
- 05:40only the definitions of words,
- 05:42but also their usage in relation
- 05:45to other words,
- 05:46and this allows for the
- 05:48abstraction of information.
- 05:52O just to clarify, Chat GT stands for
- 05:55Chat generative retrained Transformer.
- 06:00So ChatGPT is by no means in isolation.
- 06:03I think it sparked a lot of people's
- 06:05interest in large language models,
- 06:07but it wasn't the 1st and
- 06:08it's definitely not the last.
- 06:09There are a ton of these new
- 06:11models coming out right now,
- 06:12and they're rapidly expanding
- 06:14in terms of the size of the
- 06:16models and and their performance.
- 06:17This graph is already out of date,
- 06:21so it's important to note
- 06:23ChatGPT as I was explaining,
- 06:25was trained to produce plausible
- 06:28text using the probability it
- 06:30is not actually trained on what
- 06:32is true versus what is not true,
- 06:35and that means that it is capable of
- 06:38generating text that is fundamentally wrong.
- 06:41These have been termed hallucinations,
- 06:44and the reason that this is possible
- 06:46is because we're asking it to do next
- 06:49word prediction to give us a plausible
- 06:51string of text to answer a question,
- 06:54but it's not actually doing the cognitive
- 06:57task of answering that question.
- 06:59Additionally, ChatGPT does not have
- 07:01access to data behind paywalls,
- 07:04so it was trained on a huge volume
- 07:06of information on the Internet,
- 07:08but only that was freely available.
- 07:10The free version of Chat GPTGPT 3.5
- 07:14was not trained on data past 2021,
- 07:17and ChatGPT 4 was not trained on
- 07:21data past 2023, I believe April.
- 07:24So it's not browsing the Internet
- 07:26in real time or in our case reading
- 07:29scientific or medical literature that is
- 07:32coming out as it immediately comes out.
- 07:36So for a little bit more context there
- 07:38was a recent research study actually
- 07:41conducted at Yale looking at how ChatGPT
- 07:44this was the original base 3.5 model could
- 07:47perform on the USMLE exam and to sum it
- 07:50all up in one phrase it passed step one.
- 07:54That original base model which spiked
- 07:56sparked a lot of excitement and I think the
- 07:59medical community and the more recent update.
- 08:01So that was equivalent to you could argue
- 08:03about an M3 able to pass a step one exam.
- 08:06But these more recent GPT 4 which
- 08:09is open AIS latest model has been
- 08:12shown to achieve like greater than
- 08:14an 85% on all three-step exams.
- 08:17So this is performing at a pretty
- 08:20high percentile on these tasks.
- 08:26So to be clear, in this presentation
- 08:28we are not proposing using ChatGPT
- 08:31as the definitive source of truth,
- 08:34but rather as a tool.
- 08:35So think of it like a calculator.
- 08:37For doctors like a Wikipedia,
- 08:40you still need to understand the
- 08:43underlying computation of what
- 08:45you're entering into the calculator,
- 08:47but you don't necessarily
- 08:48need to do out all the math.
- 08:50So we equate using ChatGPT to having
- 08:52a conversation with a friend.
- 08:56So then this next part of the presentation,
- 08:58we're going to highlight a few use cases
- 09:01that have emerged over the past year of
- 09:04our using ChatGPT often on a daily basis.
- 09:07So this first use case is when
- 09:09reviewing multiple choice exams.
- 09:11So this is an example of a multiple
- 09:14choice question that we had on a
- 09:16midterm and I missed the question,
- 09:18I wasn't totally sure why and I couldn't
- 09:20really explain the correct answer.
- 09:22And unfortunately faculty have so much
- 09:25on their play and sometimes you want
- 09:28more of a rationale than is provided.
- 09:29So this was plugging in that
- 09:32whole question into Chat CPT.
- 09:33We were able to explore its response to
- 09:37try to better understand the question and
- 09:39you also get this nice verification of
- 09:41if Chat BT chooses the correct answer,
- 09:44that's at least some protection against you
- 09:47knowing its response is an hallucination.
- 09:51Another use case that Conrad and I
- 09:54have been discussing is practice cases.
- 09:56So we have workshops and preclinical
- 09:58where we go through practice
- 10:00cases with faculty member,
- 10:02but there's so many more important
- 10:04topics that we simply don't have
- 10:06time to cover in those spaces.
- 10:08And so we can input into ChatGPT
- 10:12and ask for workshop cases that we
- 10:14can then work through on our own.
- 10:16The advantage of doing this with
- 10:18ChatGPT over using a clinical cases
- 10:21textbook or what's available on like
- 10:24the New England Journal website is
- 10:25that we can actually interact with it.
- 10:27So we can ask it to rephrase
- 10:29things that don't make sense to us.
- 10:31We can ask for additional details.
- 10:34But perhaps most importantly,
- 10:36the cases that it's generating
- 10:37are not static.
- 10:38So we could ask it to change one finding.
- 10:41So for example,
- 10:42change a physical exam finding or a
- 10:44lab finding and then redo the case.
- 10:46And that really helps us hone our
- 10:50clinical reasoning skills to see
- 10:52how one element can change the
- 10:54whole differential or not.
- 10:56So it's allowing us to practice
- 10:58applying the pathophysiological
- 10:59frameworks that we are learning.
- 11:03We're saving a third use case
- 11:04for the end of our presentation,
- 11:06so we'll come back to it.
- 11:07But this is a more recent update
- 11:09from Open AI, the development of
- 11:12these personalized GPT models.
- 11:14And with these models,
- 11:15you can custom build a version
- 11:17of Chat PT by providing relevant
- 11:19contextual information.
- 11:20For example,
- 11:21I'm a second year medical student
- 11:22and I'm interested in learning
- 11:24content at the level of death needed
- 11:26for my upcoming USMLE STEP exam.
- 11:28Moreover, for my personal learning style,
- 11:31I like information presented concisely,
- 11:34using familiar medical abbreviations
- 11:35and bulleted lists when applicable.
- 11:39So Conrad is significantly better
- 11:41at memorizing patterns than I am,
- 11:43and I really need a framework
- 11:45for thinking that is Physiology
- 11:47driven so that I can work my way
- 11:50up from the basic principles.
- 11:52So when I was creating my
- 11:55Med school GPT tutor,
- 11:56I specifically told it that I wanted
- 11:59explanations to be longer centered
- 12:01on pathophysiological frameworks and
- 12:03building from the underlying mechanisms
- 12:06up through the presentation of disease.
- 12:09So this is really great because Conrad
- 12:11and I are using the same tool ChatGPT 4,
- 12:14but we are able to tailor it based to
- 12:16or based on what we specifically need
- 12:18for our own individual learning styles.
- 12:23So here are a few other things
- 12:25that Conrad and I have come up
- 12:26with and that are suggested in the
- 12:28literature as to ways that chat.
- 12:30GPT can be used in medical education,
- 12:33so it could be used to reduce
- 12:35the overwhelming landscape of
- 12:37available resources for students.
- 12:38It could help create a study plan,
- 12:41and it could also be a simulated patient.
- 12:43And we could practice doing a
- 12:44history taking, typing it out.
- 12:47So there's lots of potential
- 12:50taking a step back from the day-to-day
- 12:52trench of the medical school,
- 12:53whether we like it or not, large
- 12:55language models are coming to healthcare.
- 12:58Here is a smattering of a few of
- 13:00the recent papers coming out about
- 13:02potential uses of Chat Chipetia
- 13:04Medicine and since making these
- 13:05slides there's been a lot more.
- 13:07Moreover, Microsoft and Epic announced
- 13:10and expanded Bring Chat BT into the
- 13:13electronic health records at Yale.
- 13:15Whether we like it or not,
- 13:16we need to be prepared here.
- 13:19I'll also highlight the in as
- 13:21part of this expanded partnership,
- 13:24there's actually a pilot program
- 13:26that's happening at Y and HH Yell
- 13:28is one of 10 sites selected for
- 13:31this pilot program where we can
- 13:34actually have for the Mychart
- 13:36inbox messaging from physicians.
- 13:38They'll there can be so free.
- 13:40In the specific example,
- 13:41this patient asked about the poor air
- 13:44quality and whether it is impacting
- 13:46or should impact her exercise
- 13:47routine and whether she can keep
- 13:49going on walks and chat to BT GB T4,
- 13:53in this case drafts,
- 13:55this auto reply that the physician
- 13:58absolutely needs to read,
- 13:59review,
- 14:00likely edit before and of course you
- 14:02have the option to reject that draft.
- 14:05But this is coming to Epic very soon.
- 14:08So I think that's part of the impetus for
- 14:10us really learning about these models.
- 14:18So there are many proposed ways in the
- 14:21literature at large about how ChatGPT
- 14:23can be used in medical practice.
- 14:26So Conrad and I have talked about in
- 14:29particular that ChatGPT can be applied to
- 14:31reduce the administrative load of physicians.
- 14:33So for example, it could be
- 14:36used for insurance filings or
- 14:38generating discharge instructions.
- 14:39It can also be used for reducing errors.
- 14:42So for example,
- 14:43detecting potential drug drug
- 14:45interactions and it might actually be
- 14:47able to reduce the burden on patients.
- 14:49So for example,
- 14:50there may be a way to fall or do a
- 14:53follow up appointment with the chat bot
- 14:56without actually scheduling a whole
- 14:58visit for something that's very routine.
- 15:00But ultimately,
- 15:01the goal is to use automation
- 15:04to reduce physician burnout
- 15:05and to improve patient care.
- 15:10So as Conrad said, we really need
- 15:12to be prepared for this inevitable
- 15:14integration of AI into healthcare.
- 15:17And I think that part of that is
- 15:19evaluating very critically how we can
- 15:22use large language models responsibly,
- 15:24but also what the limits are of
- 15:28large language models in AI at large.
- 15:30So we think questions about hallucinations,
- 15:33bias chat, GP, TS evaluation of its
- 15:36own uncertainty and legal and ethical
- 15:38issues would be great to address in
- 15:41spaces like professional responsibility,
- 15:43which is a preclinical course that
- 15:45goes through some of the major ethical
- 15:47problems that physicians face.
- 15:50And we'll add, they did indeed
- 15:51have a session on this for the
- 15:53first year students, which we are
- 15:55really excited to see and we want,
- 15:56we'd love to see more of those
- 15:58kind of discussions happening.
- 16:01So for this part, we're going to show
- 16:04a short live demo of 1/3 use case.
- 16:07So this is something that we use when
- 16:10reviewing our like clinical cases so far
- 16:12in this first year of didactic curriculum,
- 16:15but also something that we might use
- 16:17on the wards after seeing a patient
- 16:20brainstorming our initial differential,
- 16:21but then wanting to check back and
- 16:24see if there's other differentials
- 16:26that we're potentially missing,
- 16:27things that we're that we should
- 16:29should be considering here.
- 16:30So hopefully this screen share works
- 16:34and here I'll click submit and chat.
- 16:39BT can help brainstorm this list and
- 16:43it gives us initial list linking the
- 16:45specific symptoms in the presentation to
- 16:48different potential diagnosis on that list.
- 16:51But I think the most useful thing here is the
- 16:53additional follow up questions you can ask.
- 16:55So I won't go through each one individually,
- 16:58but things ranging from which physical
- 17:00exam maneuvers might we want to
- 17:02consider to further differentiate
- 17:04between these diagnosis.
- 17:06What laboratory should we order and
- 17:08what might, what results might we
- 17:10expect for each of these diagnosis,
- 17:12what for their work up,
- 17:14what treatments, etcetera.
- 17:15And you know,
- 17:17in addition to asking across
- 17:18the board for all of them,
- 17:20you can hone in and say
- 17:22for PCOS specifically,
- 17:23what do I need to be thinking about?
- 17:25So that interactive component
- 17:27is really helpful for us.
- 17:30Yeah, we joke that you're arguing
- 17:32with ChatGPT and that's actually
- 17:33the most useful part of it.
- 17:36And the last note is the
- 17:37same thing we if we have,
- 17:39if we're rotating together and
- 17:40we had a few minutes before
- 17:42talking to the attending,
- 17:43we definitely want to go back and
- 17:45forward talking about our differentials
- 17:46and seeing if there's things that and
- 17:48Elizabeth thought of that maybe I missed.
- 17:50So we kind of think of it as
- 17:51a conversation like that.
- 17:54So we know that AI and medicine
- 17:56sounds really scary and uncertain,
- 17:59and we have our own
- 18:00hesitations about it as well.
- 18:01But the reality really is that we
- 18:04don't actually have a choice because
- 18:06this is coming whether we like it or not.
- 18:09Conrad and I are starting in the hospital
- 18:12starting in January on our clerkships
- 18:15and as we mentioned there is ChatGPT
- 18:18being integrated into EPIC at Yale New Haven.
- 18:22And so that means literally from the
- 18:24beginning of our time in the clinical
- 18:26spaces ChatGPT will be present.
- 18:29So Yale School of Medicine's mission
- 18:31statement says that the school is striving
- 18:34to create leaders in medicine and science.
- 18:37And we really believe that in
- 18:39order to be equipped to be future
- 18:41leaders in medicine and science,
- 18:43we must be at the forefront of AI and
- 18:45medicine from the beginning of this movement.
- 18:47That really is quite inevitable.
- 18:52So we built in a little
- 18:55pause midway through this.
- 18:56We have about 3 to 5 minutes
- 18:58for one or two questions,
- 19:00if people have them.
- 19:01I haven't been watching the chat,
- 19:03so maybe we want to go there.
- 19:04The one I'll I'll let you
- 19:06decide how we open things up.
- 19:08No, let's open up to the
- 19:10group if they have any chats.
- 19:12If you want to unmute yourself
- 19:13and share your questions,
- 19:15we'd love to hear them. I
- 19:18had a question about accuracy again, I want
- 19:21you to review again how is the accuracy
- 19:27of the information that you are getting?
- 19:30Like if you use it as a mentor
- 19:31in the differential before
- 19:33you present to the attending.
- 19:35I mean we're so used to everything being
- 19:37peer reviewed and
- 19:39up to date or the
- 19:40other learning modules by
- 19:42another human being.
- 19:43So who is reviewing the chat review,
- 19:47especially if the Internet they're
- 19:48looking at is like you know,
- 19:51over a year old which is what you said
- 19:53in the beginning. So
- 19:54how is the accuracy being monitored?
- 20:00So I think that's a really good
- 20:02question and something we're always
- 20:04thinking about as we're using it.
- 20:06It was trained on the Internet
- 20:07and that's a scary thought.
- 20:08We all know we've all read things
- 20:10that are not true on the Internet.
- 20:12So that's definitely something
- 20:14we need to acknowledge and like
- 20:16understand when we're interpreting
- 20:18the responses from Chatchibiti.
- 20:20Never use Chatchibiti as a like a
- 20:23replacement of up to date for example.
- 20:25That being said, it is performing.
- 20:29I would say most of our classmates
- 20:31right now are not able to hit
- 20:3385% across all the STEP exams.
- 20:36I think that's one indicator of the
- 20:39pretty high accuracy that things
- 20:41are able to achieve right now.
- 20:43So it's not,
- 20:44it's imperfect and it's really important
- 20:46that people understand that it's imperfect.
- 20:49But I do think that it can be a
- 20:51really a really helpful classmate.
- 20:53I would even argue more helpful than myself.
- 20:56I know I'm not achieving those scores
- 20:58on any of the STEP exams right now.
- 21:00So I would I would recommend
- 21:02shooting questions to to ChatGPT
- 21:04maybe before a classmate asked me.
- 21:07And that's also why we recommend arguing
- 21:10with it and why we think it's important
- 21:12that we like start using it in our
- 21:15classrooms to then be able to learn,
- 21:18like how you can tell if
- 21:20it's hallucinating or not,
- 21:21or sort of learn how we can
- 21:23implement our own best practices
- 21:25and safeguards to ensure that we
- 21:29are like checking it appropriately.
- 21:31And I think that there isn't necessarily
- 21:33a best practice established at this point.
- 21:36And I think that's where Yale
- 21:38School of Medicine comes in.
- 21:39That's an opportunity for
- 21:41us to really lead the way,
- 21:44right. So we have time maybe for
- 21:46one more question live, but because
- 21:48there are so many of us presenting,
- 21:50we'll have everybody start replying
- 21:51in the chat as well once we're done.
- 21:53So Rob, I think you were the
- 21:55first question that I saw,
- 21:56if you wouldn't mind us sharing,
- 21:58I think, I
- 21:58think Dana actually went before me,
- 22:00but I I would just wanted to
- 22:01reply to Peggy, which is to point
- 22:03out that the idea that students
- 22:06are using definitive textbooks
- 22:08only for all of their look
- 22:09UPS is not really what's
- 22:11happening, right. So people
- 22:13look up whatever they find,
- 22:14and the question is not, is it perfect,
- 22:16but is it better or at least
- 22:17as good as the alternatives?
- 22:19I mean, you know, I, as, you know,
- 22:21I teach a course and I sit
- 22:22through courses and I see people
- 22:24say things that aren't true.
- 22:26And so, you know, even at Yale,
- 22:28even with the fancy Yale professors,
- 22:30people say something that's wrong.
- 22:32I see things in textbooks that are wrong.
- 22:34So the idea that they're, you know,
- 22:36it's like the idea that somebody
- 22:37on the Internet might be wrong
- 22:38and we have to correct them.
- 22:40I think that's really not really not correct.
- 22:43I think the idea is what the really idea is.
- 22:46Is it better than the alternative?
- 22:47That's really the issue.
- 22:51Dana, I think we can sneak you in.
- 22:52What's your question?
- 22:55I was just going to add something
- 22:57that's probably not a novel thought,
- 22:59but that the input is only going to be
- 23:02as good as the clinical skills will allow
- 23:05insofar as did you ask the right question,
- 23:08you know what's missing from the STEM?
- 23:10Did you do the right physical exam?
- 23:13And it's just to say sometimes when new
- 23:15technology comes along like radio radiology,
- 23:17like CAT scans and new learners start
- 23:19to say the answer is in the CAT scan.
- 23:21And that's more sensitive and
- 23:23specific than anything else.
- 23:24So it's just an opportunity to remind
- 23:26people to use it as you are both saying,
- 23:28as a tool and that it can't be a
- 23:31substitute for clinical skills,
- 23:35right. We are gonna have
- 23:37some good discussion today.
- 23:38Conrad, do you have any last
- 23:39thoughts before I move on?
- 23:41One last insertion is just I also
- 23:43think there is value in practicing
- 23:46differentials without ChatGPT.
- 23:48I'm not saying that it should replace
- 23:50that like independent reasoning.
- 23:52And I think the parallel analogy is that,
- 23:54like, students need to learn
- 23:55their multiplication tables
- 23:56to be able to do algebra.
- 23:58And if they just use a calculator and never
- 24:00learn how to do multiplication tables,
- 24:02that's going to cause a lot of
- 24:03problems for them down the line.
- 24:04So I think it's really important
- 24:06to develop these skills, like,
- 24:07independently without these tools as well.
- 24:10I just think it's a no one would say
- 24:12no one should use calculators ever.
- 24:14It's just we have to be thoughtful
- 24:15about when we use them and and how
- 24:17they're augmenting our education.
- 24:20Awesome. Thank you so much.
- 24:21We're definitely going to have
- 24:22time at the end where we can
- 24:24fool some of these questions.
- 24:25But Conrad and Elizabeth David,
- 24:27if you want to start answering some
- 24:28of these other questions in the chat,
- 24:30that'd be great.
- 24:31So we've heard about some questions
- 24:34on student use of ChatGPT.
- 24:36But how can faculty use ChatGPT
- 24:39to augment medical education?
- 24:41And I say this as someone who doesn't
- 24:43consider themselves particularly tech savvy.
- 24:45I run a couple courses.
- 24:47But to be honest, more than anything,
- 24:49I'm a millennial who has significant fear
- 24:51of missing out more than anything else.
- 24:53And so I follow my journals,
- 24:55we all get our our updates right
- 24:57in our e-mail.
- 24:58And I'm seeing in academic medicine,
- 25:02JGME, medical teacher,
- 25:03more articles about how we can use this,
- 25:05even published medical education
- 25:07articles in JAMA and the New
- 25:09England Journal of Medicine,
- 25:10which as all of Y'all scholars
- 25:12know is very challenging to do.
- 25:14So clearly it is becoming
- 25:17mainstream not just in healthcare
- 25:19but specific to medical education.
- 25:22So with that in mind and we're
- 25:23starting to get a feeling about this,
- 25:25I wanted to see just that,
- 25:27how are you as faculty and
- 25:30learners feeling about chat GBD.
- 25:32So we're putting up a poll there.
- 25:34You can put as many choices
- 25:35as I want and I know,
- 25:36I know there are multiple
- 25:38choices in there too.
- 25:39I'll answer this after as well,
- 25:40but I'll say at least for me,
- 25:42for the past 6 three to six months,
- 25:45I've been talking to educators at Yale,
- 25:48Basic science,
- 25:49clinical science,
- 25:49and leaders across the country
- 25:52around ChatGPT.
- 25:53And these are some of the adjectives
- 25:55that I'm seeing and I know they've
- 25:57all crossed my mind to some degree.
- 26:00So I want to think about why
- 26:01we're having these concerns,
- 26:02some that have already been named.
- 26:03So the first one is the quality
- 26:05of the information.
- 26:06This is by far the biggest concern
- 26:08that's come up with educators.
- 26:10And I would certainly agree with Rob that
- 26:12while it's something to be concerned about,
- 26:14I think we have to remember like teaching
- 26:16comes from different sources all the time.
- 26:19We ask our learners to often study
- 26:21in groups together and where this
- 26:22where's that information coming from?
- 26:24I think another barrier that comes up
- 26:27is familiarity or comfort with use,
- 26:29where we hear things like large
- 26:31language models, predictive texts,
- 26:33chats,
- 26:33prompts,
- 26:34and it feels almost like we're
- 26:37coding and that's not what ChatGPT is.
- 26:39But I can understand where that
- 26:41concern comes from.
- 26:42And I think a very real one is cost.
- 26:44There's a literal cost of a monthly
- 26:46fee if you wanted to get Chachi BT4.
- 26:48But I think that more than that,
- 26:49it's the time investment kind of
- 26:51to the second point of how long
- 26:53is it going to take to learn.
- 26:55I think we're getting a lot of
- 26:58the similar concerns as when the
- 27:00EHR first came up in my training.
- 27:03So with that in mind,
- 27:05I wanted to also ask about how
- 27:08often you're using ChatGPT.
- 27:10But before that, David,
- 27:12I was wondering if you could
- 27:14show us about our our feelings
- 27:16on ChatGPT right now. So good.
- 27:18We have a lot of people who are curious.
- 27:20That's probably why you're here.
- 27:21That makes sense.
- 27:22But I'm kind of surprised we have a lot,
- 27:25a lot more optimism in the room.
- 27:27So we can build on that.
- 27:28That's right.
- 27:29We're already winning here,
- 27:31Conrad and Elizabeth and David.
- 27:32So next poll that's coming up
- 27:35is how often do you use ChatGPT.
- 27:38So David,
- 27:39we wouldn't mind closing this poll and
- 27:41opening up the next one about frequency.
- 27:44So are you never,
- 27:46have you never ever used it?
- 27:48Are you using it daily or have
- 27:50you tried it once and that
- 27:52was more than enough for you.
- 27:54And as we think about how
- 27:55often people are using it,
- 27:56I think it's important to think about
- 27:58change in the greater perspective
- 27:59of innovation and technology.
- 28:01So Everett Rodgers had some work in
- 28:04the 1960s around the diffusion of innovation,
- 28:07also known as the technology adoption cycle.
- 28:10For those of you who are Malcolm
- 28:12Gladwell enthusiasts and have
- 28:13read The Tipping Point,
- 28:15generally we think around getting to
- 28:17this 16 to 18% of any population.
- 28:20And after that point,
- 28:21you're starting to move more and
- 28:24more towards something that's
- 28:25just going to happen,
- 28:26something that's inevitable.
- 28:27And so I wanted to get David's
- 28:30take as probably the one with
- 28:32the most expertise on ChatGPT.
- 28:34David, how do you feel?
- 28:35Where do we feel like we are on
- 28:38ChatGPT in general academic use?
- 28:40And then specific to medical education,
- 28:42how many people are using it.
- 28:45So
- 28:45I think this is, this is partially
- 28:48a question of how we see it.
- 28:50You know, there's a lot of people
- 28:51who are certainly talking about it.
- 28:53There's a lot of conversations,
- 28:55there's a lot of hype,
- 28:57but and certainly students are
- 28:59using it quite more frequently
- 29:01than I ever expected them to be.
- 29:03But it hasn't quite hit that
- 29:05public consciousness moment.
- 29:07So I would estimate, you know,
- 29:08we're somewhere across the
- 29:10chasm in the early majority and
- 29:14specifically in medical education,
- 29:16I think there's been a lot of hesitancy,
- 29:18There's been a lot of questions about,
- 29:21you know,
- 29:21rightly as we've even seen today about
- 29:24validity and the the larger problems of,
- 29:26you know,
- 29:27accuracy of medical information
- 29:29amidst the Internet.
- 29:30But there's been some focus efforts to,
- 29:33to use it well.
- 29:34And this is really what we're talking about,
- 29:36right, Using it well.
- 29:37And that's going to be the tipping point.
- 29:40Once that has hit mainstream,
- 29:43once we're all talking about it
- 29:44or using it more commonly,
- 29:46that's when we'll tip to the late
- 29:48majority from the early majority.
- 29:51And I think we're really just just
- 29:52breaching the chasm right now.
- 29:54All right. So David, what, how often,
- 29:56where are we in the chasm,
- 29:58at least as far as our audience right now?
- 30:00So how's that poll looking?
- 30:03I love harkening back
- 30:03to a time of zoom polls.
- 30:05I haven't done one of these
- 30:06in a in a minute I would say.
- 30:08So we have a lot of never users,
- 30:09maybe a little bit, even some daily users.
- 30:11I'd say.
- 30:12I probably personally am a weekly user of it,
- 30:16probably every time I write up or start
- 30:19doing some sort of curriculum development.
- 30:22But I will say that honestly,
- 30:26if you're using it my my gestalt.
- 30:28If you're if you use it within
- 30:30the next like three to six months
- 30:31to some degree at least,
- 30:33like a few times,
- 30:34you probably would be in the early
- 30:36adopters is my read of the room from
- 30:38medical schools across the country.
- 30:40So how should educators use ChatGPT now?
- 30:43This review was published in the
- 30:46spring of 2023 in Academic Medicine
- 30:49on how artificial intelligence
- 30:51large language models like ChatGPT
- 30:53should be used in medical education.
- 30:56They came up with some competencies,
- 30:58knowledge of AI, ethical implications.
- 31:00Think about the bias, for example,
- 31:01of data sets.
- 31:03It can actually propagate bias
- 31:04if not reviewed correctly,
- 31:06how it's affecting clinical encounters
- 31:09and how we're working to improve it.
- 31:11But I think it's important
- 31:12to remember a couple things.
- 31:14One is that this isn't what every
- 31:17single medical educator needs to do.
- 31:19These are institutional values
- 31:21that we're trying to promote.
- 31:242 With that in mind,
- 31:26we have our whole informatics
- 31:28department here at Yale,
- 31:30for example,
- 31:30that David is a part of who are
- 31:33helping and are going to develop
- 31:35more and more curricula and
- 31:36workshops for students and faculty
- 31:38alike to develop those skills.
- 31:40So I think honestly it might be
- 31:43better just to play around with
- 31:44it ourselves and see how it might
- 31:46be beneficial to us as educators.
- 31:48So I'll tell you my story.
- 31:49It's nothing particularly special,
- 31:51but when I first started using
- 31:53ChatGPT was over the summer.
- 31:55I direct our academic support
- 31:57program and I had to develop some
- 32:00cases for SP standardized patients,
- 32:02for our students who we work with.
- 32:03And I saw Barbara Hildebrandt on the call.
- 32:05She gave me a few reminders and it
- 32:08wasn't until the day before that
- 32:09I started writing these cases and
- 32:11I blocked out a few hours and I
- 32:13was like alright,
- 32:14I can write them myself or I could
- 32:16see what ChatGPT can do for me.
- 32:18So, not feeling great initially,
- 32:20but I saw what ChatGPT can do.
- 32:23I put in a general prompt below
- 32:26this and don't worry, Medicine,
- 32:29school, medicine, curricula,
- 32:30clinical skills program.
- 32:31I didn't show the actual case or I won't,
- 32:34but I I put in the one of our cases,
- 32:37one of our representative cases.
- 32:39Use this new prompt and chief concern
- 32:41and this is what it spit out to me now.
- 32:43This is a script for a standardized patient
- 32:47and it's written very,
- 32:49very similarly, but on a new case.
- 32:52So this is talking about knee pain.
- 32:54It gives us an HPI,
- 32:55but also things like emotional
- 32:57context as well social history.
- 32:59This is more than enough for
- 33:01an SP to use to play a role.
- 33:03It even gives us a little bit
- 33:05of a differential diagnosis.
- 33:06If you wanted a faculty to have a little bit
- 33:08of teaching points that they could give.
- 33:10Obviously they could add their own expertise,
- 33:12but I saw that and at
- 33:13least for case development,
- 33:14as someone who's involved in clinical skills,
- 33:16I was like,
- 33:17this is this is pretty good enough
- 33:19for me to pay 20 bucks a month
- 33:21now for a ChatGPT for.
- 33:22Since then I've used a few times
- 33:25creating facilitator guides.
- 33:26Literally after this we'll be doing
- 33:28a clinical reasoning session with
- 33:30one-on-one consultations between
- 33:32faculty and tutors, I think.
- 33:34And Elizabeth has a slot coming up soon too.
- 33:37And we use ChatGPT for the facilitator guide.
- 33:40I've used it to update curricula.
- 33:42For example,
- 33:43we have a session on Bayesian
- 33:45reasoning in the physical exam.
- 33:47I got feedback from learners on ways
- 33:49to improve and we use that to create
- 33:52new cases and learning objectives.
- 33:54I've used the prayer feedback on the wards,
- 33:57had a couple struggling learners
- 33:59and had to deliver feedback,
- 34:00and I put in like the pearls format,
- 34:03the go ask, tell, ask model.
- 34:06And just so I didn't have to
- 34:08perseverate on the hard discussion,
- 34:10I kind of had an idea of what might
- 34:12be some things that I could say.
- 34:14We've also had within our
- 34:16clinical reasoning course,
- 34:17one of our associate course directors,
- 34:19Darius developed some exercises
- 34:21around CHAP GBT 5 prompts and that
- 34:24I think our students found OK,
- 34:27Conrad and Elizabeth were in that class.
- 34:29I think it was like tolerable.
- 34:30But I think more than anything I
- 34:32realized from that is that we don't have
- 34:35to teach and use ChatGPT all the time.
- 34:37Our students are going to
- 34:38get that information.
- 34:39It's more important for us to,
- 34:40for them to learn somewhere along
- 34:42the lines about the concerns,
- 34:44earns and ways to use it and
- 34:46then for us to use it ourselves.
- 34:48And so with that in mind,
- 34:49when it comes to teaching and these
- 34:52are some general categories from
- 34:54a review or that came out or a
- 34:56perspective piece that came out in
- 34:58that kind of medicine in August from Bascard.
- 35:00And some opportunities for teaching,
- 35:02like I mentioned around
- 35:04curriculum development are there,
- 35:05But also evaluation.
- 35:06We can look at our curricula and
- 35:08eventually with large language files,
- 35:11if not already, say like, hey,
- 35:12are we meeting our goals,
- 35:14our educational program objectives?
- 35:15We can put our content in that
- 35:18and evaluate it.
- 35:19One conversation I've had a lot with
- 35:20David is around teaching framework.
- 35:22So we teach knowledge and medical education,
- 35:25but then eventually they have
- 35:26to practice and there's this
- 35:28huge space in between about how
- 35:31students synthesize and connect
- 35:32information. And that is a place
- 35:35where ChatGPT could really help.
- 35:36How do you interact with information?
- 35:38How can you make things iterative
- 35:39in ways that we can't do as faculty
- 35:42because we simply aren't there
- 35:43all the time with students now?
- 35:45So the concerns are there,
- 35:47the quality of the resources,
- 35:48information for sure.
- 35:49But I think a sneaky concern that
- 35:52comes up that I think is a good
- 35:54concern for us is that chat DVD
- 35:56can replace a lot of the stuff
- 35:58that we're already doing and that
- 36:01might increase the need for more in
- 36:03person faculty tasks like where's
- 36:05the real value in medical education?
- 36:07Is gonna come up more often than not.
- 36:10There's certainly gonna be
- 36:12opportunities in assessment,
- 36:13creating items which will talk about rubrics,
- 36:16providing feedback both in answers,
- 36:18keys or as Anne Elizabeth talked about,
- 36:20and more about personalized tutor mode.
- 36:22The way that you won't that you
- 36:24might have it in office hours if
- 36:26it was only you and the faculty
- 36:27member going back and forth,
- 36:29and so certainly there
- 36:31are opportunities there.
- 36:32The concerns as well arise between,
- 36:35for example,
- 36:36if you have open-ended answers,
- 36:38open-ended questions with answers to,
- 36:40they're going to have to be reviewed
- 36:42carefully for AI generation for the
- 36:45ability to to answer multiple choice
- 36:47questions clearly at a very high level.
- 36:50In my opinion,
- 36:51it's probably functioning somewhere
- 36:52between the level of a sub I and
- 36:55an intern at medical knowledge
- 36:56at least ChatGPT 4,
- 36:57so strengthening proctoring
- 36:58is going to come up.
- 36:59It hasn't done great with automating scoring,
- 37:02at least in our early studies,
- 37:03but I imagine it'll get better
- 37:06as our new technology improves.
- 37:08But I will say this,
- 37:09it is already starting to be used
- 37:12in programs of study for item
- 37:14assessment or item development.
- 37:16So this is from the University
- 37:18of Colorado's medical school.
- 37:19It's a paper that came out a medical
- 37:21teacher I think maybe two months ago.
- 37:23And for their five week reproductive
- 37:25course they had a total of 290 questions
- 37:28that were included across quizzes,
- 37:31across their qualifier,
- 37:32their final exams And they used
- 37:35ChatGPT to create some of those items,
- 37:37about 10% of them.
- 37:38And they used this flow chart,
- 37:41which isn't as serious as it seems and
- 37:43honestly not as detailed as it is.
- 37:45If you look at that paper,
- 37:46the prompt,
- 37:47this third purple box was very limited.
- 37:50It was give us a four sentence
- 37:53question on a reproductive system.
- 37:54So it didn't even specify in ways that
- 37:57I did like This is how I want to format,
- 37:59this is the specific topic within that.
- 38:02And yet they still used it.
- 38:04And when all was said and done,
- 38:06they found that these questions
- 38:08were similar and difficulty
- 38:09granted after expert review,
- 38:11assessment team review.
- 38:12And they found that the difficulty
- 38:16was similar and arguably more
- 38:19importantly the discrimination,
- 38:20the discrimination ability was similar.
- 38:22So between high performers
- 38:24and lower performers,
- 38:26the question performed similarly
- 38:28and faculty found it easy to
- 38:30use and saved a lot of time in
- 38:32their question development process.
- 38:34There are other ways that ChatGPT will
- 38:36certainly be used in medical education,
- 38:38probably beyond the scope of what we
- 38:40talked about in research authorship.
- 38:42I know journals are still trying
- 38:43to figure that out right now.
- 38:45Also in admissions as well.
- 38:48I think there is something to be said.
- 38:49I know there might be an initial concern
- 38:51that people are writing their personal
- 38:53statements with artificial intelligence,
- 38:54but I think there's a counter argument
- 38:56that actually can be good for reducing
- 38:59disparities for a lot of students who
- 39:01don't have the resources that maybe some
- 39:03universities have as far preparing their
- 39:06health profession students for application.
- 39:08So there's certainly other ways,
- 39:10but at least we focused here
- 39:12on teaching and assessment.
- 39:14So some takeaways,
- 39:15at least from my use and my reading of the
- 39:18room when it comes to ChatGPT is First off.
- 39:21Again, you're still in the early adopters,
- 39:23just try it.
- 39:25Remember that you don't need
- 39:27to be a super user right away.
- 39:30Also remember that what you're developing
- 39:32should be really seen as first steps,
- 39:36not final products,
- 39:37and that way you can use your
- 39:39time and expertise to edit and
- 39:42get higher quality products.
- 39:44Another thing I mentioned reinforcing
- 39:45with Dave and I talked about,
- 39:47it can be used for synthesis and framework,
- 39:49less so for primary information.
- 39:51I don't think that students are
- 39:53using it for primary information
- 39:54and we asked them about it.
- 39:56That's not how they said
- 39:57they're using as well.
- 39:58In fact,
- 39:58that's where they've been telling us.
- 39:59They've been skeptical too.
- 40:01It will force us to find where
- 40:04the limited resources are where
- 40:06students aren't able to use it.
- 40:08So just keep your eyes open for updates,
- 40:10but ask for help when available.
- 40:13And in this sense,
- 40:14a quick shout out to David,
- 40:16whose e-mail will be at the end,
- 40:18and other colleagues as far as keep your
- 40:21eyes open for opportunities to learn more.
- 40:25So with that in mind,
- 40:25I wanted to move on to just some examples
- 40:28so you can kind of see what I'm doing.
- 40:30I know I did some screenshots,
- 40:31but in addition to that I wanted to
- 40:33show you some examples of some of the
- 40:36chats that I did and see if maybe you
- 40:38can take a look at these and say you know,
- 40:41how helpful would it be.
- 40:43So in this example,
- 40:44a basic science lecture on
- 40:46water soluble vitamins,
- 40:48your vitamin BS, vitamin CS.
- 40:51If you wanted to help it
- 40:53create a lecture outline,
- 40:54this might be an example of what it
- 40:56would look like. So I give a prompt.
- 40:59I'm a faculty member.
- 41:01I have 60 minutes.
- 41:02I'm mostly didactic,
- 41:03little bit large group,
- 41:05but also I was able to put in a
- 41:08Yale School of Medicine educational
- 41:10program objective so that could
- 41:12help me make sure I'm hitting the
- 41:15right points and spit out this.
- 41:18So I'll wait for
- 41:19510 seconds to let you skim the links.
- 41:22David, would you mind throwing the link
- 41:24to this in the chat for our faculty?
- 41:27Take a look at another time.
- 41:30But at least for me, looking at this,
- 41:32I would give this maybe like a two 2 1/2.
- 41:35I don't really do lectures anymore,
- 41:37but if I had to make 60 slides on something,
- 41:40it would be helpful start to
- 41:41figure out what I should probably
- 41:43think about putting into it,
- 41:44maybe with my own resources.
- 41:47Now let's say I wanted to add some cases.
- 41:50Let's write some cases, let's not.
- 41:51I don't want them to be too long.
- 41:53I want to chance always told me,
- 41:54God, I got to make sure I keep
- 41:56the words on my slides limited.
- 41:57So let's only make it 250 words.
- 41:59Let's see what a case might look like
- 42:02on thiamine deficiency or B6 deficiency.
- 42:05And the first paragraph is a case,
- 42:08and for me and this is a person
- 42:09who writes cases,
- 42:10who talks to colleagues around the country,
- 42:13for example,
- 42:14the clinical problem solvers at UCSF.
- 42:16They've told me that you don't
- 42:18really need to have a perfect case.
- 42:20You can tweak cases,
- 42:21you can adjust them for what
- 42:23you need to know.
- 42:24This is more than enough initially,
- 42:26and I can add and change some things to
- 42:28make it more representative based off
- 42:30of my own expertise with this topic.
- 42:32So I'd give this maybe a three
- 42:36then let's say I want to write a question.
- 42:38Now, one way that we can get
- 42:40faculty to buying in general,
- 42:41to medication, to medication,
- 42:43to our educational content is that it can
- 42:47relate to them and what their needs are.
- 42:50And so one thing you could ask it is hey,
- 42:52like let's make this format,
- 42:54USM LE1 style, USML E2 style.
- 42:57And you can at least say that
- 42:59to your students as well.
- 43:00And that might make them more interested
- 43:02and increase their motivation for
- 43:04doing some of those questions.
- 43:05And even more than that,
- 43:07you can see it can create tables for you.
- 43:09Now I certainly would adjust
- 43:11and edit this table,
- 43:13but it would give me a good start
- 43:14if I wanted to have an answer.
- 43:16Key of illness scripts for
- 43:18any of these given vitamins on
- 43:21deficiency specifically range from
- 43:23epidemiology to diagnostic testing.
- 43:25So all of this,
- 43:26it was created in less than 5 minutes.
- 43:29It might have taken a little
- 43:31bit longer if I had it truly
- 43:33searched through all of like USM,
- 43:35LE1 parameters through their database,
- 43:37but you can see that it certainly is a star.
- 43:42And so as we're nearing the end,
- 43:43I do want to move on to a large
- 43:45room discussion 'cause I know
- 43:47the chat has been pretty active.
- 43:48So we'll skip our second case
- 43:51though David put one in the
- 43:53chat of a clinical example.
- 43:55But for a large group discussion,
- 43:58I'd love to hear from y'all about
- 44:00seeing from the students and
- 44:01now the educator's perspective.
- 44:03What are your impressions of
- 44:04ChatGPT for creating content?
- 44:06Would you use it in your own
- 44:08educational practice now?
- 44:09And what are some next steps
- 44:11that you think EOS Good Medicine
- 44:14should be looking at for ChatGPT?
- 44:16So what do y'all think?
- 44:26So thalan, I love Gary's question,
- 44:28he says, Curious how much of
- 44:30this talk was AI generated?
- 44:33So I would say our cases obviously
- 44:35are that we plugged into Chat
- 44:37chi BT but no, at least none of
- 44:39my slides where I had to use,
- 44:40you know, this one myself.
- 44:42But I don't know Conrad and Elizabeth,
- 44:44Did you use Chat Chi T
- 44:46for any of your slides?
- 44:48They didn't make any of the
- 44:49slides one place we did.
- 44:50I had a few conversations back and
- 44:52forth when, like brainstorming
- 44:53ideas on how Chatship team might
- 44:55be used in medical education.
- 44:58So that was one place where I
- 44:59don't know if it added any ideas,
- 45:01but was reassuring that we'd thought of
- 45:03most of the ideas that came up with.
- 45:06We thought of them already.
- 45:07So that was one place we used it.
- 45:09But good question.
- 45:11And I will say Comrade and
- 45:13Elizabeth have been giving this
- 45:14talk in a couple iterations.
- 45:16So this is not the first time if
- 45:18it seems too smooth to be real.
- 45:20It's just their practice and expertise.
- 45:25Awesome. Allison, you have a question.
- 45:29Hi, Salon, thank you
- 45:30so much. And and to all
- 45:32of our presenters today,
- 45:33this is just really,
- 45:34really interesting and I
- 45:36appreciate the time it took
- 45:38to prepare this. I do have a
- 45:40question Thelan being on the
- 45:42continuing professional development
- 45:43side and thinking of our faculty
- 45:46and I've used it once one of I was a one,
- 45:51but there's so much opportunity,
- 45:52but I think it's a vulnerability
- 45:54of saying I don't know how to do this and
- 45:58how to best prepare. So from Acme side,
- 46:01how can we provide those opportunities
- 46:04to teach our faculty and to be
- 46:08vulnerable and to say you don't know and
- 46:11still find a a best practice
- 46:13for integrating this in, even
- 46:15if it's very slowly and
- 46:17will take some time. Yeah,
- 46:19we're figuring this out
- 46:20on your own by the way.
- 46:21If you have questions for specific people,
- 46:23feel free to like put that in the comment
- 46:26box with your question or ask initially.
- 46:28I'll take this one.
- 46:29I guess initially, Allison,
- 46:30I'll say that we're all learning together
- 46:35and I think it might be a little bit
- 46:37bumpy as we create more content.
- 46:39I've seen other institution workshops,
- 46:41they'll have these like playgrounds
- 46:43where they'll give an open space and
- 46:45all faculty can kind of use whatever
- 46:47tools that they're working on at
- 46:49any given time and ask questions
- 46:51to someone who's hosting it.
- 46:53And some people will just like take
- 46:55those links and go offline and use it.
- 46:56And that's OK.
- 46:58But I think just having low
- 47:00requirements or expectations and
- 47:02just having a lot of opportunities
- 47:04is just the best way to do it.
- 47:06David,
- 47:06I know you've like thought about this a lot.
- 47:09What do you think would be a
- 47:10good approach for helping our
- 47:12faculty learn about ChatGPT and
- 47:13other like large language models?
- 47:16So first point, do you mind
- 47:18taking down the slides so we can
- 47:19see everybody in in panorama.
- 47:21The the easy answer to that is everybody's
- 47:26talking about it from the Porvu Center,
- 47:28from us to from the larger groups
- 47:31within the Faculty of Arts and
- 47:33Science and there's been a lot of
- 47:35communities built as a product of that.
- 47:37The the I'll, I'll say the section
- 47:39for Biomedical Informatics and
- 47:41Data Science should be your first
- 47:42stop because we're here to help.
- 47:45You know myself as well as others
- 47:47are have been using large language
- 47:49models before chat EBT ever existed.
- 47:52So if you would like that kind of
- 47:55opinion or that kind of expertise
- 47:57to join you please reach out.
- 47:59I am happy to help.
- 48:00I believe the slide deck will be
- 48:02shared at the end my phone number and
- 48:04e-mail or at the end of everything.
- 48:07Very happy to help.
- 48:09While I'm talking,
- 48:10let me quickly also mention that within
- 48:13the larger strategic partnership between
- 48:15the health system and the School of Medicine,
- 48:18there are capacities to use
- 48:22LLMS with Phi with, you know,
- 48:26proprietary resources within the School
- 48:28of Medicine that are being explored.
- 48:32You know,
- 48:32like I can't even begin to name the whole
- 48:34list of people involved in this process,
- 48:37but through Biomedical
- 48:39Informatics and Data science,
- 48:40through the YNHH,
- 48:42Medical Information Officers and
- 48:45Digital Transformation Solutions folks,
- 48:47it is happening.
- 48:48If you want to be involved,
- 48:50if you want to be connected,
- 48:51reach out and I will do my best
- 48:53to get you to the right person.
- 48:56So for questions, maybe
- 48:58we can go to Dana next.
- 49:03Thanks for the presentation.
- 49:05With respect to challenges and applications,
- 49:07a quick thing that came to mind
- 49:09as a recent clerkship director and
- 49:11trying to embrace more holistic
- 49:13inclusion of social determinants of
- 49:15health into our clinical teaching.
- 49:17And I know Sheila Gupta,
- 49:18who's working on our masters right now,
- 49:20is working on this whole project of
- 49:22how to develop faculty to include
- 49:23this kind of thing in their content.
- 49:25And it seems so massive to train
- 49:27everyone and to like find all
- 49:29the articles to put them.
- 49:30So it just seems like a really good
- 49:33potential application if you can train
- 49:35it to for this individual content,
- 49:38can you incorporate and incorporate
- 49:40social determinism of health?
- 49:42It would really help a lot of faculty to
- 49:45some of those EP OS and curricular demands.
- 49:47So I love that possibility.
- 49:50And then to that point,
- 49:51we just had a case that like was
- 49:54written by faculty specialists that
- 49:56had some issues as far as like bias and
- 50:00sigmatizing language goes and actually
- 50:02to one hand developing case to ChatGPT
- 50:04allows you to spend more time looking
- 50:06at that instead of creating the content.
- 50:08But also like Conrad and
- 50:10and Elizabeth have said,
- 50:12we can ask follow up questions
- 50:13to it and for example,
- 50:15input like watch out for these types
- 50:17of terms, these type of themes.
- 50:18So it can actually be helpful in that
- 50:20sense too if you're on the lookout
- 50:25Gary. So one
- 50:27of the questions that I see coming
- 50:29up most often regarding AI is sort
- 50:31of the ethics of it and I I deeply
- 50:34recognize that were very early in the
- 50:36world of AI to to really substantively
- 50:38we have those conversations.
- 50:40But I'm wondering how each of you
- 50:43are managing transparency with this.
- 50:45And I mean, I tried writing a letter of
- 50:48recommendation this year using ChatGPT
- 50:49and I just there was like a a deep
- 50:52almost dirty feeling doing it and so
- 50:55how how are you approaching transparency
- 50:58with with this
- 51:01Conrad man? Elizabeth what
- 51:01do you think about that?
- 51:06Definitely something.
- 51:07But I think there needs to be
- 51:09better guidelines about like I
- 51:10think people need to think more
- 51:12about how we're addressing this.
- 51:13The first thing that comes to
- 51:15mind is like research literature.
- 51:16Different journals are dealing with
- 51:18this in very ad hoc different ways,
- 51:21and a recent submission I did,
- 51:24they asked for every ChatGPT log I'd
- 51:26had related to the content of the
- 51:29research that we were submitting.
- 51:31And that was a really extensive.
- 51:33I'd gone back and forward a lot with
- 51:35ChatGPT asking questions about like
- 51:37how to analyse statistical results.
- 51:40It's really good at statistics too,
- 51:41again needing to be corroborated,
- 51:44but it is a complex question and I think
- 51:47better guidelines need to be developed.
- 51:50I really agree with that.
- 51:51But I think, I mean Yale is one
- 51:53of the most prestigious academic
- 51:56institutions in the world.
- 51:58And I think like enforcing that above
- 52:02all we stand for academic integrity
- 52:04and personal integrity as physicians.
- 52:07And I think you know,
- 52:10essentially encouraging students
- 52:13to be to to act in a way that is
- 52:17consistent with those values and
- 52:19also to take personal responsibility
- 52:21and then having very severe
- 52:23consequences if they do not,
- 52:25I think makes a lot of sense.
- 52:27But definitely having like very explicit
- 52:31policies as to how you want things
- 52:33cited and how you want transparency
- 52:36to be specified in anything that
- 52:38you submit is is really important.
- 52:43Another example that I can think
- 52:45of is when I was shadowing a few
- 52:48probably months ago now in the CCU
- 52:50I was following along and as AI
- 52:52was a first year medical student I
- 52:53think at the time and really didn't
- 52:55have that much of A grasp and had
- 52:57a million questions and asked the
- 52:59attending that I was following if it's
- 53:01OK to be on my phone a little bit
- 53:04in the background Googling things.
- 53:06Well I asked.
- 53:07I admitted to using ChatGPT and
- 53:09was actually he was very interested
- 53:12and enthusiastic and read a lot of
- 53:14the responses and was very happy
- 53:16that I was using it.
- 53:17And I I think it for a lot of those
- 53:19little questions that come up it
- 53:22can be really helpful augmenting
- 53:24again recognizing not as a source
- 53:26of truth etcetera.
- 53:28But that kind of transparency
- 53:29I think can be helpful,
- 53:31but is definitely a nervous step to
- 53:34admit to your professor or attending.
- 53:37But maybe that's a tool you're
- 53:40using and being cognizant of how
- 53:41you're using it is very important.
- 53:43But that being said, like on the PhD
- 53:46side of what I'm doing here at Yale,
- 53:48so much of that type of learning is
- 53:51harnessing the tools that we have to
- 53:54produce the best research possible.
- 53:57And then when we write out our methods,
- 53:59you know, we talk about
- 54:00what tools we used and why.
- 54:02And I think that this,
- 54:05the applications of ChatGPT into
- 54:07medicine is almost pushing us to
- 54:10be more scientific in the way that
- 54:13we are thinking because we have
- 54:16to be using the methods correctly
- 54:19essentially in order to produce
- 54:21the appropriate output. I
- 54:23think that's a great, sorry.
- 54:25I think that's a great segue into
- 54:27to highlight one of the questions in
- 54:29the chat about mitigating systemic
- 54:31bias where ultimately at the end of
- 54:33the day we may not be able to right.
- 54:35This is essentially something running
- 54:38on the statistical collective
- 54:40intelligence of the Internet and
- 54:42related sources and then generating
- 54:43what that likely is going to say.
- 54:46It's going to be biased.
- 54:48We can't avoid it, but we can recognize it.
- 54:50We can identify it.
- 54:52We can work with scientific rigor
- 54:55to mitigate it.
- 54:56And if we are transparent and
- 54:59we do the best we can,
- 55:01then we are doing the best we can.
- 55:05And I think to an Elizabeth's point,
- 55:07you know, this is what leadership looks like.
- 55:11The last thing I don't want to
- 55:12run right up to the last minute,
- 55:14but I do think it's really
- 55:15important to address the chat
- 55:17comment about HIPAA compliance.
- 55:19And that is another thing that really
- 55:20needs to be emphasized to students,
- 55:22to residents, to everyone in the
- 55:24hospital is that with the public
- 55:26available versions of Chat BT
- 55:28that you use through a browser,
- 55:30no patient like private health
- 55:32information should ever go into
- 55:35those models because open AI for
- 55:37those public models specifically
- 55:39that are on the web browser,
- 55:41they do sometimes use what you
- 55:43input to further improve the models.
- 55:45So there is definitely risk
- 55:47of like Phi leakage.
- 55:49That's why the hospital has
- 55:50blocked those public models.
- 55:52And I think it's important for the
- 55:54hospital to explain why that's happening.
- 55:56And I also think that there can be
- 55:58future solutions as David mentioned,
- 56:00for research purposes right now there
- 56:02is HIPAA compliant workarounds that
- 56:04never embed Phi into these models.
- 56:07And I think there could be interactive
- 56:09like mobile uses as well in the future,
- 56:12but we don't have those yet and it's
- 56:13important that people understand
- 56:15the limitations in the meantime.
- 56:17So I think on that note,
- 56:19it's about time to wrap.
- 56:20Let's give you like a minute
- 56:21or two before whatever you
- 56:23have to do in the afternoon.
- 56:24Thank you for joining us to
- 56:26talk more about ChatGPT.
- 56:28This is going to be an
- 56:30ongoing discussion for many,
- 56:31many years as we're in medical education.
- 56:34So feel free to reach out.
- 56:36Thanks again and have a great afternoon.
- 56:38This has been MEDG.