PELC 09.26.24

Transcript

• 00:00This has been awful. So
• 00:02in case I just bug
• 00:04out, then,
• 00:06Katie,
• 00:06feel free to jump in.
• 00:09So it really is my
• 00:10pleasure to welcome Katie.
• 00:14Katie
• 00:15was really, I have to
• 00:16say, one of our stars,
• 00:18clinically and one of our
• 00:20stars,
• 00:20from the respective medical education.
• 00:24Katie was a
• 00:26resident, an intern and a
• 00:27resident here in med ped.
• 00:31Katie also did her
• 00:33a fellowship
• 00:35in internal
• 00:37medicine and at the same
• 00:38time got her MHS,
• 00:40MED.
• 00:41She then became the assistant
• 00:43professor
• 00:45in both doing medicine and
• 00:47pediatrics and was really instrumental
• 00:49in terms of the COVID
• 00:50pandemic
• 00:51in
• 00:52just
• 00:54in just adapting and helping
• 00:56all of us to adapt
• 00:58to a virtual,
• 01:00teaching platform.
• 01:03We were so sad
• 01:05a little over a year
• 01:07ago,
• 01:08when
• 01:09Katie
• 01:10elected
• 01:11to go to Emory to
• 01:13actually be the program director
• 01:15of their med peds program,
• 01:18which was which really is,
• 01:21something that they're so lucky
• 01:23to have her.
• 01:24But it definitely was a
• 01:25loss for us in terms
• 01:26of just having a medical
• 01:28education leader in that department.
• 01:30Katie's been really instrumental
• 01:32in terms of,
• 01:34her research focuses
• 01:36on EPAs.
• 01:38She
• 01:39actually
• 01:40really,
• 01:41I think, taught many of
• 01:42us the nuances
• 01:44of it,
• 01:46even though many of us
• 01:47actually work in that area,
• 01:48but I Katie's
• 01:50grasp of it,
• 01:52was just so deep.
• 01:54Other things that Katie,
• 01:55has been working on,
• 01:57certainly, she is very, very
• 01:59productive in terms of her
• 02:01scholarship,
• 02:03in terms of her invited
• 02:05presentations.
• 02:06She's worked
• 02:07on medical education research,
• 02:10qualitative
• 02:11research in medical education,
• 02:13bias in assessment,
• 02:16and
• 02:17certainly the EPAs.
• 02:19And,
• 02:20Katie, it is absolutely my
• 02:22pleasure to have you back,
• 02:24and I will say that
• 02:25Katie,
• 02:26has had
• 02:28a bit of laryngitis,
• 02:31but
• 02:32she is audible, so we
• 02:33didn't wanna miss this opportunity.
• 02:35So, again, Katie, thank you
• 02:37so much for
• 02:38coming back.
• 02:40Yes. Thank you so much,
• 02:41Penina.
• 02:43And, yes, for those of
• 02:44you who don't know me,
• 02:45this is not what my
• 02:46voice sounds like. I did
• 02:47not spend the last year
• 02:48away from Yale smoking.
• 02:50I just have some laryngitis
• 02:51right now. So,
• 02:53hopefully, you can understand what
• 02:54I'm saying, and I'll I'll
• 02:55try my best to enunciate
• 02:56as I'm talking.
• 02:58And thank you for that
• 02:59generous
• 03:00introduction.
• 03:03It's been quite a year
• 03:04for me. I just got
• 03:05my new med peds program
• 03:06approved, so,
• 03:08very exciting time.
• 03:10But I'm here to talk
• 03:11to you about a topic
• 03:13that I think is
• 03:14really interesting, and I I
• 03:15really do hope will generate
• 03:17some
• 03:18great conversation
• 03:20amongst this group,
• 03:21which is the topic of
• 03:23bias and assessment.
• 03:25As Penina mentioned,
• 03:27I work very closely,
• 03:29in the domain of untrustable
• 03:31professional activities.
• 03:33And whenever we're talking about
• 03:34assessment,
• 03:36cognitive biases and any bias
• 03:38at all becomes a really
• 03:40important thing to think about
• 03:42when you're incorporating new assessment
• 03:43paradigms into whatever program you're
• 03:45starting.
• 03:47And so,
• 03:48we'll learn as we talk
• 03:50that biases are just inherent
• 03:51in the assessment that we
• 03:52do, but it's really good
• 03:54to be aware of where
• 03:55they manifest and how they
• 03:56manifest
• 03:58in your
• 04:00assessment.
• 04:01So before we start,
• 04:04I'd like you to
• 04:07thank you.
• 04:08Mary Sarah just gave me
• 04:09some tea.
• 04:12Thank you very much.
• 04:14I want you to if
• 04:15you have access to it,
• 04:16if you're near a computer,
• 04:18pull up a recent eval
• 04:20that you did. It could
• 04:21be, MedHub. It could be,
• 04:24some other eval that you
• 04:26did recently on a trainee,
• 04:28and I this will be
• 04:29a reflective,
• 04:30exercise.
• 04:32In in just a second,
• 04:33I'll drop a worksheet in
• 04:34the chat, that we'll be
• 04:36using to kinda think about,
• 04:39this assessment.
• 04:41So I'm gonna start with
• 04:43a little bit of a
• 04:44riddle.
• 04:45Hopefully, not all of you
• 04:46have heard this one.
• 04:49A father and son are
• 04:50in a horrible car crash
• 04:51that kills the father.
• 04:53The son is rushed to
• 04:54the hospital.
• 04:56Just as he's about to
• 04:57go under the knife, the
• 04:58surgeon says, I can't operate.
• 05:00This boy is my son.
• 05:02How could this be?
• 05:07You can drop drop it
• 05:09in the chat or just
• 05:09say out loud what you
• 05:10think.
• 05:14The surgeon is his mom.
• 05:16Thank you for saying that.
• 05:17Yes.
• 05:19So, yes, that's correct, Lindsay.
• 05:22The surgeon could be his
• 05:23mother.
• 05:24But when they asked this
• 05:26riddle of college students a
• 05:27number of years ago,
• 05:29a good number of them,
• 05:31did not guess that. They
• 05:32guessed maybe,
• 05:33perhaps
• 05:34the father was actually a
• 05:36priest in the church,
• 05:38or some other things like
• 05:40that. They didn't assume that
• 05:41the surgeon was a woman,
• 05:43which I think points to
• 05:45the biases that we have,
• 05:47just in society.
• 05:50So our goals today are,
• 05:52by the end of the
• 05:52session, to understand the impact
• 05:54of assessment bias on learners
• 05:56and systems.
• 05:57And I'm gonna be showing
• 05:58a couple of papers that
• 06:00I think exemplify
• 06:01the various ways that bias
• 06:03shows up in our assessment
• 06:04world,
• 06:06to identify types of bias,
• 06:08in assessment and approaches to
• 06:09mitigate them.
• 06:11And then for all of
• 06:12us, and this is something
• 06:14I do constantly to reflect
• 06:15on your practice of assessing
• 06:17learners
• 06:17really in the in the
• 06:19interest of trying to reduce
• 06:20bias and assessment as much
• 06:22as we
• 06:23can. So what is bias?
• 06:26It's what many people refer
• 06:28to as our natural people
• 06:29preferences.
• 06:30You know,
• 06:32what we kind of prefer,
• 06:33the situations, the people, the
• 06:35other types of things in
• 06:37our lives.
• 06:38And it's often not on
• 06:40purpose.
• 06:41Our brain is wired to
• 06:43sift through information
• 06:45and look for patterns.
• 06:46And so a lot of
• 06:48the way that we interpret
• 06:49information in our world is
• 06:52completely below our level of
• 06:54awareness. We may not know
• 06:55that we're sorting through information
• 06:57in that way on a
• 06:58conscious level.
• 07:00But, unfortunately,
• 07:02the way we sift through
• 07:03all this information,
• 07:05affects our perceptions.
• 07:07It affects our perceptions of
• 07:08learners' knowledge, their ability,
• 07:10and their readiness for independent
• 07:12practice.
• 07:13So this thing that we
• 07:14sort of had as a
• 07:15evolutionary
• 07:17thing to help us sort
• 07:18through the millions of data
• 07:20points we have to process
• 07:22on a daily basis
• 07:23can actually make us,
• 07:25perceive information differently,
• 07:28depending on our background and
• 07:29our our prior experiences.
• 07:34And I think this can
• 07:35be a really uncomfortable thing
• 07:36for us to talk about.
• 07:37You know? I I think
• 07:39we all agree as as
• 07:41educators.
• 07:42Our goal is to be
• 07:44free of bias. We're trying
• 07:45to reduce bias. And I
• 07:47think intrinsically, we all believe
• 07:49we're fair. We're unbiased.
• 07:51We treat all trainees the
• 07:52same.
• 07:55And part of that is
• 07:56because of, the way our
• 07:58mind protects itself. You know?
• 08:01It's hard for us to
• 08:02embrace that. In fact, we
• 08:04may have unrecognized
• 08:06beliefs that,
• 08:09like, contribute to bias.
• 08:12And, also,
• 08:13the way that we've been
• 08:14brought up or our cultures
• 08:16can affect our preferences and
• 08:18our beliefs as well.
• 08:20So, it can be really
• 08:21hard for us to,
• 08:23look outside of that because
• 08:24it creates a dissonance, right,
• 08:26where we have this belief
• 08:27that we're unbiased people and
• 08:29we're trying our best to
• 08:30be unbiased, and yet
• 08:31we have these unconscious,
• 08:33ways of interpreting information
• 08:35that affects our assessment in
• 08:37a biased way.
• 08:40And like I mentioned before,
• 08:41this is just the way
• 08:43that our brain has adapted.
• 08:45We have millions and millions
• 08:47of data points that come
• 08:48into our brains at one
• 08:49time, sights, smells,
• 08:52sounds, those types of things.
• 08:54And our brain couldn't process
• 08:56it unless it sorted the
• 08:58information in a way that
• 08:59it could.
• 09:00So the categories that we
• 09:02form in our mind are
• 09:03sort of an intellectual shorthand.
• 09:06And one of the things
• 09:07that we think a lot
• 09:08about in clinical reasoning in
• 09:10in regards to this is
• 09:11what are called system ones
• 09:13and system two thought processes.
• 09:15So system ones are,
• 09:18fast intuitive emotional things where
• 09:20we skip a lot of
• 09:21steps in order to arrive
• 09:22at a at a decision
• 09:24where system two are slow,
• 09:25conscious, and effortful.
• 09:27And I think as physicians,
• 09:30we tend to avoid the
• 09:31system two reasoning unless we
• 09:33unless we really,
• 09:34have to because our work,
• 09:37involves us having to make
• 09:38a lot of decisions really
• 09:39quickly.
• 09:41And so it can be
• 09:41very hard.
• 09:43It could be a really
• 09:43big what we call cognitive
• 09:45burden
• 09:46to, like, actually go through
• 09:47the steps and thinking through
• 09:49in an effortful way
• 09:51about, you know, why is
• 09:52this person performing in this
• 09:53way? Why am I seeing
• 09:54the things I'm seeing?
• 09:57And bias manifests itself many
• 10:00different ways in the learning
• 10:01environment.
• 10:03I definitely, as a med
• 10:04peds person, I'm so biased
• 10:06towards med p people. I'll
• 10:07I'll admit it outright. So
• 10:09I have this affinity bias
• 10:10towards people who are like
• 10:11me. I think we can
• 10:12all think of, you know,
• 10:14those of us who are
• 10:15subspecialists. You know, you may
• 10:17be more have a more
• 10:18affinity to a student who
• 10:19says that they want to
• 10:21do the specialty that you
• 10:22wanna do.
• 10:24Perception biases or of stereotypes
• 10:27or assumptions about groups of
• 10:28people without thinking about the
• 10:30individual in front of you.
• 10:32Halo effect comes up a
• 10:33lot,
• 10:35in medical education where you
• 10:37have this projection
• 10:39of positive qualities onto people
• 10:41without actually looking deeper into
• 10:43the behaviors that they're showing,
• 10:45thinking about, you know, whether
• 10:46or not they're actually showing
• 10:47the behaviors that they need
• 10:49to be a competent physician.
• 10:51And then confirmation bias, which
• 10:53I think those of us
• 10:54who work clinically probably has
• 10:56know what that means, which
• 10:57is, you know, like, we
• 10:59kind of already have a
• 10:59preexisting notion of what is
• 11:01going on, and we we
• 11:02look for confirmation for that
• 11:04and ignore pieces of information
• 11:06that are in conflict with
• 11:07that.
• 11:10And bias manifests itself in
• 11:12training assessment a lot.
• 11:15And you'll see in this
• 11:16pyramid that, like, actual human
• 11:18and cognitive factors are a
• 11:19really big component
• 11:21of bias as it's the,
• 11:22like, base of bias.
• 11:25So,
• 11:27the in the learning environment,
• 11:30if you are teaching in
• 11:32a certain way,
• 11:34you may be teaching in
• 11:35a way, for example, if
• 11:36you're doing preclinical
• 11:37lectures,
• 11:39that's not applicable to people
• 11:40of different backgrounds, or you
• 11:42may have a assessment instrument
• 11:45where bias is built into
• 11:46the instrument itself where the
• 11:48language is,
• 11:51guiding people towards a more
• 11:52biased out view, whereas you're
• 11:54not anchoring things in actual
• 11:56behaviors and objective data.
• 11:58And then there's the implicit
• 12:00bias of clinical supervisors.
• 12:02And one of the things
• 12:03that we know about the
• 12:03type of assessment I do,
• 12:03which is assessment I do,
• 12:05which is entrustable for professional
• 12:07activities is
• 12:09the less objective the assessment,
• 12:11the more prone it is
• 12:13to bias. So things like
• 12:15multiple choice questions,
• 12:17there's actually a little bit
• 12:18of objectivity to that. You
• 12:19know, you can you can
• 12:20build it in a way
• 12:21that there's internal validity
• 12:24and reliability
• 12:25on how people respond to
• 12:26those questions.
• 12:28Whereas in the real clinical
• 12:29workplace, when you have an
• 12:30assessor looking at somebody doing
• 12:32clinical work and deciding how
• 12:34much supervision they need, that
• 12:36is a very subjective process,
• 12:38and that is really vulnerable
• 12:40to, clinical supervision bias
• 12:43because we each have our
• 12:45own world perceptions
• 12:46of what is the proper
• 12:48way to do things,
• 12:49and we can sometimes bring
• 12:50that to the table.
• 12:53Any questions before I move
• 12:55on to some of the
• 12:55evidence in the literature?
• 13:00Also using an excuse
• 13:01to have some tea.
• 13:09Okay. Great.
• 13:12So I'm just gonna go
• 13:13through a couple papers that
• 13:14I just found incredibly intriguing
• 13:17and illustrative
• 13:18of the bias that exists
• 13:20in assessment in,
• 13:22various domains.
• 13:24So,
• 13:25this one, I thought was
• 13:26a really fascinating paper. It
• 13:28came out about a year
• 13:29and a half ago,
• 13:31in the family medicine literature
• 13:33in, academic medicine.
• 13:36And, essentially, they did a
• 13:37very simple experiment.
• 13:40They,
• 13:41filmed two patient encounter videos,
• 13:44one with a male trainee,
• 13:45one with a female trainee.
• 13:47They were scripted videos, so
• 13:49the trainees
• 13:50said the exact same thing,
• 13:52and the patients said the
• 13:53exact same thing in both
• 13:54videos.
• 13:56And then they
• 13:57for each of those videos,
• 13:58they split them into two
• 14:00versions.
• 14:01They were exact same video,
• 14:03but one version said, this
• 14:05is a video to assess
• 14:06a learner who is above
• 14:08average,
• 14:09or this is a video
• 14:11who to assess a learner
• 14:12who is below average.
• 14:13So total four videos,
• 14:16two male, two female,
• 14:17one above, one below, one
• 14:19above, one below.
• 14:22And then they had seventy
• 14:23faculty observers,
• 14:25who are randomized to one
• 14:27of four videos.
• 14:28So they saw the prompt.
• 14:30This is to assess somebody
• 14:31who is up above,
• 14:33average, and then they watch
• 14:35the male or female video,
• 14:36for example.
• 14:38And what they found was,
• 14:41when they use the word
• 14:43below average as a prompt,
• 14:45the,
• 14:46faculty raters significantly rated those
• 14:48people below
• 14:50people who had above average
• 14:52as a prompt.
• 14:53And this isn't a really
• 14:55good example of what's called
• 14:56the halo and her horn
• 14:58effect,
• 14:59where if somebody is labeled
• 15:01with something,
• 15:02this is a difficult learner,
• 15:04or this this resident is
• 15:06stellar.
• 15:07It biases the viewpoint of
• 15:09the assessor to believe that
• 15:11they are already at that
• 15:12baseline.
• 15:14And what they conclude from
• 15:16this paper is that a
• 15:17single evaluative word above or
• 15:19below
• 15:20was associated with systemic differences
• 15:22assessment score.
• 15:24And I think we can
• 15:25all think of a situation
• 15:26in which we have
• 15:28inherited a learner who's already
• 15:30been labeled with something like
• 15:31this is somebody who's struggling.
• 15:33This is somebody who's stellar,
• 15:36and how that might affect
• 15:37the way that we think
• 15:38about them without really looking
• 15:40at the objective because because
• 15:41there was nothing different about
• 15:43these videos.
• 15:44When they did sub analysis
• 15:45on male versus female, they
• 15:46did not find any differences
• 15:48in case you're curious about
• 15:49that. But don't worry. If
• 15:50I get, you know to
• 15:51male, female pretty soon.
• 15:54Alright. I'm
• 15:57gonna mute you.
• 16:00Like, hopefully, I can.
• 16:02Okay. There we go.
• 16:04So let's look at gender.
• 16:06So this was a paper
• 16:07that came out in emergency
• 16:08medicine literature,
• 16:10and they were really curious
• 16:12about the effect of gender
• 16:13on assessment. So they looked
• 16:15at a lot a lot
• 16:17of data. They did a
• 16:18qualitative
• 16:19analysis of comments
• 16:20across the five programs. So
• 16:22that was
• 16:23over, you know, almost three
• 16:24hundred residents, ten thousand comments.
• 16:29And, they found,
• 16:30when they were able to
• 16:31categorize things that men were
• 16:33more likely to receive specific
• 16:35feedback, so things that they
• 16:37could actually act on,
• 16:39receive competency based feedback.
• 16:42So rather than talking about
• 16:43their character, they were their
• 16:45competency was commented
• 16:47on and were more likely
• 16:48to be rated at above
• 16:49expected performance
• 16:51irrespective of the faculty's gender.
• 16:55In women residents,
• 16:56comments about low skill level
• 16:59were very,
• 17:00commonly associated with comments about
• 17:02their confidence,
• 17:05which I think we for
• 17:06those of us who are
• 17:06female physicians, that sounds very
• 17:08familiar,
• 17:09and, is not really a
• 17:11competency,
• 17:12comment. It's not really a
• 17:13commentary on
• 17:15how somebody can do something
• 17:17as a physician,
• 17:18versus men
• 17:20who received comments when those
• 17:22who were at a low
• 17:23skill level received comments including
• 17:26actionable items.
• 17:29And then lastly,
• 17:31interestingly, they found that women
• 17:32faculty
• 17:33were more likely to rate
• 17:35residents as performing low levels.
• 17:37So women were harsher graders
• 17:40than men,
• 17:41as faculty.
• 17:44So this shows the same
• 17:46data,
• 17:47visually,
• 17:48but you'll see, like, the
• 17:50gender biases in either direction.
• 17:52So men were more likely
• 17:54to be rated, above,
• 17:57performance level, whereas women were
• 17:59more likely to be related
• 18:00below.
• 18:01And things like adaptability,
• 18:03confidence, and assertiveness with treatment
• 18:05were commonly women, oriented
• 18:08comments.
• 18:11Here's another paper that came
• 18:13out in JGIM, which is
• 18:14the journal of general internal
• 18:15medicine in twenty nineteen,
• 18:17and they looked at, clerkship,
• 18:20comments
• 18:21and,
• 18:22separated them by, URAM versus
• 18:25not and gender.
• 18:26So here's how they separated
• 18:28out,
• 18:29women and men looking at
• 18:31honors versus past grades.
• 18:33And they found that
• 18:36women who received honors grades
• 18:38were more likely to get,
• 18:40character comments, like wonderful,
• 18:42empathetic,
• 18:43fabulous,
• 18:45whereas men were more likely
• 18:46to get relevant,
• 18:49modest,
• 18:50humble,
• 18:51those types of comments, which
• 18:53is interesting.
• 18:55And similarly, when they separated
• 18:57out URIM versus nonURM comments,
• 19:01people who received honors who
• 19:02are nonURM,
• 19:04got top stellar excellent.
• 19:07Whereas people who received,
• 19:09not many people received honors
• 19:11who are URM, but which
• 19:12we'll get to in a
• 19:13second too,
• 19:15got things like native Spanish
• 19:17cultural,
• 19:19in their comments,
• 19:20which I think is really
• 19:21interesting.
• 19:24Lastly, I'll go through, the
• 19:26paper that really got me
• 19:28quite interested in assessment,
• 19:30thinking about, like, what are
• 19:32the downstream effects of bias
• 19:35in assessment?
• 19:36Like, how is this actually
• 19:37impacting learners,
• 19:39in a way that affects
• 19:40their career development?
• 19:42So I think this was
• 19:43one of the most seminal
• 19:44papers on the topic,
• 19:46and I encourage you guys
• 19:47to check it out because,
• 19:48it I think it was
• 19:50an exercise in humility for
• 19:51this medical school.
• 19:53This was published by UCSF
• 19:56twenty eighteen,
• 19:58and,
• 19:59their paper is titled how
• 20:00small differences in assess clinical
• 20:02performance amplify to large differences.
• 20:05So what they noticed as
• 20:07medical school
• 20:09was that,
• 20:10despite recruiting
• 20:12more underrepresented
• 20:13minorities in medicine,
• 20:15they were not finding that
• 20:16their URMs
• 20:18were receiving,
• 20:19as many,
• 20:21honors and awards,
• 20:22and they were not getting
• 20:23into competitive
• 20:24specialties
• 20:25at the rate that their
• 20:27white,
• 20:28students were.
• 20:29And they they
• 20:31they were very surprised by
• 20:32this because they assumed if
• 20:34they increase the proportion of
• 20:35URMs, the proportion of URMs
• 20:37going into competitive specialties would
• 20:39likewise increase.
• 20:41So they took a really
• 20:41honest look at their grading
• 20:43policies
• 20:44to try to understand why
• 20:46that was happening.
• 20:48So they did a study
• 20:49on med students at a
• 20:50single institution,
• 20:52and they found that grading
• 20:55consistently favored non students.
• 20:59And when they looked,
• 21:01deeper,
• 21:02they realized that the size
• 21:04and magnitude of the differences
• 21:05were incredibly small,
• 21:08but they made a big
• 21:09difference. So, for example, in
• 21:11order to receive an honors
• 21:13on a particular rotation,
• 21:15they might need to receive
• 21:16a certain average score on
• 21:18their assessment forms.
• 21:20And the difference between
• 21:21the, you know, whites non
• 21:23URM students
• 21:25and the URM students on
• 21:27average was, like, something like
• 21:29point two on a five
• 21:30point scale.
• 21:31But because of their grading
• 21:32policies of this proportion of
• 21:34the class gets honors,
• 21:36they were very explicitly,
• 21:39excluding students from getting honors
• 21:41who were only at, like,
• 21:43two or one points away
• 21:45on an average scale.
• 21:47So the size and magnitude
• 21:48of this, differences were incredibly
• 21:50small,
• 21:51but what it resulted in
• 21:53is that URMs received half
• 21:55as many honors grades,
• 21:56and URMs were three times
• 21:58less likely to be selected
• 21:59for honor society. And for
• 22:00society. And for any of
• 22:01us who are in residency
• 22:03recruitment,
• 22:04those things make a really
• 22:05big difference in helping an
• 22:06applicant really stand out when
• 22:08you're considering them for your
• 22:10residency program.
• 22:12And so they entitled this
• 22:13the amplification
• 22:14cascade.
• 22:15These tiny, tiny differences
• 22:17accrued over time in space
• 22:20to actually lead to larger
• 22:22differences in grades and selections
• 22:23for awards.
• 22:25And it allowed them to
• 22:27really look at the systems
• 22:28they had in place for
• 22:29grading
• 22:30and think about, like, what
• 22:32are the sources of bias
• 22:33that are happening here? What
• 22:35systems things have we created
• 22:37to,
• 22:38cause this bias? Do we
• 22:40have any cultural structural things?
• 22:42And what are our assessors
• 22:43at the frontline really looking
• 22:45at? Can we make more
• 22:46objective?
• 22:48So this was a really,
• 22:49I think, a really thought
• 22:50thought provoking paper for a
• 22:52lot of us.
• 22:53Really forced us to think
• 22:54honestly about what the impact
• 22:56of our assessment was.
• 22:59Okay.
• 23:01I see something in the
• 23:02chat.
• 23:05Okay.
• 23:06CME code.
• 23:07So
• 23:08I'm kinda curious.
• 23:11And,
• 23:12this can be actually
• 23:14to yourself.
• 23:15Have you seen bias bias
• 23:17manifest in the assessment of,
• 23:19learners where you work or
• 23:20teach?
• 23:22And how have you seen
• 23:23it?
• 23:24Actually, do do you guys
• 23:25feel comfortable talking about it?
• 23:26Have you guys seen bias
• 23:28manifest?
• 23:30I'd be kinda curious about
• 23:31the ways in which you
• 23:33have encountered it.
• 23:38I definitely agree with you,
• 23:40Katie, on the the horn
• 23:41effect.
• 23:43I feel like when we
• 23:45have trainees who are labeled
• 23:47as struggling or labeled as,
• 23:49not performing to the level
• 23:50of their peers, things that
• 23:51they do
• 23:52are looked through that lens.
• 23:53And I feel like somebody
• 23:55who has a label of
• 23:56being excellent,
• 23:57people are more likely to
• 23:58give them a pass for
• 23:59that and not focus in
• 24:00on it. So everything you
• 24:01do is more under a
• 24:03microscope, and then I think
• 24:04it's really hard to change
• 24:05that perception.
• 24:07So they're sort of,
• 24:09behind the eight ball for
• 24:11Yeah. For a long time
• 24:12and a really difficult situation
• 24:14to get out of.
• 24:16In my experience with those
• 24:17folks is they feel a
• 24:18little persecuted.
• 24:20They feel like they're under
• 24:21a microscope,
• 24:22and, it it's really hard
• 24:24to build trust with them
• 24:26again after they've gotten that
• 24:28label of being the struggling
• 24:29learner.
• 24:31Yeah.
• 24:32Any other thoughts?
• 24:35Katie, I I would just
• 24:37add that,
• 24:38one of the
• 24:40one of the issues is
• 24:42that we have a very
• 24:43short exposure to these, trainees.
• 24:45So, yes, that that makes
• 24:47this horn
• 24:49and hollow effect,
• 24:50truly amplify.
• 24:52In in my program, which
• 24:54is a small program and
• 24:55we work with them closely
• 24:56for two years,
• 24:58I think that helps
• 25:00mitigate mitigating that kind of
• 25:02a bias because you you
• 25:03do have the chance to
• 25:04readjust and reappreciate these residents
• 25:06over long period of times.
• 25:08But I agree when you
• 25:09work with them for a
• 25:10week,
• 25:11it's hard not
• 25:13to fall into the trap.
• 25:15Yeah.
• 25:17And I think, you know,
• 25:19an argue argument can be
• 25:20made for
• 25:22when you're working with people
• 25:24for a short period of
• 25:25time.
• 25:26Like, is it useful to
• 25:27pass along information about what
• 25:29they're working on so that
• 25:30even though you have a
• 25:31short period of time, you
• 25:32can continue the work of
• 25:33the previous faculty who is
• 25:35working with them?
• 25:37And it's a double edged
• 25:38sword. Like,
• 25:39by good communication
• 25:40and clear clearness of goals,
• 25:42you can actually share with
• 25:43them things that you're working
• 25:45on together so they could
• 25:46continue to grow. But it
• 25:48also sort of sets them
• 25:50up for bias of this
• 25:51label of, like, hey. They're
• 25:53struggling.
• 25:54So when I do do
• 25:56educational handoffs, I try to
• 25:57be intentional about being objective,
• 26:00about the specific goals that
• 26:01we're working on and the
• 26:02domains
• 26:03that we're working on with
• 26:05the understanding that, like, you
• 26:07know, it doesn't make them
• 26:07a bad learner. They're just
• 26:09working on something.
• 26:11Any other thoughts?
• 26:13I'm just well, I'm not
• 26:15answering your question, but I'm
• 26:16wondering if that sort of
• 26:18opens up the question of
• 26:20is there
• 26:21destructive bias, and is there
• 26:23productive bias?
• 26:25Yeah.
• 26:26Great question. What do you
• 26:28think?
• 26:29Yeah. Gary, I I was
• 26:31gonna I'm gonna tag on
• 26:32to that because I do
• 26:33think when we have
• 26:35people that we are either
• 26:36formally or informally remediating,
• 26:40Could that be constructive, Gary?
• 26:41And and how do we
• 26:42benefit
• 26:44from the attention and the
• 26:45work we wanna do while
• 26:46not having them feel persecuted?
• 26:48I feel like that is
• 26:49been a real challenge because
• 26:50we're I I think my
• 26:52frame is we're providing more
• 26:53resources. Our program's trying to
• 26:55support
• 26:56them. Their frame may be
• 26:57one of persecution. That that's
• 26:59really interesting. I don't know,
• 27:00Gary, if you have ways
• 27:01that you've dealt with it,
• 27:02but I think it can
• 27:02be really challenging because other
• 27:04people here within the system,
• 27:05right, they're like, oh, that's
• 27:06the resident that's, you know,
• 27:07getting remediated or that's been
• 27:09a problem.
• 27:13Yeah. I mean, I think
• 27:14that for me in in
• 27:15in the residency program, I
• 27:17think controlling the conversation
• 27:19in the most productive way
• 27:20is is the best approach
• 27:22because I I think if
• 27:23you
• 27:24ignore these things, they become
• 27:25gossip, which is not productive
• 27:27in any way. And we
• 27:28we know ultimately there are
• 27:30going to be learners that
• 27:32that have struggles.
• 27:34And and
• 27:36I think we have to
• 27:36confront it and be honest
• 27:38about it in order to
• 27:39to to make them better
• 27:40off. I think also, you
• 27:42know, with, with
• 27:43residents
• 27:44who are, you are IMs
• 27:45just by calling them, you
• 27:46are IMs we're biasing.
• 27:49Yeah.
• 27:50And, and that has, that
• 27:52has again, protect,
• 27:53you know, productive and, and
• 27:56potentially destructive qualities as well.
• 27:59And trying to navigate that
• 28:00whole conversation too is is
• 28:02is a challenging one,
• 28:04because there are realistic,
• 28:07very important issues that we
• 28:09have to deal with
• 28:10that that create that that
• 28:12paradigm, but
• 28:14but there there can be
• 28:15destructive components as well. That's
• 28:17that's all. Katie, while we
• 28:18talk about that, I'd be
• 28:19really interested if you have
• 28:20any comments on foreign medical
• 28:22graduates as well, because I
• 28:23think that's another one Lindsay
• 28:24and I and some others
• 28:25have sort of tried to
• 28:26navigate through different cultures, different
• 28:28backgrounds in their training.
• 28:29Yeah. I mean, I think,
• 28:32sometimes it can it can
• 28:33be hard too if there's,
• 28:35like,
• 28:36multiple domains of competency
• 28:39affected,
• 28:40and you're trying to honestly
• 28:41assess.
• 28:42So I'm actually
• 28:43the reason why I'm I'm
• 28:44actually physically in New Haven
• 28:46right now, by the way,
• 28:47guys.
• 28:48The reason why I'm here
• 28:49is I'm I'm co teaching
• 28:50the ACGME,
• 28:52developing competencies and assessment
• 28:54course. And we are talking
• 28:56a lot about,
• 28:57like, really setting firm objective
• 29:00goals and, like, outcomes that
• 29:02you're trying to achieve with
• 29:03learners. Because the more objective
• 29:05you are, the more competency
• 29:06based you are, the less
• 29:07prone to bias you're going
• 29:08to be. You know, like,
• 29:10how like, is this thing
• 29:11observable that you're trying to
• 29:12achieve? Is it something that
• 29:14you could actually measure?
• 29:16Because that will help to
• 29:17reduce bias.
• 29:18The thing I would just
• 29:19emphasize is that,
• 29:22unfortunately,
• 29:23because of the nature of
• 29:25the work we do and
• 29:26the subjectivity
• 29:27of the work that we
• 29:27do, bias is a reality
• 29:30of assessment. It is just
• 29:31going to be part of
• 29:32assessment always.
• 29:34And,
• 29:35part of our jobs as
• 29:37people who process that assessment
• 29:39information
• 29:40is that we need to
• 29:41recognize that it exists
• 29:43and really think if if
• 29:45if a bias is affecting
• 29:47what we're seeing in front
• 29:48of us,
• 29:50because we will never be
• 29:51able to get rid of
• 29:51it. There's always gonna be
• 29:53that attending who's a harsh
• 29:54grader.
• 29:56But if you know they're
• 29:56a harsh grader, you can
• 29:58kind of take that into
• 29:59stride, or there's always gonna
• 30:01be residents who struggle. You
• 30:03know?
• 30:04But,
• 30:05like, you can control the
• 30:06language of which by which
• 30:08you describe that individual in
• 30:10your training program.
• 30:11So there's a lot of
• 30:12things that you can do
• 30:13to try to mitigate,
• 30:15bias.
• 30:16I don't think I talked,
• 30:18answered your question about,
• 30:20like, foreign medical grads. Was
• 30:22there something specific you were
• 30:24asking about, Mark?
• 30:26I guess I'll I'll I'll
• 30:28Lindsay and I had done
• 30:29a workshop on this way
• 30:30back when, but just, like,
• 30:31when they're from a diff
• 30:34like, our milestones
• 30:35maybe
• 30:36might not
• 30:37meet where their culture is
• 30:39in terms of interprofessional education,
• 30:41in terms of some of
• 30:42those things. So it's like,
• 30:44you know, again, there's
• 30:45a bias, but I think
• 30:46what Gary was hitting on,
• 30:47it might be
• 30:48reasonable to just flag that
• 30:50bias and say that that
• 30:51bias is probably something we
• 30:52need to
• 30:54explore further.
• 30:55You know, power dynamics with
• 30:56nursing, I guess, is the
• 30:57point the one that I'll
• 30:58put out there that is
• 30:59quite different
• 31:00other countries.
• 31:01Yeah.
• 31:02But I would say if
• 31:03they're hoping to practice in
• 31:04the US, they will need
• 31:05to learn
• 31:06how to navigate
• 31:07those relationships. So always going
• 31:09back to your,
• 31:11you know, your residency milestones
• 31:14as a, like,
• 31:15homing point of, like, where
• 31:17you're trying to achieve because,
• 31:19understandably, they might not have
• 31:20been brought up in that
• 31:21culture of,
• 31:23this is the collaborative way
• 31:24in which we work with
• 31:25our nursing staff in the
• 31:26US.
• 31:27So, yeah,
• 31:29learning learning opportunity there, but
• 31:31I hear what you're saying.
• 31:34Okay.
• 31:35So how can we move
• 31:36forward?
• 31:37This is a hard thing.
• 31:39You know, I think,
• 31:41whenever we hear the word
• 31:42bias, like, you immediately kind
• 31:44of like, oh, I'm not
• 31:45bad. I promise I'm not,
• 31:46like, a bad person.
• 31:48It was always thought about,
• 31:49like, as aberrant or
• 31:51intentional, like, you were trying
• 31:53to hurt someone by being
• 31:54biased, but it's sort of
• 31:56normative, unconscious, and largely unintentional.
• 32:01So, you know,
• 32:03social cognitive theory theory kind
• 32:05of dictates that,
• 32:07personal experiences become hardwired into
• 32:09cognitive function. So we have
• 32:11our intentions,
• 32:12our wiring kind
• 32:14of re rejiggers everything, and
• 32:16then we translate that into
• 32:18actions.
• 32:20And it can be really,
• 32:22really hard, I think, to
• 32:24change the way we see
• 32:25things because it's so hardwired
• 32:27into our brain. So
• 32:29how many people think that
• 32:31a is darker than b?
• 32:36I see a is darker
• 32:37than b. I know you
• 32:38know the I know you
• 32:39know this is a trick
• 32:40question.
• 32:41But when you line them
• 32:42up with pars,
• 32:44they're the exact same color.
• 32:46I still have a hard
• 32:47time seeing that b is
• 32:49lighter than a. My brain
• 32:51is so wired to see
• 32:52that shadow
• 32:54that I actually see them
• 32:55as this as a different
• 32:56color even though I know
• 32:57they're the same color. So
• 32:59it can be really hard
• 33:00to rewire our brain.
• 33:02And,
• 33:03unfortunately,
• 33:04we as physicians
• 33:05and we as academic clinicians,
• 33:08we have all the stresses
• 33:10that make us much more
• 33:11prone to bias. So things
• 33:13that make people more prone
• 33:14to bias are stress, multitasking,
• 33:17time constraints,
• 33:18need for closure, which I
• 33:20have a lot of, and
• 33:21fatigue. Like, this is all
• 33:22things that I,
• 33:24still have in my daily
• 33:25life,
• 33:27now many years post residency.
• 33:28So very much still have
• 33:30all those stressors that make
• 33:32me wanna make those cognitive
• 33:33leaps and just close-up that
• 33:35encounter or close-up that evaluation
• 33:37form as quickly as possible
• 33:39so I can move on
• 33:39with other things that I
• 33:40have to do.
• 33:43So like I said, I
• 33:45I have come to the
• 33:46conclusion myself as somebody who
• 33:48does assessment on a daily
• 33:49basis.
• 33:51We can't eliminate bias completely,
• 33:54but we can reshape our
• 33:56implicit attitudes and try our
• 33:58best in various ways to
• 34:00curb their effects on our
• 34:01assessments.
• 34:02And objectivity,
• 34:03self reflection, and external feedback
• 34:06can help.
• 34:08So couple things to think
• 34:10about in your own programs
• 34:11or in, whatever locus of
• 34:13control you guys have is,
• 34:16learn to recognize inferences.
• 34:18Right now, we're doing the
• 34:19ACGME course, and there have
• 34:20been a lot of instances
• 34:22where I'm like, why do
• 34:23you think that person is
• 34:24not confident?
• 34:25You know, like, what why
• 34:26are we making those inferences
• 34:28about that person?
• 34:30And call them out when
• 34:31they're happening
• 34:32and really try to transform,
• 34:35the language that we're using
• 34:36to describe learners into behaviors
• 34:38as much as possible,
• 34:40and avoid describing their personality.
• 34:43It's all well and good
• 34:44that that physician seems sweet
• 34:46or nice, but, like, what
• 34:47are the behaviors we're seeing
• 34:49that make us feel like
• 34:50they're able to form a
• 34:52good relationship with patients?
• 34:56And for,
• 34:58assessment instruments,
• 34:59really thinking about what language
• 35:01are we using to guide
• 35:02our assessors,
• 35:03you know, prioritize
• 35:05observation,
• 35:06use criterion referenced,
• 35:09scales.
• 35:10So really describing the behaviors
• 35:12that we're looking for and
• 35:13using competency based tools or
• 35:15even checklists. Like, behavioral checklists
• 35:17can be a really objective
• 35:19way to assess learners.
• 35:22I have found in my
• 35:23EPA work,
• 35:25one of the things we
• 35:26look at a lot with
• 35:27EPAs is reliability.
• 35:29And we have found in
• 35:31order for us to achieve
• 35:32reliability,
• 35:33that validity that we're looking
• 35:35for
• 35:36to reduce the interrelator
• 35:38reliability
• 35:39of assessors,
• 35:41one of the things that
• 35:42we,
• 35:43do is just increase the
• 35:44number of observations.
• 35:46So, it might be that,
• 35:48you know, like, two observations
• 35:50isn't enough to reduce bias
• 35:52to a point where you
• 35:53feel assured what you're seeing
• 35:55is actually what's happening. You
• 35:56may need to double it.
• 35:58You may need to look
• 35:59more,
• 35:59from different
• 36:01perspectives to truly understand what's
• 36:02going on.
• 36:05We have our own role
• 36:07to play.
• 36:08So recognizing that we all
• 36:09have biases,
• 36:11in various ways due to
• 36:13our training or life circumstances,
• 36:16many other things. And
• 36:18I I have found it
• 36:19personally helpful to get feedback.
• 36:22So,
• 36:23one of the things I
• 36:24did as an exercise
• 36:25last year is I had
• 36:26somebody else read my letters
• 36:28of recommendation,
• 36:30for trainees that I've written
• 36:32in the past, give me
• 36:33feedback on whether or not
• 36:34they seem like they were
• 36:35biased.
• 36:36And that was really illuminating
• 36:38to me. I didn't realize
• 36:39I was using certain language
• 36:41to describe female trainees
• 36:43that was different from my
• 36:44male trainees.
• 36:46Reviewing your assessments can also
• 36:48be really helpful or asking
• 36:50a trusted colleague,
• 36:52to,
• 36:53review them or using tools,
• 36:55which we'll do as a
• 36:56exercise later.
• 36:59And practice constructive uncertainty. So
• 37:02observe yourself in action and
• 37:03more thoughtfully consider your perspectives,
• 37:06which I'll talk about in
• 37:06the next slide.
• 37:08Inhabit that awkwardness and discomfort
• 37:10because I think talking about
• 37:12these things explicitly can be
• 37:13really hard.
• 37:15But I think acknowledging that
• 37:16we have biases is actually
• 37:18really reassuring to our trainees
• 37:20that they know that we're
• 37:21thinking about that,
• 37:22and we're taking that into
• 37:24account when we're talking with
• 37:25them.
• 37:27And, you know, engage people
• 37:28who are different, which I
• 37:29think this pediatrics department does
• 37:31a really good job thinking
• 37:33about bias and equity and
• 37:35those types of things,
• 37:37and, like, be really appreciative
• 37:39of those different viewpoints.
• 37:42This is a framework that
• 37:43I found pretty helpful.
• 37:45It's called pause.
• 37:47So paying attention to what
• 37:49you're assessing in the moment,
• 37:51really thinking about what are
• 37:52the behaviors of the things
• 37:53that I'm looking at.
• 37:55Acknowledge your own reactions or
• 37:57judgments. Just because somebody's doing
• 37:59something differently than you would
• 38:00doesn't necessarily mean they're doing
• 38:02something that's bad for patients
• 38:04or for patient care.
• 38:06Understanding,
• 38:07other viewpoints,
• 38:09and trying to be as
• 38:10objective as possible to frame
• 38:12your assessment.
• 38:14You know, what are the
• 38:15behaviors I'm looking for? I
• 38:16understand that I feel like
• 38:18this person's not showing confidence,
• 38:20but how can I sit
• 38:21in front of this person
• 38:22and tell them that in
• 38:23an objective way
• 38:24so that they can
• 38:28execute an assessment that minimizes
• 38:30bias? So
• 38:32one exercise I've done too
• 38:34is looking for,
• 38:37gender bias and by assessments.
• 38:40So this is an assessment
• 38:41I wrote many years ago
• 38:43on a learner who has
• 38:44long since graduated.
• 38:46So
• 38:46and has been altered
• 38:48to not reveal anything about
• 38:50this individual.
• 38:52But I wrote this and,
• 38:54decided to see if I
• 38:55could find evidence of bias
• 38:56in it. So I very
• 38:58much liked working with them.
• 38:59They worked hard.
• 39:02They were a great team
• 39:03player.
• 39:03Appreciate the effort they went
• 39:05into caring for their patients.
• 39:07They're very open to feedback.
• 39:09They also worked hard on
• 39:10improving the cardiac exam.
• 39:13Often their differentials are based
• 39:14on previously established diagnoses or
• 39:16a limited list of other
• 39:18possibility
• 39:18possible alternatives.
• 39:23So I put it into
• 39:24this gender bias calculator just
• 39:25to see what would happen,
• 39:27which we'll do as an
• 39:28exercise in just a moment
• 39:29so we still have time.
• 39:32And I realized that I
• 39:34used hard worker and worked
• 39:35a lot to
• 39:38associate with a trainee.
• 39:40Do you guys think that,
• 39:42this, like,
• 39:43or sorry, was associated with
• 39:45female female characteristics?
• 39:47And I didn't realize I
• 39:48said hard worker so much
• 39:50in relation to
• 39:51female trainees, but when I
• 39:53looked back at my other
• 39:54assessments, I said that a
• 39:55lot. And I didn't say
• 39:57that males, trainees were hard
• 39:59workers at nearly the same
• 40:00rate. It made me think
• 40:02about, like, why why do
• 40:03I think this person like,
• 40:05why do I think women
• 40:06are more likely to be
• 40:07labeled as a hard worker?
• 40:09And it sort of just
• 40:10made me be more thoughtful
• 40:11about the words I was
• 40:12using in my assessment, which
• 40:14I found to be a
• 40:14helpful exercise for myself.
• 40:17So I dropped a worksheet
• 40:19in the chat,
• 40:21and I want you guys
• 40:22to try if you can.
• 40:23If you're at a computer,
• 40:26I'll drop it in again.
• 40:28I put in the
• 40:31link to,
• 40:33the
• 40:35gender calculator,
• 40:37in the worksheet.
• 40:39Just pull up an old
• 40:40eval that you had in
• 40:42MedHub.
• 40:43Could be something recent, could
• 40:45be from several years ago,
• 40:46and pop it in there.
• 40:47And just see if you
• 40:49have any
• 40:50more female or male biased
• 40:52words,
• 40:53just as an exercise.
• 40:55So I'll give you a
• 40:55couple minutes to do that.
• 40:59Thanks, Lindsay.
• 41:00And after that, we'll just
• 41:01do a couple of minutes
• 41:02for reflection.
• 42:31Folks
• 42:31doing? You need a little
• 42:33bit more time?
• 42:36Little bit more. K.
• 42:41Hello, doctor Johnson.
• 42:45Hi. Sorry I was late.
• 43:20Katie, I'm gonna I'm gonna
• 43:21need to jump off in
• 43:22a moment, but I I
• 43:23just was curious,
• 43:24not on my question on
• 43:25the chat. We can discuss
• 43:26that later maybe. But, in
• 43:28terms of the calculator,
• 43:30like, if we're seeing even
• 43:31numbers of items in each
• 43:33of those words, if I
• 43:34post a
• 43:35a recent letter, is that
• 43:37that's good? Or
• 43:39I I I've I def
• 43:41this question has come up
• 43:42a number of times when
• 43:43I've taught this session. I
• 43:44don't think the goal is
• 43:45necessarily,
• 43:46like, an even number or,
• 43:48like, not using any gender
• 43:50terms.
• 43:51I think, really, the goal
• 43:52is just just reflect on
• 43:53the words that you're using
• 43:55and see if there's any
• 43:56patterns that are emerging in
• 43:58the way that you,
• 44:00use those words for assessment.
• 44:02Okay.
• 44:02And and do you agree
• 44:03with the wording? Because I
• 44:05I'm really surprised to me
• 44:06in in one of my
• 44:07letters, mentor,
• 44:09was, like, the word that
• 44:10I kept getting female associated
• 44:12words for, which I Oh,
• 44:13interesting.
• 44:14Didn't really frame mentor as
• 44:15being a gender biased word.
• 44:18Yeah. I don't know. Like,
• 44:21like,
• 44:22I I wonder if leader
• 44:23would be a word that
• 44:25would be more male gendered.
• 44:28I think they use this
• 44:30from a whole bunch of
• 44:30different datasets, and they've gone
• 44:32through several different versions of
• 44:34this gender bias calculator,
• 44:36based on a whole bunch
• 44:37of data inputs.
• 44:38But I don't think it's
• 44:38been titrated to, like, academic
• 44:41medicine.
• 44:42So it may be that,
• 44:44mentor in different domains means
• 44:46something different from the way
• 44:47that we interpret it.
• 44:50I'm curious about your
• 44:53oh, yes. So for the
• 44:55validity evidence stuff,
• 44:57having,
• 44:58when you look across validity
• 44:59evidence studies and EPAs,
• 45:02having more assessors is a
• 45:04double edged sword because the
• 45:06greatest source of,
• 45:08variability in assessment almost all
• 45:10come from assessors,
• 45:12rather than the learner themselves,
• 45:13which is not what we're
• 45:14going for.
• 45:15So,
• 45:17having a lot of different
• 45:18assessors,
• 45:19but actually just having more
• 45:20assessments
• 45:22tends to be quite helpful.
• 45:24We just put out a
• 45:25paper about a year ago
• 45:26on, like, the kind of
• 45:28a conglomeration
• 45:29study of all the core
• 45:31EPA for entering residency,
• 45:33pilot data. And even at
• 45:35at, schools where they had,
• 45:37like, dedicated coaches who were
• 45:39trained
• 45:40on how to assess,
• 45:41they still did not achieve
• 45:43really good,
• 45:44inter rater reliability.
• 45:47So I I personally believe
• 45:49that,
• 45:50like,
• 45:51having many inputs from many
• 45:53assessors is always good, gives
• 45:55you lots of viewpoints on
• 45:56the learner.
• 45:57But,
• 45:59just recognize that there's probably
• 46:01gonna be a lot of
• 46:01variability between assessors.
• 46:03Hopefully, that answers your question.
• 46:10Alright. I I'll open it
• 46:11up to the greater group.
• 46:12Thanks for joining, Mark.
• 46:15Any thoughts or reflections from
• 46:16that exercise?
• 46:25Recognizing that talking about this
• 46:26stuff can be a little
• 46:27awkward.
• 46:35So, Katie, I had the
• 46:37same issue as Mark is
• 46:38that there were, like, twenty
• 46:40thing twenty
• 46:42or twenty five words that
• 46:43were listed as female oriented
• 46:46or female gender,
• 46:47and,
• 46:49ten of them were
• 46:51educator,
• 46:52education,
• 46:53or trainee.
• 46:54And so
• 46:56yeah.
• 46:58It must be in reference
• 46:59to, like, being a teacher,
• 47:01which is Right. Exactly.
• 47:03Yes.
• 47:06But one of the things
• 47:07that I found in my
• 47:08own letter review, like, I
• 47:09had I had a peer
• 47:10review my one of a
• 47:11couple letters of recommendation,
• 47:14and,
• 47:16I kept on alluding to
• 47:17this person being really wonderful.
• 47:20And,
• 47:21the person who reviewed my
• 47:22letter said, never in the
• 47:24letter did you actually call
• 47:25this person a leader. Would
• 47:27you call them a leader?
• 47:28And I'm like, absolutely. They're
• 47:29a leader. They're incredible.
• 47:31And I didn't use the
• 47:32word.
• 47:33So it just it made
• 47:34me really think about the
• 47:35language a little bit more
• 47:37carefully, which is what this
• 47:38is really all about.
• 47:41Any other thoughts or afflictions?
• 47:45I'm
• 47:46I'm kinda curious.
• 47:48What are the low hanging
• 47:49fruits here? Because it's just,
• 47:51you know, there's, like, big
• 47:52systemic issues, obviously.
• 47:54There's a lot of nuance.
• 47:56Like, to you, what are
• 47:57the the the smallest adjustments
• 48:00that that we can make
• 48:02apart from, like, you know,
• 48:03putting our our letters in
• 48:04these in these things
• 48:07to be better serve
• 48:08serving the trainees.
• 48:10And I did I I
• 48:11I also wanna make a
• 48:11point. I did think about,
• 48:13like, bias versus
• 48:15that other aspect, like, constructive
• 48:17I think, ultimately, bias is
• 48:19about intent. Right? Like, it's
• 48:21about a lack of intent,
• 48:22really. It's like you're not
• 48:23trying
• 48:24to be productive or destructive.
• 48:26You're it just exists.
• 48:28But what are the low
• 48:29hanging fruits here?
• 48:31To me, I think really,
• 48:34in either your own personal
• 48:35life or if you are
• 48:37in charge of any faculty
• 48:38development or assessment forms,
• 48:41keeping things as objective
• 48:43as possible.
• 48:45And that that
• 48:46forces us as
• 48:48educators, as supervisors
• 48:50to actually form words around
• 48:52the things that we're looking
• 48:53for objectively in the workplace,
• 48:57of what we expect of
• 48:58trainees and what we're looking
• 48:59for.
• 49:01Because I think where
• 49:02things get, you know, really
• 49:04mushy is when we're not
• 49:05really thinking actively about what
• 49:07are the behaviors that I
• 49:09need to see for this
• 49:10person to actually be a
• 49:11competent physician.
• 49:13We often will point to
• 49:15them being really nice
• 49:17or, wow, I really enjoyed
• 49:18working with them. You know?
• 49:20Like, these things that aren't
• 49:22helping them to actually grow
• 49:23and actually aren't,
• 49:25connecting,
• 49:27with their competence with the
• 49:29behaviors that they're showing.
• 49:31So I I try to
• 49:32be as careful as I
• 49:34can when I'm providing feedback
• 49:35to trainees and when I'm
• 49:36writing my assessments
• 49:38to actually talk about behaviors.
• 49:41And if they're not showing
• 49:43confidence or if they are
• 49:45being nice, like, using actual
• 49:47objective
• 49:48descriptions
• 49:49of what what I saw
• 49:51that made me feel that
• 49:52way.
• 49:54That's helpful for the trainees,
• 49:56because it allows them to
• 49:57wrap their brains around,
• 49:59like, what exactly do I
• 50:01need to do to improve,
• 50:02or what am I actually
• 50:03doing that's good,
• 50:06in a way that is
• 50:07tangible for them. So I
• 50:08think that's probably the lowest
• 50:10hanging fruit. It takes a
• 50:11lot of brain work. I
• 50:13I'm sure you guys have
• 50:14the same feeling as I
• 50:15do. When I sit down
• 50:16to write an assessment, I
• 50:17feel exhausted before I even
• 50:19start typing,
• 50:21because it just takes a
• 50:22lot of brain space to
• 50:24spend the time thinking about,
• 50:26like, what you observed and
• 50:27how they can improve.
• 50:29But it's usually worth the
• 50:31effort because that could be
• 50:32a really meaningful piece of
• 50:33feedback for that person.
• 50:36Any other thoughts about that?
• 50:37I'd be curious from the
• 50:38group. Other thoughts or things
• 50:40that they found helpful.
• 50:42So I'm curious, Katie. In
• 50:44that letter that you cited,
• 50:46I don't know if you
• 50:47remember.
• 50:49But and, also to me,
• 50:50sometimes letter of recommendations
• 50:53are
• 50:54seem almost higher stakes in
• 50:56some respect than just a
• 50:57straightforward
• 50:58assessment.
• 50:59And so was that individual
• 51:01struggling? There was a lot
• 51:02of time spent on
• 51:04practice based, like, learning improvement
• 51:08and in letters of recommendation.
• 51:10Sometimes when there are issues
• 51:12in terms
• 51:13of competency or performance
• 51:15performance,
• 51:16I almost feel like there's
• 51:18there could be a tendency
• 51:20for a euphemism,
• 51:22which which might translate into
• 51:24they really, really tried hard,
• 51:26but I didn't quite say
• 51:27that,
• 51:28you know, they just were
• 51:30unable to attain this competency
• 51:32in the letter of recommendation.
• 51:34I'm just curious.
• 51:35Yeah. I you know, as
• 51:37somebody who's spending all my
• 51:39free time reviewing,
• 51:41applications for residency right now,
• 51:45It's really interesting, I think,
• 51:47in, how little data actually
• 51:49translates
• 51:50on trainees in those types
• 51:52of things like MSPEs.
• 51:54And we're almost forced to
• 51:56read between the lines
• 51:58on things,
• 51:59and we're trained to do
• 52:00so.
• 52:02I I
• 52:04I think that can be
• 52:05a really tricky thing,
• 52:06to navigate,
• 52:09especially, like, letters of recommendation
• 52:11and,
• 52:11MSPEs
• 52:12and other things like that.
• 52:15Ideally, I think we would
• 52:17all benefit from there being
• 52:19transparent,
• 52:20transmission of information on how
• 52:22trainees are actually doing,
• 52:24between you and me and
• 52:25GME in particular,
• 52:27for a lot of reasons.
• 52:29There's a lot of incentives
• 52:30to not do that, and
• 52:32I think that exists too
• 52:33when residents exit out of
• 52:35training or fellows exit out
• 52:36of training,
• 52:37and they go into the
• 52:38real world, and we're writing
• 52:39their letters of recommendation.
• 52:42I wonder
• 52:43who that serves
• 52:44over time, but,
• 52:46I I don't have a
• 52:47good answer for you, Penina.
• 52:48I think that can be
• 52:49a really tricky thing,
• 52:51to navigate.
• 52:52Thanks, Katie.
• 52:54It's always a surprise when
• 52:55you go back and look
• 52:56at their residency milestones,
• 52:58when they're now in fellowship.
• 53:00Yes. Like, wow. I wish
• 53:01I would have known that
• 53:02earlier.
• 53:03Yes.
• 53:06I'm just gonna share my
• 53:07last slide, but I'll leave
• 53:09it open for any additional
• 53:10questions.
• 53:14We're prone to bias.
• 53:16It has it can have
• 53:17impact on learners.
• 53:20Implicit bias can be recognized
• 53:22through thoughtful reflection and external
• 53:24feedback.
• 53:25I encourage you to
• 53:27get feedback on your assessments.
• 53:29Ask for it from your
• 53:30peers if there's somebody that
• 53:31you trust.
• 53:33I personally have found it
• 53:34to be a really helpful
• 53:35exercise. It takes a lot
• 53:36of humility.
• 53:37But, ultimately, if the intention
• 53:39is for us to help
• 53:40our learners,
• 53:41sometimes that external feedback can
• 53:43be really helpful.
• 53:45Any other questions or things
• 53:46that people wanna talk about
• 53:48in our last two to
• 53:49three minutes?
• 53:54I'm gonna ask something if
• 53:56nobody else is. Anybody looking
• 53:58at AI for this?
• 54:01Yes. There's a lot of
• 54:02excitement in the assessment world
• 54:04about AI.
• 54:06We've been using natural language
• 54:07processing for quite a lot
• 54:09of right a long time
• 54:10in AI.
• 54:11Whether or not it actually
• 54:12translates to a better understanding
• 54:14of trainees is still up
• 54:16up for questioning.
• 54:22But if you're interested,
• 54:23let me know. I would
• 54:24love to collab.
• 54:35Penina is nervously smiling that
• 54:37I might take on another
• 54:38project.
• 54:40Not at all. I think
• 54:41it's such a great project.
• 54:43So no. Not at all.
• 54:45But, Katie, thank you so
• 54:47much. I feel like now
• 54:48that you're in the building,
• 54:49we need to keep you
• 54:50here. Like, I was shocked
• 54:51you're like, Mary Sarah brought
• 54:53you tea, and I thought
• 54:54it was like a I
• 54:55thought it was like a
• 54:56meta physical
• 54:58cyber tea or something. But
• 55:00No. Actually, I I got
• 55:02laryngitis on my flight over,
• 55:03actually.
• 55:05Oh my gosh. Well, as
• 55:06I said,
• 55:07it is
• 55:08always amazing to have you
• 55:10back, to hear your insights,
• 55:13and
• 55:14I just wish we could
• 55:15maybe Mary Sarah can, like,
• 55:16rip up your ticket or
• 55:17something and keep you here.
• 55:19But thanks so much for
• 55:22coming back.
• 55:24Thanks, everyone. It was really
• 55:25good to see everybody's faces.
• 55:26Hopefully, I'll get to see
• 55:27you again soon. Yes. Alright.
• 55:29Take care. Bye bye.