2/2 YES!: Bias in Assessment

February 02, 2024

ID11265

To CiteDCA Citation Guide

00:00Let's go. And we're being recorded, Sir.
00:04Thank you, Dana. Thank you all for
00:07joining us at this Friday noon time.
00:11We recognize some some repeat participants,
00:15which is heart warming and
00:18wonderful and some new faces,
00:20which is also lovely. Today,
00:22we're presenting as part of the Yes series,
00:25Yes with an exclamation point,
00:27which is a Yes,
00:29the Yale Medical Educator Series.
00:32And this is a brainchild of my
00:34partner in crime, Dana and I,
00:36and this is this is the first
00:39year that we have done it.
00:41So we are really eager to know what works,
00:45what you want more of,
00:46what you want less of as we optimize it.
00:50The Yes series is loosely
00:51divided into 3 topical areas.
00:53One is preclinical,
00:55one is clinical bedside and the
00:58other is more scholarly, academic.
01:00But all of them have two things in common.
01:03One, they're meant to be very,
01:05very practical, very, very practical.
01:08And what's the second thing
01:09they have in common?
01:10I don't know.
01:11They're,
01:11they're fun and we have wonderful speakers
01:13and today is certainly no exception
01:16because backed by popular demand.
01:18Dana Dunn,
01:18please tell us who is our speaker today.
01:21Yes, happy to see everybody
01:23backed by popular demand.
01:25I'm so excited to see on screen my
01:29friend and expert educator Katie
01:32Gillison who was here after going to
01:35Med school at University of Chicago.
01:37She was here as a Med PEDs resident
01:40and then did a two year general
01:44medicine fellowship with Donna
01:46Windish during which time she was
01:48enrolled in the Master's program
01:49at the NOW Center for Medical
01:51Education and got her master's and
01:53is real expert and shooting Uprising
01:56star in the world of assessment.
01:59We were happy for her but sad for us
02:02when she got scooted away when she
02:04got the the position as the inaugural
02:07program director for a new Meds PEDs
02:10program that has been approved,
02:12that she's going to head up as
02:14the new program director at Emory.
02:16But thanks to Zoom,
02:17it's like she never left,
02:19except she's got kind of a plant
02:20that looks like it thrives at
02:22warmer climbs in the background.
02:24So she's going to talk to us
02:26about bias and assessment.
02:27So Katie,
02:28take it away.
02:29Thanks Dana. It's really exciting
02:32to see some familiar faces,
02:35some familiar names.
02:38I feel like I've never really left
02:40Yale because I'm still involved
02:41in a number of things at Yale.
02:43I still edit the Yale office based medicine.
02:46I'm excited to talk with you guys
02:48about a a factor of assessment that
02:51I think has been getting a lot more
02:54attention in recent years and I'd
02:56be excited to hear perspectives
02:57and thoughts as we go through.
03:00I did this session last year,
03:01but I'm really open to feedback and
03:04you know helping this session to
03:06grow and change as as time goes on.
03:08Mostly today we're going to be sort
03:10of just having some thought experience
03:13experiments and just being a little bit
03:15more actively conscious of the bias
03:17that exists in the assessment space.
03:19And my hope is that I can offer some
03:22new insights and some new food for
03:24thought for those of you who work
03:26with learners or who are in charge of
03:29assessment in whatever space you work in.
03:32So one of the things I'll just alert
03:35you to just to be aware of is later
03:37in the session I'm going to be asking
03:39you to pull up an evaluation that
03:41you've done in the recent past.
03:43So that could be in Med,
03:44hub or whatever interface that
03:46you currently use as an educator.
03:49But ideally it should be a written
03:50assessment of a learner that you
03:52work with directly.
03:55So let's start with a bit of a Riddle,
03:59just to start to start
04:01thinking about these themes.
04:03So the Riddle is a father and son are in a
04:06horrible car crash that kills the father.
04:09The son is rushed to the hospital and
04:11just as he's about to go under the knife,
04:13the surgeon says I can't operate.
04:16This boy is my son.
04:18Explain So you can pop in the chat.
04:22Why? How? How could this be?
04:36Joseph has seen this Riddle before.
04:39OK, yes, we I think this is a
04:41common one that we see come up.
04:43When they first did a study
04:45on this in college students,
04:47only 14% of the students guessed
04:50that the surgeon was the mom.
04:52Other guesses that came up was
04:54that the father who died in the
04:56car crash was actually a priest,
04:58or maybe it was a same sex couple.
05:00There was a a number of things,
05:01but very few people statistically
05:04speaking actually guessed
05:06that the surgeon was a woman,
05:09which I think just exemplifies just
05:13straightforward bias that often
05:15for myself as a woman in medicine
05:17have encountered many times.
05:19I've often been assumed to be a resident
05:21or often been assumed to be a nurse,
05:24even after I introduced
05:25myself as a physician.
05:26So this is something that we see
05:29in our everyday practice today.
05:31For the session,
05:32I would like to just talk about
05:35the impact of bias on learners and
05:37also systems and how that impacts
05:40our learners in the long term.
05:42Identify some different sources
05:43and types of bias that we often
05:46see in the learning space.
05:48And I want,
05:48I hope and and expect that this
05:50session will allow you to reflect
05:52a little bit on your own practice
05:55of assessing learners and how bias
05:57could be entering into that practice.
06:02So what is unconscious bias?
06:05It is our natural people preferences.
06:08As humans,
06:09our brains are hard wired to
06:12categorize and group things together.
06:14It is an unconscious process.
06:17It just is an adaptive thing
06:18that our brain does because we
06:20have to process so many bits of
06:22information throughout the day.
06:23Sights, sounds, smells,
06:25thoughts,
06:25all the things that we combine together.
06:28Our brain is hard wired to bucket
06:31things together into categories
06:33and those categories can often
06:36introduce things of bias.
06:38It is sort of our own predisposition
06:40towards certain things that are familiar
06:44that that our brain naturally does.
06:46And this can impact us a lot and impacts
06:49us as clinicians if we practice in
06:52taking care of patients and it can
06:54impact our perception of learners.
06:56It can impact the way we
06:57think of their knowledge,
06:58their ability,
07:00their readiness for independent practice.
07:05I think this can be a hard thing for
07:07to embrace because it it forces us to
07:11actually recognize that there are things
07:14that we naturally do that are against
07:16what we think is perhaps morally right.
07:19And I think most of us have the assumption
07:21of ourselves that I'm fair, I'm unbiased.
07:24I really do try to treat all my trainees
07:27the same way, if at all possible.
07:31But part of that is that our brain sort of
07:35protects itself from the reality of bias.
07:38You know, what we think societally and what
07:40we think personally is morally correct.
07:43Our brain kind of refuses to recognize
07:46that belief that conflicts with what we
07:49think is good or right and society also
07:52what society thinks is good or right.
07:56But like I mentioned before,
07:57biases are just evolutionary
08:00adaptive behaviors.
08:01We make many, many conscious and
08:04unconscious decisions daily.
08:05I think for those of us who are
08:07clinicians or those those of us
08:08working in the learning space,
08:10we are constantly making decisions
08:12under a lot of time pressures,
08:14under a lot of informational pressures,
08:17cognitive burden.
08:18And so our brain can only functionally
08:21deal with so much information at one time.
08:24And so we use these categories,
08:26these systems of thinking to allow
08:29us to make cognitive leaps.
08:31And in those cognitive leaps,
08:32bias can often enter into those decisions.
08:35So for those of you who are interested in
08:37or have learned about clinical reasoning,
08:40one of the things we talk a lot about
08:42in clinical reasoning is system one
08:44and system two thought processes.
08:46So system one is our fast intuitive
08:50emotional leaps of cognitive
08:53thought where a system conscious and
08:57effortful thinking or think through
08:59things step by step.
09:01And those system one thought
09:03processes can be really where a
09:04lot of biases enter into clinical
09:06reasoning and the same can be true
09:08in the assessment space as well.
09:12There are a lot of different ways that
09:14bias can manifest in learning spaces.
09:16So affinity bias is one that comes up a lot.
09:20I as a, you know, Medpedes program
09:22director like if I know I'm going to
09:24have a student who's interested in
09:26Medpedes working with me on service,
09:28I'm immediately going to have
09:29an affinity bias towards them
09:31because I feel connected to them.
09:32I feel like I want them to join into my
09:35tribe, as my program director calls it,
09:38perception bias.
09:40So stereotyping individuals,
09:43you know categorizing them in a way
09:45that doesn't necessarily align with
09:48themselves as an individual person.
09:51Halo effects is something that
09:52we see a lot in assessment.
09:54So just an underlying belief that
09:57somebody's has good traits and
09:59therefore I like them and their
10:01likability is going to affect the
10:02way that I assess them rather than
10:04objectively thinking about the
10:06behaviors that I saw in the workspace.
10:09Or confirmation bias,
10:10which I think for those of us,
10:12our clinicians see a lot where we kind
10:14of want to confirm our pre-existing
10:17notions about things and aren't
10:19really able to see outside of them.
10:22These are just some of the biases that
10:25can enter into the assessment space.
10:27I'll talk about a couple different
10:29examples to sort of cement things
10:30down in just a second.
10:34The other things to kind of
10:36think about in the bias spaces,
10:38ways that they manifest systemically.
10:42So learning environments have various cues.
10:45I think historically in medicine we
10:48have recruited certain subpopulations
10:50of people into our academic spaces and
10:53have become accustomed and communicating
10:55in certain types of way or teaching
10:57in certain types of ways and that
10:59might not be applicable to all types
11:02of backgrounds that we encounter.
11:03Now that we are a little bit more diverse
11:06in our recruitment and thoughtful about
11:09our recruitment of individuals into
11:11medicine and other healthcare specialties,
11:13we can also bake assessment the bias
11:17into our actual assessments which
11:19I'll show some examples of from
11:21the literature in just a second.
11:23And then when we do assessments,
11:25especially when we're doing
11:27competency based assessments,
11:28so assessing you know someone's
11:30ability to take care of patients
11:32or to make complex decisions,
11:34we often bring ourselves to the table
11:37when we make those assessments as
11:39as supervisors or as assessors in
11:42clinical workspaces in particular,
11:44we bring our own preconceptions
11:46of what's right,
11:47what's wrong and sometimes
11:48impose that on our learners.
11:50And that's something that increasingly
11:52I've gotten interested in as somebody
11:54who does research on entrustable
11:56professional activities where trust
11:58is the measure of assessment.
11:59So how much bias enters into that
12:03decision of trusting someone?
12:06So let's let's talk about some
12:08specific examples of bias in
12:10assessment in medical education.
12:12I think these are really highlights.
12:14There's a ton of literature out there
12:16on bias that is very interesting
12:18and helps to reveal some of the
12:20different ways that things manifest.
12:22But I picked out a couple studies
12:24that I thought revealed different
12:26aspects of bias that I thought
12:29were interesting and enlightening.
12:31So here's an example of how Halo
12:34effect can really change the way
12:37that we think about learners.
12:39So this is a really well done study.
12:40It was published in academic
12:42medicine just last year.
12:44It was done in family medicine and
12:47the investigators created 2 videos.
12:49One was with a female standardized
12:53resident and one was with a
12:55male standardized resident.
12:57The two videos were exactly the same.
12:59They follow the exact same script
13:02and for each of those two videos
13:05they labeled each one as either
13:08above or below average for male and
13:10above and below average for female.
13:12They were the exact same video.
13:15They were just labeled as above
13:17average or below average.
13:18So they had faculty look at these videos.
13:21They said, aha,
13:23this is a above average trainee.
13:25We want you to assess them on this form.
13:29So they had 70 faculty.
13:30Observers were randomized to one of
13:33the four videos, Above average male,
13:35below average male, above average female,
13:38below average female.
13:39And they analyze the results.
13:41And what they found was that just
13:44by putting the words below average
13:47on any of the videos,
13:49they were rated significantly
13:50lower compared to above average,
13:52regardless of gender.
13:54So what that told
13:56what the investigators concluded
13:58was that just one word above or
14:01below was associated with systemic
14:03differences in assessment score.
14:05And you might ask,
14:06why is this applicable to my
14:08learners and my learning environment?
14:10I think this is a really
14:12good lesson on labels.
14:14So using labels on learners,
14:16like a struggling learner or somebody
14:19who's stellar can actually impact
14:21the way that you feel about them
14:23irregardless of other factors.
14:25So labels become a really
14:28powerful factor in assessment.
14:30So something to really think about
14:31when you're working with learners.
14:35Let's talk about gender.
14:37There's actually a lot of studies on
14:39the effect of gender on assessment.
14:41This was a really interesting
14:43one and robust in the numbers of
14:45individuals that they analyzed.
14:46So they looked at emergency medicine
14:50residents across five programs and they
14:53were looking at narrative comments
14:54and they were interested in the
14:56differences in narrative comments,
14:57comments based on gender.
14:59So this was like 10,000 comments
15:02that they were looking at.
15:04When they analyzed these based
15:06on their narrative responses,
15:08they found that men were more likely
15:11to receive specific feedback feedback
15:13based on their competence and to be
15:16rated above expected performance.
15:18And this was irregardless of
15:20faculty gender of the assessor.
15:22For women,
15:23they were more likely to get comments
15:26about low skill levels and get get
15:29more comments about their quote UN
15:32quote confidence versus men who often
15:34when they were at a low skill level,
15:37they received comments that were
15:39associated with some actionable items
15:42that they could actually work on.
15:45Lastly,
15:45the assessor gender actually
15:47had an impact as well,
15:49so women faculty assessors were more likely
15:53to rate residents as performing below level.
15:57So women were stricter graders than men.
16:02Here's a schematic of their findings from
16:04this JAMA article that was published in 2022.
16:07So you can see some of the
16:09things that favored men were
16:11comments about professionalism,
16:13critical thinking, trustworthiness,
16:15whereas women were more associated with
16:18comments about confidence and care,
16:20planning for example,
16:21or adaptability, excuse me.
16:24So there were really substantial
16:26differences in the assessment language that
16:29was used in describing these learners,
16:32and that can really have an impact
16:34on training. It can have impact.
16:36The authors commented on things like
16:39eligibility for becoming a chief
16:42resident and other opportunities
16:44that came up.
16:45This is a different study that was
16:47done in internal medicine for medical
16:50students that was published in Jgym.
16:52They used a process called natural
16:55language processing where they looked
16:57at narrative comments together and
16:59they looked on two different axes,
17:02women and men and honors grades
17:05and past grades.
17:06And they found that women who received
17:09honors grades received words like wonderful,
17:14empathetic, delightful,
17:15whereas men got words more like relevant,
17:19active, certain.
17:21So those types of things came up.
17:24So 62% of words that differ by gender
17:28represented personal attributes and
17:31then they also looked at URM students
17:33versus non URM and the honors and
17:36pass schematic as well and personal
17:39attribute words were more common for URM.
17:42So things like lovely, kind,
17:44those types of things whereas
17:46competency based language, able,
17:48smart,
17:49superb were more common with non
17:53URM students.
17:54So language differs quite a lot between
17:57different categories of individual.
18:00So that's where bias gets introduced
18:05and the last thing I will talk about
18:07is how bias can be folded into systemic
18:11or programmatic aspects that can really
18:14impact not only learners individually,
18:17but it can impact their careers.
18:19So this is a really fascinating
18:22perspective piece that was put
18:24out by UCSF in 2018 and they were
18:27really curious about why they were
18:29seeing their URM students were not
18:32achieving awards at the same rate
18:34as some of their non URM students.
18:37So they took it upon themselves to
18:39do almost a quality improvement
18:40study to understand a little bit
18:43more about why that was the case,
18:45Why did they observe these differences?
18:48So they looked at 4 cohorts of students
18:51at their institution and they found
18:54that if they looked over the course of
18:57all of the training from pre clerkship
18:59all the way to the end of training
19:02that there were differences in grading
19:04that consistently favored non URM students.
19:07And these differences were
19:09absolutely been minuscule.
19:11We're talking about on an average,
19:13you know on a one to five point scale an
19:16average score difference of 4.7 for the
19:20non URM and 4.4 for the URM students,
19:23so .3 difference.
19:24I think any of us who are
19:26in an educational space,
19:28who fill out these types of forms would
19:30agree that a difference of .3 on a one
19:33to five scale means absolutely nothing
19:35for the ability for that individual
19:37to care for patients and their ability
19:40to become a physician or a clinician.
19:42But despite the size being
19:45small of those differences,
19:47the actual effect of those
19:49differences were huge.
19:51So because of grading policies at the school,
19:54having a quota of certain number
19:56of honors grades at that school,
19:57so you know 30% of the internal medicine
20:01clerkship students could get honors.
20:03That meant that URM students
20:05received about as half as many
20:07honors grades as non URM students.
20:10And URM students,
20:12therefore cascading down,
20:13were three times less likely to
20:16be selected for things like AOA,
20:18which is a big deal if you're thinking
20:20about applying into a competitive specialty
20:23like orthopedics or that type of thing.
20:26These types of wards can have really
20:29big impacts on your ability to match
20:31into your intended field or to
20:33match into competitive residencies.
20:36So they called this their
20:37amplification cascades.
20:38So these tiny little differences
20:41that occurred over the course of
20:43their training resulted in very
20:46big differences in these individual
20:49selection selected for awards.
20:51They decided to take a really honest
20:54look at themselves and really
20:56pinpoint where they were seeing
20:59bias being introduced at various
21:01parts of the grading cascade.
21:03So they looked at the individual students.
21:06They looked at the cultural and
21:08structural aspects of the school.
21:10And I think in particular for
21:12many of you attending here,
21:13the faculty and resident reader factors
21:16were they had faculty taken on this,
21:19look at the way that they were doing
21:22assessments in the actual real workspaces
21:24and also the grading policies that
21:27were present. So it was a really
21:29fascinating paper in that they
21:31were able to make some definite
21:33changes in their grading policies
21:36in order to be less biased overall.
21:40So I wanted to take a pause here.
21:43Does anyone have any clarifying
21:46questions or there's anything that
21:48came up in the chat that would be
21:50important to talk about before we
21:51take a little small group pause?
21:58Cool. Well, I'll do a quick poll.
22:03Oh, I don't see the poll.
22:06Here we go. So I'm going.
22:09Can you guys see the poll now?
22:12Yes. Oh, here
22:15you can, because I can't see it.
22:18Let's try relaunch questions.
22:21There we go. Do you see it now?
22:25All right. So the first question is,
22:27have you seen bias manifest in the assessment
22:29of learners where you work or teach?
22:50OK Can you see the results of the fall?
22:54I'm just. OK, perfect.
22:56So it looks like about 60% have said yes,
23:01definitely 81 is unsure in which I can
23:03understand where that's coming from,
23:05where I'm not sure if what
23:06I saw was actually bias.
23:08I don't want to label it as such,
23:10but maybe and then it's about 6% said no.
23:15Now I'm going to give you an
23:18opportunity to share if you have ever
23:21seen assessment bias manifest in
23:23your clinical learning environment,
23:26how have you seen that occur?
23:28So let's see if I can,
23:31Katie, can I ask a question?
23:33Yeah. So first of all, hi,
23:35nice to see you. Hey, so you're,
23:39you're familiar with this and I'm,
23:41I'm sure that everyone here has
23:43encountered this or thought about this.
23:46Would it be considered a form of
23:47sort of systemic or institutional
23:49bias if we consistently
23:53grade inflate or I'm trying to
23:54think of a more political way to
23:56say that if we if we give glowing
23:58sort of assessments across the board
24:00without really giving sort of,
24:01you know what I mean?
24:03Yeah, If we just if we're giving
24:05great assessments to the overwhelming
24:07majority of learners. Yeah,
24:09I like, that's such a complex question,
24:12Bennett, and I love that you ask it,
24:13'cause I think that's a national
24:16debate that's going on. You know,
24:18like what is the purpose of assessment?
24:20What is, what are we hoping to
24:23accomplish by use of assessment?
24:25And for me, like in the post
24:27purest sense of the, the answer,
24:29it's to get the right information
24:30to the right people about
24:32how this individual's doing.
24:33That includes the individual themselves.
24:35That includes the person leading the course,
24:38That includes the person
24:40overseeing their education.
24:42And I think there's a lot of
24:44different pressures that come
24:46into the conversation where we're
24:48talking about grades in particular.
24:50Like for example,
24:51the medical school has a,
24:54you know,
24:55like a invested desire to make
24:57sure that that individual gets
24:59to the next step of their career
25:01in the best fashion possible.
25:05And so and that individual wants
25:08to accomplish the things that they
25:10want to do in their career as well.
25:13So I think it's a form of like
25:16Halo bias that I think nationally
25:18a lot of clerkship directors are
25:20discussing and certainly a lot of
25:23residency directors are discussing
25:24about how useful is this information
25:27actually when we're looking at,
25:28you know, thousands of applications.
25:31So I'll leave it at that.
25:32I don't know if you have any
25:34thoughts about that.
25:37Oh, I'll, I'll, I'll happily hear
25:39other people's questions and
25:40suggestions and contributions.
25:42It's, it's complex, as you say, Katie.
25:44Gary has a question.
25:45Yeah. Hey, Gary.
25:48Hey Katie. I'm wondering if you know most,
25:51most of those those words were complimentary.
25:53I'm wondering if people have looked
25:56at at critical language as well
25:58and and how that impacts bias.
26:02Yeah, I mean the the gender study and
26:05emergency medicine that I mentioned before
26:07did look at critical language and I just,
26:10I glossed over it a little
26:12bit which they, for example,
26:14if a a woman was underperforming,
26:17they would talk more about confidence
26:19and they wouldn't use really necessarily
26:22specific language on what could be improved.
26:25Whereas male residents were more
26:27likely to receive like very specific
26:29competency based language on what
26:32specifically they could improve.
26:34There's a little bit of variation
26:37in especially around gender and
26:40language in the literature.
26:41So some studies will show no difference
26:44and it sort of depends on how the
26:46investigators are looking at the information.
26:48I just actually reviewed an article
26:51from the A/C GME that got accepted
26:53that they looked at standardized
26:55videos and there was a huge difference
26:57in assessed scores and competencies.
26:59So there's a little bit of variation
27:01in the literature out there,
27:03but there is some negative like
27:05negative language is associated
27:07with gender for example.
27:10Does that make sense
27:11and is that is that largely presenting
27:13as as less specific versus more specific
27:18And that I I I think my question is
27:21ultimately is there almost a a rebound
27:24effect of some of the bias training
27:26that we're we're almost being overly
27:29cautious and not you know and not
27:31as productive as as we want to be
27:33with some of these learners. I think
27:35that's a really interesting question.
27:38I don't, I don't know if
27:40anyone has actually asked that
27:42question in the literature.
27:43I will say the propensity of
27:45information that I've seen has
27:47been that women are more likely,
27:49for example, to receive feedback
27:52on personal characteristics
27:55rather than actual actions.
27:59Like you were kind
28:05that type of thing or you
28:08weren't confident enough.
28:09And as somebody who's been a
28:10recipient of that type of feedback,
28:12it's extremely frustrating
28:13because what does that mean?
28:16Like, objectively speaking,
28:20any other questions?
28:23Now, Katie, before we get the
28:24response back from question two,
28:25there was some in the chat that said that
28:27they couldn't see the question anymore.
28:30Let me try. I did.
28:32I did put it in the chat.
28:33One person had responded,
28:36which we can, I can, I can relate.
28:39Unless you want to try to
28:40launch it again or just.
28:42Yeah, I James, would you be willing to
28:45share, if you're if you're open to it,
28:47about the experience that you had.
28:51Yeah. It's interesting.
28:52It's it's going it's going it's going
28:54back to my own training, which is
28:59sorry. It's going
29:01back to my own training
29:03which is now 35 years ago.
29:06I I I felt perhaps it was more
29:08of a personality trait that they
29:10were looking for as opposed to clinical
29:13skills or or or or compassion traits. I
29:15don't know if anyone else had that
29:17experience during their own training years.
29:21Sounds like that first kind of bias Katie,
29:23that you were identifying.
29:24I can't remember its name or
29:26they're looking for somebody who
29:27looks like that with respect. Yeah,
29:31it's it's, I think it's really
29:33fascinating the way that we
29:36group in medicine in particular,
29:38you know like is was I attracted
29:41to Med PEDs because I, you know,
29:44like there was a affinity that
29:46was generated around it or
29:48is it because I was genuinely
29:50interested in the the field itself?
29:52I like to think it's the latter,
29:54but I do think that there's some very
29:57interesting cultural things that
29:58happen in medicine that by assess
30:00one way or the other or might even,
30:03for example, if you were interested
30:05in surgery and you know,
30:07got this implicit discouragement
30:09to enter into surgery,
30:11how we're biasing our students
30:13away or towards certain things?
30:14Well, it may have been a projection
30:16of the fact that I was
30:17more interested in internal medicine
30:18and eventually pursued internal
30:20medicine. So maybe that was
30:21reflected in my own behavior.
30:23So that's fair. Totally.
30:26All right, I'll see what we have for
30:30our responses here, which I cannot see.
30:38Huh. I'm sorry guys. I can't see the
30:41written responses on the question too.
30:43I know some people entered in their thoughts
30:47it might it shut down before I back to me.
30:50Sorry, it it shut down before I could save.
30:55I'm sorry about that.
30:56Would anybody else be willing to
30:57share an experience that they had?
31:11I just can I just add one thing.
31:13I mean the the Halo effect does strike me.
31:17Just having that evidence is really
31:19useful because I always remember as when
31:22I was a clerkship director I used to
31:24question why we wouldn't kind of feed
31:26forward information about the trainees,
31:29the students as they were going
31:30from 1 clerkship to the another so
31:32that we'd be able to support them.
31:34And the administration was concerned
31:36that that would kind of poison the
31:39well and that you want to have
31:40somebody have this fresh start.
31:41And I I it's really nice to have the
31:44evidence to support the fact that that
31:46was potentially a really good idea
31:48in a high stakes grading environment.
31:51I think in a pass fail environment
31:52we would hope that we could just
31:54continue to support them.
31:55But I could see the tension if it was a
31:57higher stakes and somebody could be labeled.
31:59Yeah,
32:00I think that's such a a double edged
32:03sword conversation that I think
32:04all educators need to have about
32:08our. Often times our education systems
32:10are chopped up such that when a student
32:13enters into a new educational space,
32:15it's almost like they're starting over.
32:17Which I think could be a really great thing.
32:19And that we're not biased towards or
32:21against anything with this new learner.
32:23But it also could be a detrimental
32:25thing in that a learner might not
32:28have the ongoing support they need
32:30to continue to build their skills
32:32with the support of a experienced
32:34educator or experienced clinician or
32:36whatever field they're working in.
32:38So it can often be really challenging to
32:41think about these types of things because
32:43we do want to be fair to our learners,
32:45but we also want to make sure they're
32:47supported in the reaching the ultimate goal,
32:48which is to practice medicine or to become
32:52a medical librarian or to become a,
32:54you know, a physician's associate
32:57or whatever their goal is.
33:00So the question is how?
33:01How can we move forward?
33:03I think this brought up a lot of
33:05really controversial and also just
33:07challenging things that we wrestle
33:09with that I don't think are just going
33:11to disappear with a lot of training.
33:13They're they're going to be ongoing
33:15things that we have to be attentive to.
33:17So number one is to just be aware
33:20that bias is part of the lexicon.
33:22Bias is part of our daily lives.
33:25You know,
33:25we used to really see bias as like bad,
33:28that it's a aberrant, intentional,
33:31conscious thing that we do.
33:34But in fact,
33:35we understand bias now as normative.
33:37It's unconscious,
33:38it's often simmering under the surface,
33:41and it's largely unintentional.
33:42I don't any of us go into encounters
33:45with our learners intending to be
33:48biased towards or against them.
33:50We want to be fair.
33:53And so we just have to recognize
33:55that this is part of our hard wiring
33:58and sometimes even though we go into
34:00interactions with the best of intentions,
34:02depending on what might be going on,
34:05you know whether it be a high stakes
34:08situation or past patterns or those
34:10types of things that can result
34:13in actions that we don't intend.
34:15And just as another piece of evidence
34:17of that hard wiring in our brain,
34:19I think all of us when we're looking
34:21at Block A and Block B we can,
34:23we are looking at it.
34:24They are definitely different colors,
34:26but when we actually line up
34:29that they're the exact same color
34:31using these columns here.
34:33And Despite that,
34:34I still look at Block A and Block
34:36B and I see two different colors.
34:38It's kind of like that phenomenon
34:40with the dress online,
34:41that meme with the blue dress
34:43and the gold dress.
34:44No matter how many times
34:45I looked at that dress,
34:46I could never see gold and yet I
34:48knew people who saw it as as blue
34:51or as as gold themselves.
34:53So it's it can be really hard to
34:55unweigh themselves of that hard
34:57wiring of our brain.
34:58So first of all, one of the things
35:00that can be helpful is just being
35:02conscious of the things that
35:04make us more prone towards bias.
35:05So stress, time constraints,
35:08doing multiple tasks at the
35:11same time need foreclosure.
35:13So I don't know about you,
35:15but sometimes I get a little bit
35:17behind on my evaluations and I
35:19need to get them done to meet that
35:21deadline for that clerkship director.
35:24So I need to do that evaluation as
35:26quickly as possible to get that closure fear.
35:28And I think Ben brought up that good,
35:32that good point of you know,
35:34like what is the impact of this
35:36learner if I were to evaluate them
35:39based on this rubric fatigue.
35:41So these are all things that all
35:43of us no matter what our workspace
35:46is encounter all the time.
35:47So these are things that make us
35:49a little bit more prone to bias,
35:50more prone to make those cognitive leaps.
35:55Can we control this?
35:56Well, it can't be totally
35:58trained out of our brains.
35:59Just as we looked at
36:00those block A and Block B,
36:02it can be hard to escape that hard wiring.
36:05But we can try to be very
36:07intentional about our implicit
36:09attitudes and try our best to curb
36:12their effects on our assessments.
36:14One of my very good friends was just
36:17telling me about a mental exercise she
36:19does in this respect about patients,
36:21where every single day she and her
36:24residents would say something kind
36:26and positive about every patient
36:28on their service just to create an
36:31implicit attitude of positivity
36:33towards the patients Being objective.
36:37Honestly, self reflecting,
36:39recognizing that you will have biases
36:41and trying to pinpoint where those
36:43might be coming from and what kind
36:45of implicit assumptions that you come
36:47to the table with can be really helpful.
36:49I would say external feedback
36:52can also be extremely helpful.
36:54One of the exercises that I have
36:56found personally helpful is
36:58for letters of recommendation,
37:00having someone else read them to
37:02make sure that you're using language
37:04that actually encapsulates the
37:06ability of the individual that you
37:09are writing that that letter For.
37:11Some things that you can do to help
37:14to reduce bias and assessment.
37:15First and foremost,
37:17attending sessions like this.
37:19Starting to recognize where you're
37:21making inferences about learners
37:23assumptions or categorizing
37:25learners into certain buckets.
37:27Starting to move your language into
37:29more behaviorally based language.
37:30I understand there was a session recently
37:34for YES where there was some talk
37:37about creating a good written assessments.
37:39So avoiding personality focused
37:41language like somebody was kind and
37:44instead focusing more on what specific
37:47actions that they took to made them
37:50seem kind or made them seem in that way.
37:53And then also focusing on how you're
37:57writing words using instruments and
37:59guides and actually reading through
38:02what language you're expecting to rate
38:04or grade that person on can be very helpful.
38:07So using those actual criteria
38:09about what you physically observed
38:11in the workspace or in the learning
38:14space can be very helpful.
38:16Using checklists can be very helpful too,
38:18to objectify things a little bit more,
38:22and then also for those of you who are
38:24course directors or clerkship directors More.
38:27Observations from different faculty
38:28of different backgrounds can also
38:30be helpful in mitigating biases
38:32on a single learner
38:36individually. Just recognize that
38:39you have bias and give feedback.
38:42You can review your assessments individually,
38:44so taking a step back from your assessments
38:47and just reading them specifically
38:49only to try to detect bias in language.
38:52Have a trusted colleague read them,
38:54or I'll introduce a tool on gender
38:56bias in just a second that you can
38:58use to see to start to detect some of
39:01the language that you might be using.
39:04Practice constructive uncertainty.
39:06So really try to pause as you're
39:10making assessments,
39:11observe yourself in action and
39:13more thoughtfully consider the way
39:15that you're seeing that situation.
39:17And I'll give you a little acronym
39:20in the next slide to kind of think
39:23about embrace the awkwardness
39:25and discomfort of this.
39:27Again, I think a lot of us feel uncomfortable
39:30recognizing that we do have biases.
39:32And so embracing that and recognizing
39:35that that is part of our hard
39:39wiring as humans and by recognizing
39:41where we have implicit assumptions
39:43and addressing those head on,
39:45that can be very helpful and engage
39:48with those who are different,
39:49hear their perspectives,
39:51be interested in their viewpoints
39:54and the way that they see the world.
39:57I found this acronym to be really helpful.
39:59It's called pause.
40:00And this could be a way that you as you
40:03fill out those mud hub evaluations or
40:05filling out an intrusible professional
40:07activity or any assessment that you
40:10do in your learning space could be a
40:12great way to think about your learner
40:14as you're doing that assessment.
40:16So 1st,
40:17pay attention to what you're assessing.
40:19You know,
40:19what is the objective activity or skill
40:22set that I'm trying to look at here?
40:25So what are the goals for
40:28this particular learner?
40:29Acknowledge that you come to
40:30the table with your own biases
40:33and reactions and judgments,
40:34and really try to think about
40:36why you're seeing it that way.
40:38And understand that maybe somebody might
40:40see it a different way than you would
40:42try to be as objective as possible,
40:45really be actionable in the way that
40:47you provide that feedback to that
40:49learner and execute the assessment
40:51in a way that minimizes bias.
40:55So I wanted to share with you an
40:58assessment that I did on on a learner.
41:00This is anonymized and this is person
41:02that graduated many years ago.
41:03So I wanted to see if you guys could see
41:08if there might be any biased language
41:10in this individual's assessment.
41:12Would anyone be willing to volunteer
41:14to read this out loud to the group
41:20I don't mind reading.
41:21All right. Thank you so much.
41:25I very much enjoyed working with
41:27M They worked hard to ensure
41:28their patients were attended
41:29to and was a great team player.
41:31I appreciate the effort they went
41:33to and caring for their patients,
41:34calling consults, pharmacies and
41:36doing other intern level tasks.
41:38They were very open to feedback
41:39and actively worked to improve
41:41their performance every day.
41:42It was clear they heard and
41:45incorporated feedback consistently.
41:46They also worked very hard on
41:47improving their cardiac exam and by
41:49the end of our two weeks together
41:51was better able to identify and
41:52describe the murmurs They heard.
41:54M could work on developing their
41:56differential diagnosis of common
41:57inpatient internal medicine complaints,
41:59GI bleed, tachycardia, angina.
42:01Often their differentials were based on
42:03either previously established diagnosis,
42:05even in the setting of new data,
42:07or a limited list of possible alternatives.
42:09Some of the assigned cases
42:11simulation online courses provided
42:12by the clerkship would be helpful.
42:16All right. What do you guys think?
42:19Do you see any language in here?
42:21And this is not a loaded question at all.
42:24Is this a unbiased?
42:26Do you see some language in here that
42:29could potentially point to certain
42:31gender or other affinity biases?
42:41I think you expand on the
42:43characterization as a great team player.
42:46But in the next sentence,
42:47but maybe maybe also there were
42:50other ideas or emotions you're
42:52experiencing when you typed that, that,
42:57that, that biased that characterization.
43:00What what do you what are
43:01you thinking, Ben? Oh,
43:04like maybe you liked the person,
43:06'cause they had a, you know,
43:08they had appropriate use of humor and they
43:11made cookies and they were willing to do,
43:15I don't know, you know,
43:17they're willing to go pick up
43:19paperwork or something like that.
43:21I don't know. It's not the best example.
43:23It's just kind of the first
43:25thing I landed upon.
43:25Yeah, the cookie bias.
43:27I I will say if somebody makes cookies,
43:29I do. I do feel positive thoughts
43:32about them. Anyone else want to
43:34thank you so much Ben for reading. I
43:36would add, you
43:37know normally I would say I
43:38don't see a lot of bias in this,
43:40looks like actually a pretty good
43:42evaluation with some specific details.
43:43I think after what we spoke about
43:45I I'm just going to take a wild
43:47guess and say this was someone
43:49evaluating a man because they were
43:51very specific as opposed to saying
43:53like they lacked confidence or so
43:55in a way I wonder about that but but
43:57honestly I don't see a lot of bias.
44:01Great. I would
44:04I would say leading with
44:07with a subjective assessment.
44:09I very much enjoyed working with
44:11so and so automatically sets up the
44:14reader to interpret the the assessment
44:18in a very specific way as opposed to
44:22leading with objective measurements.
44:25I would wholeheartedly agree with that,
44:26and reading this again and again has
44:29made me realize that I start almost
44:31all my evaluations that way, which,
44:35like I did, enjoy working with them.
44:37But is that the most helpful thing?
44:39I don't know.
44:42OK, go ahead, Ben. I just noticed
44:44one peculiar turn of phrase
44:47in the second sentence.
44:48I appreciate the effort they went
44:50to and caring for their patients
44:52and that struck me as odd,
44:54like caring take is work rather
44:58than saying I appreciate the care
45:01they provided to their patients.
45:04So you know, that struck me as a little odd.
45:09And perhaps you're trying to say that
45:11this is not an empathetic patient person.
45:14They have to work hard to show
45:17that they cared perhaps and.
45:20And and of one perspective.
45:23Yeah, no, I think that's
45:24a really thoughtful and
45:26interesting perspective for sure.
45:27I hadn't even noticed that.
45:29See this is how external
45:30feedback can be quite helpful.
45:33Can I just point out that there's
45:35another sort of tension in the whole
45:38medical school and you know and course
45:41director domain which is that to me,
45:44I think starting with a statement
45:45like I very much enjoyed working
45:47with M and casting that, you know,
45:49throwing out that Halo to begin
45:51with in a way that's kind of like I
45:55think it's helpful potentially in a
45:57departmental letter for example, right.
45:59I mean people are going quickly
46:01you know program directors are
46:03reading a lot of those letters.
46:05They get they're getting nice
46:06positive feelings at the beginning
46:07of each of these evaluations.
46:08They're getting good feedback,
46:09good feedback, good feedback.
46:11I don't know.
46:11I mean it's it's the the goal of
46:14that letter is very different than
46:16the goal of the feedback here and
46:18yet we directly import the feedback
46:20into that letter as an example of,
46:23you know, how the student was assessed.
46:24So I'm not sure if that has more
46:26of a place in this conversation,
46:28but it's just something that comes to mind.
46:30No, I think that's just another
46:32example of that cascade we were talking
46:35about before about like how little
46:37things just feed together to change
46:40someone's course in their educational,
46:41you know, trajectory, which is unfair.
46:45And there are so many dynamics there
46:47that it's it's just hard, I think,
46:49for us as educators to navigate.
46:51It's also hard for us to even talk
46:54with students about it, you know,
46:56'cause we want them to be successful,
46:58not only to achieve their goals,
46:59but also to achieve the goal of becoming
47:02the physician that they want to be
47:05or whatever the educational goal is.
47:07And to me that that requires hard work
47:09and an honest reflection on your ability.
47:12And I don't know if we always are
47:14doing our due diligence with that for
47:16students in that they're we're really
47:18actively reflecting on what are they
47:20able to do when work they actually grow.
47:23Instead we're writing things like I
47:25enjoyed working with them, which is good,
47:28it's good to enjoy working with someone,
47:30but how can that help that person
47:34grow so great?
47:36These are excellent reflections.
47:38I ended up putting my evaluation into
47:41a gender bias calculator so you can
47:45in the worksheet that I provided,
47:47which should have been dropped in the chat.
47:49You can find this gender bias
47:52calculator online and I found that
47:55the words working and effort were
47:58very female associated words.
48:01I had no idea that that was the case,
48:04but it made me really think about that
48:06this was indeed a female trainee.
48:08It was a female medical student and I
48:12wondered why I was using Hard worker so
48:15much when actually they had achieved,
48:18as you pointed out,
48:20Frederick,
48:20they had achieved good patient care.
48:23So it it just made me a little bit
48:26more thoughtful about the types
48:28of language that I I used.
48:30Moving forward,
48:31I think this can be incredibly hard,
48:34though,
48:34because a good evaluation in and
48:36of itself takes a lot of thought.
48:38And then to add another step of
48:41making sure you're not gendered
48:43or biased in your assessment can
48:45take another step of extra thought.
48:47And that takes a lot of cognitive load.
48:49So it can just take that extra,
48:52extra step,
48:53but it's worthwhile and that you're
48:55going to be able to get your learner the
48:58information they need to be successful.
49:00So I I wanted to, as a thought exercise,
49:03just take 5 minutes.
49:05I think we have enough time.
49:06Yeah, 5 minutes. Pull up a recent evaluation.
49:11I thought we could do this.
49:12Just have you do it individually.
49:15Ideally,
49:15pick something that has a lot
49:17of words or narrative.
49:19You'll find the worksheet in the chat.
49:21You can use the worksheet to
49:23evaluate your written assessments.
49:24I dropped the link for the gender
49:26calculator in there and in about,
49:30we'll say 4 minutes.
49:31So at 12:55,
49:33I'll just ask volunteers to share any
49:36reflections they had from that activity.
49:50I apologize. Katie, where's the worksheet?
49:57I'll drop it in the chat
49:59myself. Oh, thank you.
50:02And Linda put it in there a couple of oh,
50:05oh, that was that one. Yeah. OK. Thanks.
52:39Nice. Thanks Ben for sharing
52:41your results in the chat.
52:43I love your honest self reflection.
52:46It looks like you have a nice balance
52:48of male and female gendered terms.
52:52So I think you have a good question there,
52:55Ben, about like how do you interpret,
52:58I think it's just mostly for you to become
53:00more attentive to the types of words
53:03that you're using in your assessments.
53:05If you are gearing more
53:07towards one side or the other,
53:09really thinking about, OK, well,
53:11if I'm using mostly female gendered terms,
53:15am I on a female trainee,
53:17Do I really need to think about whether
53:19or not I'm being objective in my
53:22assessment of their clinical ability?
53:24And So what I would suggest for
53:26if you are noticing a skew on one
53:29side or the other is to look at
53:31the actual assessment form and look
53:33at the objective language that is
53:35recommended on that form and see if
53:37there's something that fits with the
53:39learner's ability in terms of what
53:41they can or they they're growing in to
53:43see if you can reflect some of that
53:45objective language in your assessment
53:50E was execute an unbiased assessment. James,
53:56any other thoughts about or
53:58reflections after doing that activity?
54:00I'm just putting the evaluation
54:01pull up just in case folks
54:03need to leave a little early.
54:12I, I, I have a comment
54:13Katie oh great to see you.
54:17I I pulled out one of my evaluations.
54:19It was of a female trainee and I I
54:24realized I do the same thing that
54:25you do which is I said oh this
54:27person is very pleasant that you.
54:29But I think I do that almost
54:30for everybody, you know,
54:32and I personally maybe I'm old school,
54:35think that's not bad because it gives
54:37color to the evaluation, you know,
54:39the whole bunch of just traits traits,
54:41traits can be a little dry in my opinion.
54:44But anyway, so this particular trainee I,
54:48I, I emphasize things like
54:50cooperation puts patients at ease,
54:52you know, diffuse while you,
54:53you know volatile situations,
54:54which this person was really good at.
54:57And I put it into your gender
54:59bias calculator and I thought
55:00it's going to tell me I'm very
55:02female biased and it was opposite.
55:05It said I used a lot of
55:07male associated words.
55:08So it's like that wasn't my initial
55:11you know impression but anyway I I,
55:15I do think some of that emotionally
55:18based language if it's the only thing
55:21is is perhaps bias.
55:24But it does give color to to
55:27otherwise people just appear
55:29like they're all the same.
55:31I I mean, I I don't know. That's
55:32just. Yeah. And there's nothing
55:34wrong with that language at all.
55:36I would, I think having a caring
55:39physician is really nice.
55:40You know, I think somebody who
55:42shows that they care and have
55:44compassion for another person is a
55:46good trait to having a physician.
55:48But if that's all the language that
55:50is being used on one individual,
55:51that's where things become
55:53a little problematic.
55:55But I I'm glad that you use strong
55:57language to describe a female training.
55:59That's great.
56:02Frederick.
56:03I was interested in the pyramid, excuse me,
56:07that you shared in an earlier slide.
56:10It didn't have a reference
56:12associated with it.
56:13So is that something you've
56:15created yourself or is there,
56:18you know, a publication we can
56:19read to learn more about it?
56:22Which one are you referring to, Frederick?
56:25Way back in
56:26the beginning. Yeah, way back in
56:27the beginning where there's a
56:28there was a there in in the
56:30lower right corner of the slide.
56:32There was a pyramid. And
56:35I would love to Google image
56:36search where I got that,
56:37that that's not of my creation.
56:40Oh, OK. I'll see if I can find the
56:42reference and I can share it with the
56:46formerly known as the teaching
56:48and Learning Center folks who can
56:49maybe send it out to the group.
56:51Yeah, we can definitely send it out. Katie,
56:54thank you very. Thank you very much.
56:56It was, it was interesting,
56:58but you'd need to sit with
57:00it a little bit to really to
57:02really process it. Thank you.
57:04Yeah, my pleasure.
57:07Just in the interest of time.
57:10And then I'll happy to talk about
57:13any additional thoughts people have.
57:15There are some additional
57:17yes sessions coming up.
57:19I would love to attend both of them.
57:21Getting published is definitely
57:22something that we all enjoy doing
57:24So or to potentially attending that
57:27one with Doctor Martin and then Bill
57:30Coutreir who's one of the faculty
57:33at Vanderbilt and a wonderful
57:34educator is going to talk about
57:36becoming a master adaptive learning.
57:38I've attended this session at a national
57:40conference and it's incredibly useful.
57:42So highly recommend any
57:46last remaining thoughts.
57:48Thankfully,
57:49we ended exactly on time.
57:54Thank you, Katie.
57:55Hopefully people will
57:55fill out the evaluation.
57:57Tell us what you find useful.
57:58We'll get this feedback to
58:00Katie and we'll see you 2
58:05weeks from now.
58:09Thanks again for joining.
58:11Thank you and all of these
58:14slides and the talks are on on
58:17Saved to the Center website.
58:21People want to go back.