# “Lessons Learned from a Faculty Peer Observation of Teaching Program”- Heather Collette, MD, MHS - 3/20/23

May 23, 2023## Information

Pediatric Educational Learning Community (PELC): Heather Collette, MD, MHS - 3/20/23: Lessons Learned from a Faculty Peer Observation of Teaching Program

ID9955

To CiteDCA Citation Guide

- 00:00Heather doesn't need an introduction,
- 00:02but I always love and and plus I had
- 00:06the opportunity to look over your CD and
- 00:08your CD Part 2 a couple times lately.
- 00:12So. So I always love learning about,
- 00:16you know some of the really,
- 00:19really wonderful educators in our community.
- 00:22And so all of you know Heather,
- 00:25Heather Collette, AKA Dahlquist and so.
- 00:30Heather did her residency
- 00:32at University of Minnesota.
- 00:35I wish she was chief resident,
- 00:36which I should have known based on
- 00:38how comfortable you are with kind of
- 00:41like a B stuff and technical stuff,
- 00:43as well as your leadership skills.
- 00:46And then came here in about 2016 and really,
- 00:49really distinguished herself by,
- 00:52you know, by her first,
- 00:54her teaching skills and her
- 00:57commitment to education.
- 00:59She very quickly became the
- 01:03Director of the Pediatric resident
- 01:05Community Hospital rotation,
- 01:07followed by the pediatric sub internship,
- 01:12became one of our seals at Hospital
- 01:15Medicine and then Associate Fellowship
- 01:18Program Director in our newly
- 01:22established Hospital Medicine program.
- 01:25So.
- 01:25On top of that,
- 01:28she did complete her MHSMED and
- 01:31one of her areas that she had
- 01:35focused on was on initiating EPA
- 01:39assessment in the Subintern and
- 01:42that has actually become kind of a
- 01:47paradigm in UME and so you know,
- 01:51crossing potentially other departments.
- 01:56Heather has also emerged as
- 01:59an educator of our educators.
- 02:02She has certainly focused on
- 02:06increasing active teaching skills and
- 02:09then very quickly became our go to
- 02:13person leader in virtual teaching,
- 02:16which certainly coincided with the
- 02:18COVID pandemic when we all have to do it but.
- 02:22Heather walked many of us through
- 02:25it as well as really became an
- 02:28expert in the institution.
- 02:30And then over the last,
- 02:32let's see,
- 02:33probably about two years or so really
- 02:38focused on faculty development and
- 02:40became the director of the teaching,
- 02:44Optimization and Pediatrics are the top
- 02:46program, which was really her brainchild.
- 02:50And then.
- 02:51This last fall also started the J Fame
- 02:55course for our junior faculty members,
- 02:59which is based on the fellows
- 03:02as medical educators.
- 03:04And certainly as I said,
- 03:07Heather has really become
- 03:10our teachers teacher and so.
- 03:13Take it away,
- 03:14Heather.
- 03:14I couldn't be more proud of what you've
- 03:17achieved in the last couple of years.
- 03:19And as I said,
- 03:20I love reading it because it just reminds
- 03:22me of the really impressive impact
- 03:25that you've had in our department.
- 03:29Thank you, Panina. Well,
- 03:30you were there every step of the way,
- 03:32and I couldn't be here without you. So
- 03:34I really appreciate you too,
- 03:36Okay. Let me go ahead and share my screen.
- 03:43Okay. I'm going to be
- 03:44using men to meter today.
- 03:45So if any of you have ever
- 03:47been to one of my talks before,
- 03:48I love using mentee.com in order to,
- 03:52you know, be really engage
- 03:54learners and hybrid teaching.
- 03:55So if you could go ahead and use
- 03:57your phone to scan this QR code,
- 03:59I'll be having some question prompts
- 04:02throughout the presentation that
- 04:04you can answer on your phone.
- 04:05And once you're able to connect,
- 04:07if you could designate one
- 04:10of the little emojis.
- 04:11You can choose a thumbs up or
- 04:13a heart or a cat if you want,
- 04:17Just to let me know that you've
- 04:18been able to join the presentation.
- 04:20So I'll give you a second to do that
- 04:25right. And I'm also really
- 04:28excited to see that a lot of
- 04:29people in the group I know well,
- 04:30so it's fun to teach with friends
- 04:35and colleagues who I respect.
- 04:37Okay, Great. So my title today is
- 04:40Lessons Learned from a faculty Peer
- 04:42Observation of Teaching Program.
- 04:44And I just before I jump in,
- 04:46I know a lot of you guys know me,
- 04:48but Panina gave some of my background.
- 04:50But this happens to be me and
- 04:52my husband and our dog Henry.
- 04:54And we're getting a new puppy
- 04:56in a few weeks named Cooper.
- 04:58So this you'll probably be
- 05:00seeing him on a lot more of my
- 05:03presentation swimming forward.
- 05:04Okay objectives today I'm going
- 05:07to be identifying what existing
- 05:09resources we have at Yale to support
- 05:11faculty and effective teaching.
- 05:13I'll be talking a lot about
- 05:15teaching selfefficacy.
- 05:16It's been a particular new interest
- 05:18of mine and so I've been doing a
- 05:20lot of reading about it recently
- 05:21and I think it's really important
- 05:23in medical education to think about
- 05:25our educators sense of teaching
- 05:27selfefficacy and I'll have an
- 05:29opportunity at the end if you want
- 05:31to perform your own self analysis.
- 05:33And then my main point is going
- 05:35to be reviewing the outcomes and
- 05:36implications from this faculty peer
- 05:38observation of teaching program,
- 05:40the top program that Panina
- 05:43mentioned earlier.
- 05:44So before we start,
- 05:45I would like to know what learner
- 05:47audience do you typically teach and you
- 05:50can go ahead and answer on your phone,
- 05:52you can answer more than one as well,
- 06:01right? Looks like a little
- 06:04bit all across the board.
- 06:05I know a lot of you guys, so I know you
- 06:07teach a lot of different populations,
- 06:08but it's just helpful for me to know
- 06:11where you guys are at. OK, good.
- 06:14And then one other question.
- 06:16Have you ever had a peer provide formal
- 06:18feedback on your teaching? Yes or no?
- 06:24OK. Looks like a couple people
- 06:27said yes, one said no. OK, great.
- 06:35All right. So I'm going to talk a little
- 06:37bit about the background of clinical
- 06:39faculty teaching and how I got
- 06:41interested in this particular program.
- 06:43So with the current status
- 06:44is really that you know,
- 06:46we as clinical faculty faculty are
- 06:48expected to teach frequently with
- 06:51minimal education and medical school and
- 06:52residency on how to teach effectively.
- 06:55Now I want to give Penina a shout
- 06:57out because I think she's really
- 06:58tackled this problem so well within
- 07:00our pediatric department and now we
- 07:02have so many opportunities to work on.
- 07:05Our teaching skills and advance
- 07:07them and makes us feel more like
- 07:09better teachers where we have more
- 07:11opportunities to get skills on all
- 07:14these things and and so but one thing
- 07:16that we do is that we get feedback from
- 07:19learners but not a lot from peers or
- 07:22trained experts in medical education.
- 07:24So we're really relying primarily on our
- 07:27Medhub evaluations of our teaching to
- 07:29let us know if we're doing a good job.
- 07:31So some of the existing resources
- 07:33that I mentioned,
- 07:33obviously these pelk sessions,
- 07:35pediatric learning community
- 07:36sessions are great to discuss
- 07:38different educational topics on.
- 07:40The medical education discussion group in
- 07:43the School of Medicine is wonderful too.
- 07:45We have our medical education fellowship,
- 07:47the MHS program that I did,
- 07:49the top program now which is this
- 07:52peer observation of teaching.
- 07:53We have a new junior faculty as
- 07:55medical educators course and then
- 07:57the Poor View teaching center which
- 07:59Janet Haffler heads up is also.
- 08:01Has lots of resources online
- 08:02when it comes
- 08:03not the poor vu oh oh
- 08:07it's different,
- 08:08it's different, it's different.
- 08:09The teaching and learning said OK,
- 08:10sorry, thanks Janet.
- 08:13By the way, I was just writing
- 08:15in the chat not to interrupt
- 08:16is you gave a shout out to me.
- 08:18I have to give a shout out to Janet
- 08:20who oversees many of these programs,
- 08:22which was the inspiration for many
- 08:25of our programs and being so. Thanks
- 08:28Panina. Thank you.
- 08:30So lots of different resources
- 08:32when it comes to teaching skills.
- 08:35So I you know, when doing a little
- 08:36bit of a literature research,
- 08:38it's been documented that faculty
- 08:40teachers believe that student
- 08:43is feedback is useful in their
- 08:45clinical development but they can
- 08:47be uncertain about the honesty or
- 08:49accuracy of student assessment.
- 08:51And so I wanted to see if how
- 08:53much you guys actually agree
- 08:55with this particular statement.
- 08:57The one that was published was done
- 09:00in internal medicine group of faculty
- 09:02in these semistructured interviews.
- 09:04So I'm curious to understand how how
- 09:07much you agree with this statement.
- 09:18I know that sometimes as a a faculty,
- 09:21teacher and medical education,
- 09:22it's difficult for learners
- 09:24to give you, you know, true,
- 09:26honest or accurate feedback with
- 09:27because of the power dynamic that
- 09:29comes between you and the learner.
- 09:31So sometimes it can be.
- 09:34Difficult to get true, honest,
- 09:35or accurate assessments if there's
- 09:37any type of implication for
- 09:40that particular relationship.
- 09:41All right, so it seems like the majority
- 09:44of people strongly agree with this one.
- 09:47And then also in the literature
- 09:49there they say that faculty teachers
- 09:51prefer to receive feedback from a
- 09:53competent clinical teacher who's an
- 09:55expert in educational performance.
- 09:57So I'm curious to see if you guys
- 09:59have the same rating of agreement
- 10:00with this particular statement.
- 10:10Okay, great. So about the same,
- 10:14maybe even a little bit stronger than
- 10:16the previous statement. Okay, great.
- 10:18So it's been all documented.
- 10:20I mean, there's a the published article
- 10:23that I'm referencing here was from 1983,
- 10:25but there's been a lot of a lot of data
- 10:28showing that written teaching feedback.
- 10:29It increases clinical faculty
- 10:31teaching evaluation scores over time,
- 10:34their awareness of teaching behaviors,
- 10:36their willingness to be evaluated,
- 10:38it improves their objective teaching skills
- 10:40and it increases their passion for teaching.
- 10:43So getting feedback on your teaching
- 10:45is is really important for us as
- 10:48clinical faculty and right now we
- 10:50have this feedback spectrum where
- 10:51we have what the learners want to
- 10:54know and how they want to learn it.
- 10:56And then we have what the teacher
- 10:58thinks the learners need to know
- 11:00and how they want to teach it.
- 11:02And so we have these two ends of
- 11:04the spectrum and what what's missing
- 11:06in the middle is something to
- 11:08kind of join those two together.
- 11:10And so really it highlights the need for
- 11:12a multifaceted approach to assessment
- 11:14of our clinical teaching with an
- 11:16understanding of how student and feedback,
- 11:18faculty feedback interrelate
- 11:20to create the best.
- 11:22Teaching encounter.
- 11:23So the the thing that or the
- 11:27concept that I'm proposing you
- 11:28know with this top program is to
- 11:31have this trained peer faculty,
- 11:32observer to to who's just disconnected
- 11:35from the teaching encounter and
- 11:37is observing the learners and
- 11:40is observing the teacher and is
- 11:42working to you know bring both of
- 11:45these pieces of feedback together.
- 11:48So with a trained peer faculty observer,
- 11:50the the the benefits is that
- 11:52they're really not involved in
- 11:54the teaching encounter at all.
- 11:55There's no stake in learner
- 11:57or teacher success.
- 11:59They can provide some relatively unbiased
- 12:02feedback without any consequences or
- 12:05implications to the success of the
- 12:08learner or the success of the faculty.
- 12:11And so this is where the the
- 12:12top program comes in.
- 12:13So I'm going to go ahead and describe
- 12:16the program as a whole and go into
- 12:19some of the outcomes that we have.
- 12:21So the top program is a pure observation
- 12:24of teaching for pediatric faculty.
- 12:26At the beginning of the program,
- 12:27we recruited 6 faculty observers.
- 12:30And we were able to get access to
- 12:33student resident and fellow lectures
- 12:36at and at this time they were
- 12:38primarily happening via zoom and
- 12:39we're mostly hybrid lectures where
- 12:41some of the learners were in person
- 12:43and some of them were via Zoom.
- 12:45But all of the observations
- 12:47were occurring via Zoom.
- 12:48And this started back in July 2021 and
- 12:52ramped up over the the past year and a half.
- 12:56So with the program we trained
- 12:58the six observers and using this
- 13:00top observation tool.
- 13:01And here is where I will definitely
- 13:03give a shout out to Janet because
- 13:05she is the original creator of this
- 13:07particular tool and with her permission,
- 13:11Uma Fatak and one other colleague and I,
- 13:14we made a few changes to this to include
- 13:17another domain such as the use of media.
- 13:21And changing some of the wording around
- 13:24just to highlight our particular population.
- 13:26So as you can see here, there are 6
- 13:29domains of the top observation tool,
- 13:30including the learning environment,
- 13:33promoting learner engagement,
- 13:35managing the session content,
- 13:37verbal and nonverbal behavior,
- 13:40and then the use of media with several
- 13:43different types of behaviors under each.
- 13:45And then in the middle column,
- 13:46the observer can mark
- 13:49whether the behavior was.
- 13:51Exhibited, yes.
- 13:52It's somewhat or no or not applicable.
- 13:56They can fill in the comments on
- 13:57the right and then at the bottom
- 13:59they can enter one to three
- 14:01effective teaching skills and one
- 14:03to three areas for improvement.
- 14:04And we created this in Microsoft Word
- 14:07as a form so that they're able to just
- 14:10type in the responses and click on
- 14:12the boxes and it marks it with an X.
- 14:15So this is the tool that we used.
- 14:18It's not officially validated,
- 14:19although with this top program
- 14:21we're actually working on collecting
- 14:23some validity data for it and are
- 14:26working toward that, hopefully soon.
- 14:30The next thing that we did were
- 14:32was to observe the OR, sorry,
- 14:34train the observers.
- 14:35In the use of the ADAPT model for feedback.
- 14:38So I'm not sure if any of you
- 14:40have gone to any of Uma Fatak's
- 14:42lectures on giving feedback at,
- 14:44but I know she really likes
- 14:46this format and I do too.
- 14:48Subsequently,
- 14:48I think it works really well as a
- 14:51quick way to train faculty and how to
- 14:54give feedback by asking for feedback
- 14:57from the learner discussing it.
- 14:59You know,
- 15:00clarifying points and then
- 15:01planning next steps together.
- 15:02So the this type of training was given via a
- 15:07a a short video that the observers watched.
- 15:11So since the beginning of the
- 15:14program in 2021,
- 15:15we have 70 lectures observed in
- 15:17total with 50 faculty participants,
- 15:20with 20 faculty kind of completing
- 15:22the program,
- 15:23which means that they completed
- 15:272 observations.
- 15:28So the faculty themselves.
- 15:30They receive written and verbal feedback
- 15:32on two different teaching encounters,
- 15:35with the hope that they make changes
- 15:37on their second encounter based on the
- 15:39feedback they received from the first.
- 15:42We also had them fill out a
- 15:44qualtric survey of a pre and post
- 15:46teaching self efficacy survey score,
- 15:48which I'll share with you here.
- 15:51So this is an example of the PTSQ.
- 15:54We focused on the 1st 11 components
- 15:57of it because the last components
- 15:59involved working with patients.
- 16:02So we didn't feel like it was applicable,
- 16:04but the PTSQ is a validated instrument
- 16:07to assess physician's clinical teaching
- 16:10self efficacy and so it has 11 items.
- 16:14Where they list certain behaviors
- 16:17and the faculty can indicate to what
- 16:20degree the item applies to them.
- 16:22So you can get a score for each
- 16:24item from zero to four,
- 16:26zero where the item does not apply
- 16:28at all to them to four where the
- 16:31item fully applies and some of
- 16:33the items here you can see.
- 16:35I'll just list a couple examples because
- 16:36I know it might be a little small,
- 16:38but if it even if students
- 16:41ask difficult questions,
- 16:42I'm able to answer them correctly.
- 16:44Even if I'm under time strain,
- 16:46I'm able to concentrate and I'm able to
- 16:50integrate even the weakest of students.
- 16:52You know, even when I'm faced
- 16:54with big student groups,
- 16:55I can reach every student,
- 16:56things like that.
- 16:58So these are different items that
- 17:01would create a challenge or for for
- 17:04the teacher and so it reflects at to
- 17:07like to what degree they feel like.
- 17:09They can handle that particular
- 17:11challenge whether it's coming from
- 17:12within themselves or it's coming
- 17:14from the relationship between them
- 17:15and the student and we'll talk a
- 17:17little bit more about this shortly.
- 17:21So teaching self efficacy I thought
- 17:23was an important item to include
- 17:25in this particular study because
- 17:27not only do we want to increase
- 17:29objective teaching behaviors of.
- 17:31Of of our faculty.
- 17:32But we want them to feel like they're
- 17:34doing a good job and that they can
- 17:37do a good job when they're faced
- 17:39with different teaching tasks.
- 17:40So teaching self efficacy in
- 17:42the Med Ed realm is described as
- 17:45physician's belief in their capacity
- 17:47to carry out the actions necessary
- 17:50to provide high quality teaching.
- 17:52And it has been well documented that
- 17:54when teachers have a high self efficacy,
- 17:57it does influence their teaching performance
- 17:59and it has an effect on learner success,
- 18:02which is ultimately why we all teach.
- 18:05So our learners can succeed.
- 18:07And self efficacy has been
- 18:09shown to affect motivation,
- 18:10so it can affect the our ability
- 18:13to set goals for ourselves.
- 18:15It affects how much effort we expend it.
- 18:18It affects how long we can persevere
- 18:21in the face of difficulties and
- 18:23our resilience to failures.
- 18:25So you know, a lot of us are,
- 18:28we have,
- 18:29you know,
- 18:30stressful jobs working as clinicians
- 18:32and research and teaching and so
- 18:35when we're faced with time,
- 18:38time constraints or difficulties
- 18:40if we don't have a high sense
- 18:42of teaching self efficacy.
- 18:44We may not set the goals or expend that
- 18:47effort to teach the way that we are as
- 18:50much as we have a potential to do so.
- 18:53And like I mentioned,
- 18:55teaching self efficacy has a positive
- 18:57impact on students academic achievement,
- 18:59which again,
- 19:00is why we're all here,
- 19:03All right. So when we're talking
- 19:05about teaching self efficacy, oh,
- 19:07sorry, I'm just going to go here.
- 19:08We, we looked the main 2 outcomes really,
- 19:11of the program we're looking at.
- 19:13The change in their objective teaching
- 19:16scores as well as the change in
- 19:18their teaching self efficacy scores,
- 19:20again with the hopes that both would
- 19:23increase with the intervention
- 19:25that we're providing of this
- 19:27written and verbal feedback.
- 19:29So with the top score sheet here,
- 19:33there are 27 total points
- 19:36that are applicable.
- 19:38However, when we did score this,
- 19:40some of the items weren't,
- 19:42they were in the not applicable column.
- 19:44So what we ended up doing was scoring it as
- 19:47a percent of the total achievable points.
- 19:50So I will share here that we did
- 19:55have a significant increase in the
- 19:57average change in the top score from
- 20:0088 to 95% out of the achievable
- 20:03points on the top score.
- 20:05So an average change in.
- 20:08Seven points.
- 20:08So as you can see,
- 20:10a lot of our faculty are already
- 20:13pretty good teachers, which is great.
- 20:15A few outliers,
- 20:17which we can talk more about later.
- 20:18But one of the one of the themes
- 20:22that we really noticed with the top
- 20:25program was that people who were
- 20:27potentially already teaching at
- 20:29a high level were more likely to
- 20:32accept the invitation to be observed.
- 20:35Or potentially seek out the
- 20:38the opportunity to be observed.
- 20:39So we realized that with the top program,
- 20:43at least in the initial initial
- 20:44part of the program,
- 20:46we were probably reaching the
- 20:48higher level teachers already who
- 20:50were teaching a lot.
- 20:51They were giving a lot of clerkship lectures,
- 20:53residency lectures because they're
- 20:54known to be high quality teachers,
- 20:56so they're just teaching more frequently.
- 20:58We because of this,
- 21:00we did pivot a little bit to
- 21:02include more fellow lectures where
- 21:04more faculty are just assigned to
- 21:06teach as part of their job.
- 21:08They're not being asked specifically.
- 21:10So we're hopefully see some
- 21:13changes in that overtime.
- 21:15So we were happy to see that there was
- 21:17a significant change in the top score.
- 21:20After our intervention,
- 21:22quick question, Heather,
- 21:22for that one. It's the 20 faculty
- 21:25who had the pre and post, right?
- 21:28Yes, that's right, Okay.
- 21:30And did you go back just cuz I
- 21:33haven't seen the data and kind of
- 21:35look at the baseline for everybody
- 21:37now that you've had a chance to
- 21:39include some of the fellow lectures.
- 21:42I'm just curious.
- 21:43Like your hypothesis is that maybe
- 21:45now you're reaching some that
- 21:47weren't as better performing.
- 21:48And so like any sense of what
- 21:52the average is these days,
- 21:55yeah, it's probably more
- 21:56toward on the 80% side like 75 to 80%.
- 22:01Also it's you know although we
- 22:03had initially the six observers,
- 22:05we're having more observers now.
- 22:07So we're actually we're having
- 22:09a little more of a range and.
- 22:12On in scores as well.
- 22:13So hopefully it's a little bit more
- 22:14going to be more accurate instead
- 22:16of just doing all the observations.
- 22:17So yeah,
- 22:18OK, right. OK, thanks.
- 22:22OK. And then we did do some sub analysis
- 22:26initially looking at each domain to
- 22:29see if there were any particular one
- 22:30of these domains that had a more
- 22:33significant change than the others.
- 22:34And so we were able to find that three of
- 22:38the domains had a significant increase.
- 22:41And that would be in establishing
- 22:43the learning environment, promoting
- 22:45learner engagement and the use of media,
- 22:48which in my experience certainly reflects
- 22:51the areas where I I as an observer.
- 22:55I made a lot of comments,
- 22:57especially on using cases and different
- 22:59types of polling software to,
- 23:01you know, engage learners.
- 23:03And then also, you know,
- 23:05I think one thing I emphasized
- 23:08a lot as an observer was to.
- 23:11Spend more time and pay more attention
- 23:13to establishing a learning environment,
- 23:15especially when you're teaching
- 23:16in a virtual or hybrid session.
- 23:18And so I think a lot of the faculty
- 23:20took that to heart and spent more time,
- 23:23you know,
- 23:23establishing some type of a learning
- 23:25environment and getting to know
- 23:27their learners and their learner
- 23:28level before they jumped into their,
- 23:30their teaching content.
- 23:32So that was great.
- 23:36So with the PTSQ score.
- 23:39Or the teaching self efficacy.
- 23:42There's eleven items with
- 23:44a Max total points of 44.
- 23:47Again each item is ranked zero to four.
- 23:50So when we looked at this again,
- 23:53we have a small number.
- 23:55So we're hoping to get
- 23:56more significance later on.
- 23:57But we did see a change,
- 23:59average change in their self
- 24:02efficacy score from 27 to 30
- 24:04with a P value of point O 7.
- 24:06So we're approaching significance
- 24:08but not quite there yet.
- 24:1111 item that was really interesting.
- 24:13I think Pina you might remember
- 24:15initially when I was analyzing this
- 24:17data we were finding that their self
- 24:20efficacy scores were actually going
- 24:22down and so I did a little bit of of
- 24:24research into that and it's fairly,
- 24:27it's fairly well established that.
- 24:30You're especially when you started
- 24:32a high level of of self efficacy,
- 24:35you're more likely to go down after
- 24:38feedback because you're you kind of
- 24:40realized that maybe you weren't kind
- 24:42of as good as you thought you were,
- 24:44but that when after the
- 24:47implementation of feedback,
- 24:48your self advocacy actually goes
- 24:50even higher than it was before.
- 24:53So really the goal of this
- 24:55program is to have that feedback
- 24:57implemented with the goal of.
- 24:59Them,
- 25:00you know,
- 25:00faculty teachers feeling even more
- 25:02efficacious than they did previously.
- 25:07So what's interesting about this
- 25:10PTSQ score is that it's broken
- 25:12up into two different sections.
- 25:14The first six items are related to
- 25:18self regulation and the last five
- 25:20items are related to dyadic regulation.
- 25:23And I'm going to talk about that because
- 25:25I did not exactly know what that meant.
- 25:27So with the selfregulation components,
- 25:30these behaviors basically reflect
- 25:32cognitive or emotional threats
- 25:34to the teacher that require some
- 25:38type of internal selfregulation.
- 25:40So you know, it's about,
- 25:43you know, time management.
- 25:45It's about, you know, staying calm,
- 25:47managing your emotions even
- 25:49when they're triggered,
- 25:50things like that.
- 25:51So that's what those first six items
- 25:54reflect and then the last five items.
- 25:56Or the dyadic regulation,
- 25:58where these reflect challenges
- 26:00that arise between you as a teacher
- 26:02and the learner that requires some
- 26:04type of intervention that's really
- 26:06aimed more towards the learner.
- 26:07So it's about trying to reach
- 26:10every student in in your group,
- 26:13or adapting to different knowledge levels
- 26:17or potentially you know if you're given.
- 26:21An assignment to teach on short,
- 26:23short notice that these are kind of
- 26:25external things that require regulation,
- 26:27kind of outside outside of yourself.
- 26:31So we haven't,
- 26:32we don't have enough numbers to,
- 26:33you know,
- 26:34figure out whether there's a
- 26:36significant change more in one area
- 26:39or the other based on the program yet.
- 26:41But I think this is going to be really
- 26:44important when it comes to targeting
- 26:46interventions to support our teachers.
- 26:49You know whether it comes to teaching
- 26:51them more self regulation teaching
- 26:54techniques versus more learn learner
- 26:56targeted interventions to help improve
- 26:58their sense of teaching self efficacy.
- 27:00So more to come there.
- 27:04So really have some of the
- 27:04lessons that we learned from the faculty,
- 27:08teachers themselves where that,
- 27:09you know faculty really benefit and
- 27:12appreciate a teaching framework.
- 27:14I know a lot of us value certain teaching
- 27:17behaviors more than others, whether it's.
- 27:20Organization or just trying to think,
- 27:24you know, doing, making sure we have
- 27:25objectives and things like that.
- 27:27But when we can,
- 27:29when faculty see a list of,
- 27:31you know, expected behaviors,
- 27:33I think that really helps them touch
- 27:35on everything that's expected.
- 27:38Effective teachers, like I mentioned are,
- 27:39are more likely to seek and accept feedback.
- 27:42So certainly finding ways to
- 27:45reach all teachers is something
- 27:47that we still have to work on.
- 27:49Another thing that's really
- 27:50come up and Panina knows,
- 27:51I've been talking to the folks that meant
- 27:53to meter to try to get a a license for us.
- 27:56But more support in the use of
- 27:57technology and teaching is needed,
- 27:59especially in our virtual age now
- 28:03where we're teaching not only it via
- 28:06zoom but hybrid is really I think the
- 28:08number one mode of teaching at this point.
- 28:11And so trying to engage both,
- 28:14you know, in person and virtual
- 28:16learners is really difficult.
- 28:18And so technology is a way
- 28:20to bridge some of that gap.
- 28:21Formal feedback is appreciated.
- 28:24You know,
- 28:25I think a lot of the faculty,
- 28:26teachers, you know,
- 28:28aren't have never gotten feedback from,
- 28:30you know,
- 28:31a trained peer faculty and medical education.
- 28:33So I think that's really nice.
- 28:37And then I think like I mentioned before,
- 28:39that the teaching self advocacy can go down,
- 28:41but it's important to remember that it can
- 28:44go back up once the feedback is implemented.
- 28:48So some of the lessons that we
- 28:50learned from faculty observers,
- 28:51so again,
- 28:52these were the the trained faculty
- 28:54who were doing the observations was
- 28:57that observers again appreciate
- 28:58that rubric to follow.
- 29:00They benefit from that training
- 29:03and feedback delivery,
- 29:05especially when it's peer-to-peer.
- 29:07Certainly there's no sense of power dynamics.
- 29:10So you want to give feedback in a way
- 29:13that maintains that relationship.
- 29:16I I found that observers are
- 29:18really pleasantly surprised by what
- 29:20they learned from one another,
- 29:21not only from a medical education
- 29:24technique perspective, but also from content.
- 29:26I know a lot of faculty, you know,
- 29:30you know an endocrine like to
- 29:31go to a lecture on Kawasaki,
- 29:33and you know it reminds them of you know,
- 29:36of things that they haven't
- 29:37learned about in a while.
- 29:38So that's always kind of fun.
- 29:39And I think that once observers
- 29:43see teaching techniques.
- 29:44That are used successfully,
- 29:46they're more motivated to
- 29:47try them themselves.
- 29:48So whether it is using,
- 29:51you know,
- 29:52technology and teaching like pole
- 29:54everywhere or meant to meter or
- 29:57using cases or or doing chalk talks
- 29:59or are doing more hands on teaching,
- 30:01when they're seeing that you
- 30:03successfully then they'll be
- 30:04more motivated to try that too.
- 30:06And then I think that I think has
- 30:08been the most difficult is to make
- 30:09sure that these verbal feedback
- 30:11sessions are happening.
- 30:12I think it's certainly easy to fill out
- 30:14your teaching rubric and pass it along
- 30:16to the faculty and solicit questions.
- 30:19But I've found that these verbal
- 30:20sessions are really helpful just
- 30:22to review different comments.
- 30:23And it allows the faculty teacher
- 30:26to reflect on some of those items
- 30:28and come up with a, you know,
- 30:30a plan that they might try differently
- 30:32for the next teaching encounter.
- 30:36And then this is a little off topic,
- 30:38but it was another kind of side project
- 30:41I have here is looking at some of
- 30:42the the feedback from the faculty
- 30:44versus some of some student feedback.
- 30:47So I've been working with UMA and
- 30:49Michael comparing some of the Medhub
- 30:52feedback on clerkship lectures
- 30:54based on the the faculty feedback.
- 30:56And so I think I wanted to just highlight
- 30:59that the number one item that really
- 31:02overlapped between the two populations was.
- 31:05Learner engagement.
- 31:06So really making sure that whatever
- 31:10type of lecture you're giving to include
- 31:13not just interactive components,
- 31:16but activities that promote critical
- 31:19thinking within within your within
- 31:21your lecture.
- 31:22So it doesn't mean that people have
- 31:24to notice necessarily speak up,
- 31:26but just promoting that critical
- 31:28thinking is really important,
- 31:30another important item to note
- 31:32in this particular.
- 31:33Side study was that students focused
- 31:36a lot more on content of the lecture,
- 31:39so making sure that it was relevant
- 31:41to their level of learning,
- 31:43how much content was presented,
- 31:45whether it was presented in
- 31:47an organized fashion.
- 31:49And they really liked things that were
- 31:51interactive specifically with cases,
- 31:54whereas faculty feedback focused a
- 31:56little bit more on learning environment,
- 31:59how facile the use of media was,
- 32:02and PowerPoint.
- 32:02And the overall management of
- 32:04the session in terms of pace and
- 32:07organization and things like that.
- 32:08So those are just some initial
- 32:10reflections that I thought
- 32:11were interesting to include.
- 32:14So really when we're talking about
- 32:17implications from the program,
- 32:18we can think about how how our findings
- 32:22and and how this particular program
- 32:25and the tool itself could set teaching
- 32:28expectations potentially for new hires.
- 32:30Maybe some teaching
- 32:31expectations for promotion,
- 32:33especially with the new academic
- 32:35clinician track that's coming up soon
- 32:38and helping faculty who have areas for
- 32:41improvement and then also appreciation
- 32:43for faculty who are teaching well
- 32:46again to get that positive feedback to
- 32:48continue to have them teach over time.
- 32:51So hopefully I've highlighted,
- 32:52you know the need for this multifaceted
- 32:55approach to assessment of clinical teaching.
- 32:58Between students and faculty and I wanted
- 33:01to give you an opportunity to do your own.
- 33:05I'm teaching self efficacy analysis.
- 33:07It's a link to a qual trick survey
- 33:10where you can fill out eleven of the
- 33:12the components and you can put in your
- 33:14e-mail and I'll send you a little of your
- 33:17own analysis if you feel comfortable,
- 33:19and then we can reflect on
- 33:21that a little bit together.
- 33:23So I'm oh, I know I ended quite early,
- 33:26but I'm available for any questions
- 33:28or comments about the program
- 33:29or the study in general.
- 33:36I'm torn between my question or doing that,
- 33:42but I I was intrigued because I remember
- 33:46the earlier data that you had shown in
- 33:49terms of the PTSQ going down and so.
- 33:53How did you figure out if they
- 33:55had incorporated the feedback?
- 33:56Does it relate to what you said?
- 33:58Like if you wait longer for them to redo
- 34:02the survey, that's when it goes up?
- 34:05So how did you tease that out?
- 34:07Yeah, honestly, I was looking.
- 34:09I looked at the PTSQ scores and like who
- 34:12had the most change versus who didn't.
- 34:14And then I I I looked at their,
- 34:16their teaching observations and
- 34:18I was looking at the feedback
- 34:20that they received from the first
- 34:21one compared to the second one.
- 34:23And I I was just noticing that the the.
- 34:28Even even the faculty who had
- 34:30more than six months in between,
- 34:32like the longer almost the better.
- 34:34I don't know what it was maybe more
- 34:36time to reflect on the the feedback
- 34:38or more time to prepare like a
- 34:41different type of presentation.
- 34:42I think, you know,
- 34:44a lot of us have these canned
- 34:46lectures that we give all the time
- 34:48and it takes energy to change
- 34:49things whether it's the format or
- 34:52the interactivity that we choose.
- 34:53And so the the observations that
- 34:56were within a month or two I could.
- 34:58Not a lot of changes had been made
- 35:01probably from time constraints or motivation,
- 35:04but so yeah,
- 35:05the longer the better,
- 35:07which was really interesting.
- 35:08That's great.
- 35:09And Heather, cuz I was, I was,
- 35:11I was just so intrigued by that.
- 35:13And so in terms of the literature,
- 35:15cuz you kind of had talked about it,
- 35:17is there literature to show that
- 35:20once you've instituted changes or
- 35:22the duration of time in terms of
- 35:25the PTSQ you know or perceives?
- 35:28I always say perceived self advocacy,
- 35:30but it's really self advocacy is the term.
- 35:34You know, is there data to say that
- 35:37the longer you wait or the more things
- 35:39that you institute that it actually
- 35:42goes up or is that your hypothesis?
- 35:44I think that's my hypothesis.
- 35:45I don't have any data on the
- 35:49time frame, but you know,
- 35:51one thing that was interesting too was.
- 35:53That the folks who had a higher
- 35:56self efficacy, sense of self
- 35:58efficacy to begin with often had
- 36:00less of a change than those prior.
- 36:06Yeah, you can call on people to unmute
- 36:08them. I don't have to. But Janet,
- 36:11sure I have just a bit to add.
- 36:12Thanks a lot. We've really taken
- 36:14the program forward and what I did
- 36:17want everyone to know Zandi Garino,
- 36:20who's the associate Dean for PA education
- 36:22now her thesis when she was at Florida,
- 36:25I I had mentored her through that,
- 36:26but her whole thesis was on
- 36:29feedback and how you incorporate.
- 36:31Certain characteristics of
- 36:33feedback are incorporated or not,
- 36:36and it's a wonderful thesis.
- 36:38She then published it.
- 36:40Three years ago in Advances
- 36:42in Health Science Education,
- 36:44it's worth looking at.
- 36:45But it really it was a qualitative study,
- 36:48very well done and she really
- 36:51talks about learner readiness
- 36:53to accept the feedback.
- 36:54So again,
- 36:55it's it's pretty good,
- 36:56it's great work.
- 36:58Thanks Janet. I'm just curious from the
- 37:01group because I would know I was one
- 37:03of the first who volunteered to do it.
- 37:07Is there anybody here? Well,
- 37:09I don't want to put anybody on the spot,
- 37:10but reasons for not engaging in this and the
- 37:14reason why I ask is you wrote as a future,
- 37:17you know, as a future thought,
- 37:18which I think is a great one,
- 37:21which is new faculty,
- 37:23you know, to get them early
- 37:25and give them feedback early.
- 37:28But what would be the barriers
- 37:31to doing that and why don't some
- 37:34of our path like I I think?
- 37:36Most of the people that you reach out,
- 37:39if they answer you, they agree to do it.
- 37:41But ideas about those who are not
- 37:44getting the feedback at this point,
- 37:50and Heather, you can answer, I'm just,
- 37:52I was actually going to start with the
- 37:53group and say who hasn't done this,
- 37:55why would you not do this?
- 37:56But I decided I wasn't going
- 37:58to put anybody on the spot.
- 37:59So let's go more general as to
- 38:03reasons why faculty don't do this.
- 38:08Or don't engage in it, you know,
- 38:10it's interesting when it's structural
- 38:12Panina. And if it were just
- 38:14required that everyone would do it,
- 38:17because if it's very safe and you might
- 38:19not even want to use the word evaluation,
- 38:22I often call it that.
- 38:23I'll have a dialogue about
- 38:25what I just observed. So again,
- 38:27thinking of it being totally safe,
- 38:30knowing that it's not going to go into
- 38:32your portfolio or it won't get back to you,
- 38:34Panino or any other leader.
- 38:36So the safety issue I think
- 38:38is really critical.
- 38:40And also you know,
- 38:42being very careful saying if you
- 38:45wanted for it to be more formal
- 38:47as an observation that would
- 38:49actually go into their portfolio
- 38:51when they're coming up for a MP.
- 38:53So you could have sort of two arms of this.
- 38:56But again,
- 38:57really probably if you were saying
- 38:59University of British Columbia
- 39:01was one of the first places to
- 39:04implement the structure of people
- 39:06having a formal peer review process.
- 39:09Yeah, I actually. Right.
- 39:12And so I would, I would just see
- 39:16the benefit in having all comers.
- 39:19And since you see the other side, Janet,
- 39:21I know Heather and I talked about.
- 39:24You know, kind of how this could
- 39:26be incorporated into CV Part 2,
- 39:28especially when there's an improvement
- 39:30and kind of your thoughts about
- 39:33sitting at that table, you know,
- 39:35in terms of promotions and having
- 39:37this as a piece of somebody's success
- 39:40as a teacher and self improvement.
- 39:45So there's a question for you,
- 39:46Janet, about kind of the,
- 39:49the potential utility,
- 39:51I think ideally. We
- 39:53should be relying on peer
- 39:56observation and learners.
- 39:58So if you triangulate the data,
- 40:00because it's fabulous data,
- 40:02but we'd also have to be very careful.
- 40:04So when I do my observations,
- 40:06I'm very careful if someone's
- 40:08asking me to also write a letter
- 40:10to go into their folio versus the
- 40:13observation as a learning piece.
- 40:20And I think also what Franz Gellerno did
- 40:23in OBGYN, she observes all the faculty.
- 40:27We did a couple of years of training
- 40:29with her, and then she observes all her
- 40:31faculty and then writes a letter to them.
- 40:34And they were so grateful about that.
- 40:36So again, there are ways that we can
- 40:39really buff this up and peed so it,
- 40:41you know, because it's just such a great
- 40:43program that you're doing. Heather.
- 40:46We do went for the the 20 faculty
- 40:48that have completed the observation.
- 40:50We do send them a completion letter that
- 40:53it says you know thank you for completing
- 40:56this top program with two observations.
- 40:58You know in the state in this state
- 41:00your you know your teaching score
- 41:03increased this to this your your
- 41:05sense of selfefficacy change this
- 41:07way and then we highlight some of
- 41:10their their teaching strengths.
- 41:12And so it is signed by Panina and
- 41:15myself for them to, you know,
- 41:17include in their promotional
- 41:19materials if they desire.
- 41:23I've done the top program,
- 41:25so I'm not a person who didn't.
- 41:27But I will tell you,
- 41:28I think it's all social, right?
- 41:30So it's just this weird social
- 41:32barrier of signing up for something
- 41:34that you're like, what even is this?
- 41:36And is this for me?
- 41:38And by signing up, am I admitting
- 41:40that I think I'm a bad teacher?
- 41:42And so I suspect,
- 41:42and I don't know if this is true,
- 41:44but I suspect you probably got a lot of
- 41:45people in the first pass that are like,
- 41:47I feel okay about my teaching,
- 41:48but come on in, teach me more, right?
- 41:50Because you have to have a certain
- 41:52degree of selfconfidence to
- 41:54volunteer for this type of feedback.
- 41:56So if you're meeting people
- 41:57being a little timid,
- 41:59that's why I suspect it's a little like
- 42:01they don't want to be the the nerdy kid.
- 42:04Who signed up for the program too soon?
- 42:06But they also were just feeling nervous
- 42:08about what they might hear and whether
- 42:10or not they're going to find out what
- 42:11they thought they were pretty good at.
- 42:12Maybe they aren't so good at.
- 42:15That's just my impression of how residents
- 42:17interact with opportunities like this.
- 42:19That,
- 42:20I suspect is also true for junior faculty.
- 42:23Also, Heather, I think you know,
- 42:26one of the every time I'm asked
- 42:29to participate in something,
- 42:30I'm always like balancing out how
- 42:31much time is it going to take.
- 42:33You know,
- 42:34how is it going to interfere
- 42:36with my clinic schedule,
- 42:37all the other things I have going on?
- 42:39So I I feel like I don't want
- 42:41to commit to something and not
- 42:43have the time to really kind of,
- 42:46you know, foresee the end for it.
- 42:48So I think my hesitation is always like,
- 42:52do I have the time and the
- 42:53day to really get this done?
- 42:56Kind of, yeah.
- 42:57Ruchika. Oh, I was just going to say I think.
- 43:02The the other side of it,
- 43:03I had signed up and then I did not
- 43:05have an observer join the sessions.
- 43:08I'm sure they're more like
- 43:09happening at your end, Heather,
- 43:11and how you are kind of.
- 43:13It's more like a comment and a
- 43:15question because I I did volunteer,
- 43:17but I did have my observer
- 43:19come in for that session.
- 43:22Yeah, I think Heather,
- 43:22did you a pilot arm.
- 43:23So I know I had some feedback. So
- 43:27your lecture we're actually
- 43:28using to validate the tool. So
- 43:30we'll watch your thing over and
- 43:33over again. Got it. Don't worry.
- 43:37Yeah. Well, I haven't seen your
- 43:38name on any upcoming lectures
- 43:40that we'll get you in. Yeah,
- 43:42yeah. Yeah. No. So I just,
- 43:43I feel like there's also,
- 43:44I guess time constraints on the
- 43:47observers that are also offering
- 43:49to sign up for the program, so.
- 43:51Yeah, it's true because that, you know,
- 43:53I do receive some protected time for this,
- 43:55but I'm asking other to faculty to
- 43:57help me observed, which is, you know,
- 43:58a time constraint on them that they're
- 44:00not getting protected time for.
- 44:01So yeah, it's true.
- 44:06Yeah. And another thing, yeah,
- 44:07I think time is a major thing.
- 44:09You know, when faculty know
- 44:11that they're being observed,
- 44:12they really want to take the time to
- 44:14make this the best luxury that they can,
- 44:16which again, you know, is time consuming,
- 44:18you know, to revamp a lecture
- 44:20that you've already given.
- 44:21So I think, yeah,
- 44:23that being a little feeling vulnerable is,
- 44:26is one major issue and then the other is,
- 44:29is just time to prepare and time to
- 44:32participate in the program. I mean,
- 44:34I certainly try to make it clear that.
- 44:36You know, we're just there observing.
- 44:38We're going to send you this sheet.
- 44:39If you're open to 15 minutes
- 44:41of feedback via Zoom,
- 44:42then we do that.
- 44:43And and that's pretty much
- 44:45it in terms of your time,
- 44:47except for filling out those 2P TSQ
- 44:49surveys at the beginning and the end,
- 44:52which takes just a couple minutes.
- 44:53But I certainly recognize that this is
- 44:57a voluntary program that they don't need
- 44:59to participate in if they don't have to.
- 45:01So it's when you're busy.
- 45:04And it's not a priority,
- 45:05but probably won't be taken advantage of.
- 45:08So I have two followup questions for that.
- 45:10And that'll let everybody go because Janny,
- 45:13you actually raised something that
- 45:14I think is really interest, well,
- 45:16raised many things that are interesting,
- 45:18but that I was struck by and I'm not sure
- 45:24we deliberately address, which is 1.
- 45:27The idea that it's formative and
- 45:30not summative that it doesn't.
- 45:32That it doesn't necessarily go anywhere.
- 45:35And so Heather, do we or do you address that,
- 45:40I mean specifically to say that you know it,
- 45:44it won't go anywhere,
- 45:45it's just for your own personal,
- 45:49you know, kind of self efficacy.
- 45:52Yeah, I mean I don't think so.
- 45:56I I thought it was implied,
- 45:57but now that you mentioned it,
- 45:58yeah, now that it was a specific statement
- 46:00that. This is for your eyes only.
- 46:03Yeah. So that's a really good point.
- 46:06And then secondly,
- 46:07I'm just curious about the group. I mean,
- 46:09just in terms of your thoughts or feedback,
- 46:12if this actually just became
- 46:14standard for all junior faculty,
- 46:16Like if it was clear that this is formative,
- 46:20not summative, but it was expected
- 46:23of all junior faculty when they come,
- 46:27What do you think?
- 46:28The what would be your sense
- 46:30if you were a junior faculty?
- 46:32How would you perceive this,
- 46:34Nicole, if you're still on when
- 46:35you're a junior faculty,
- 46:37how you would perceive this?
- 46:41Any any thoughts?
- 46:45I just have one quick thought and that
- 46:47I'm I'm giving the fame presentation
- 46:49at once, so I'm going to sign off.
- 46:51I I think it's actually really
- 46:53appreciated if they know that
- 46:55it's formative and very safe.
- 46:57And I think you really have to be so
- 47:00careful with training your feedback givers.
- 47:04It takes such and to really have a
- 47:07dialogue versus I'm giving you feedback.
- 47:10So I would even say, you know,
- 47:11make sure that you're checking in on the
- 47:14people who are trained to do this because
- 47:16if it doesn't feel safe or helpful.
- 47:20So you know it's interesting where
- 47:22the three positive and and the three
- 47:25areas of improvement came from.
- 47:26Sometimes you just feel that you
- 47:29know you are giving too many sort of
- 47:33critical pieces again thinking about
- 47:36improvement and focusing on just maybe
- 47:39even one area that is correctable.
- 47:44So I think it'd be great we
- 47:46could try it. Yeah, thanks.
- 47:47I'm here and I I can add something and
- 47:50sorry Nicole and I have
- 47:51to run just because
- 47:54I'll see you at one again Janet, I'll be.
- 48:00So I was just going to add,
- 48:02I think it's something that's is
- 48:04very valuable and I actually had
- 48:06Heather give me feedback from for
- 48:09one of my presentation with the
- 48:12residents and it was very useful
- 48:14especially for my thesis like.
- 48:16I developed a curriculum.
- 48:18I implemented the curriculum,
- 48:19and as part of that evaluation,
- 48:21I need to know how well I'm doing.
- 48:25So it was useful and I really,
- 48:26really appreciate it.
- 48:27So, and I didn't even know about this.
- 48:30I forgot the four letters of the 11 question.
- 48:34Oh, PTSQ. Yeah. Yeah.
- 48:37The PTSQI think it's also a great idea.
- 48:39So I I'm going to do it, Heather.
- 48:42OK. Thank you.
- 48:46I was, I was gonna say if we just
- 48:48implemented it as a standard for
- 48:50every new faculty to come in,
- 48:52I think it should go well.
- 48:53I I don't see a reason why not.
- 48:56I do think it's a great program
- 48:59and at least for those who are
- 49:01really motivated to remain on this
- 49:03educational kind of track or are
- 49:05interested in medical education,
- 49:07I don't see why it they would
- 49:10not be willing to participate.
- 49:13Other than the other constraints of time.
- 49:15And yeah, of course.
- 49:16Right. Talk about. That's
- 49:19great. Any other thoughts?
- 49:23I think it'd be interesting
- 49:24to talk to not just about the junior faculty,
- 49:28but some of the faculty who have
- 49:31been teaching for a long time and
- 49:33maybe could use some updates.
- 49:34And in terms of incorporating more
- 49:37interactive components of their lectures too,
- 49:40but that could be a different discussion too,
- 49:42yeah. Don't totally agree, Heather.
- 49:46Great. OK. Well, thank you so much,
- 49:48Heather. Thank you for all you've
- 49:50done for our department and in
- 49:52terms of faculty development.
- 49:53And thanks for updating us
- 49:55about your terrific work.
- 49:56We really appreciate it. Thanks.
- 49:58I appreciate everyone's feedback. Thank you.
- 50:00Great. See everybody in a month.
- 50:03All right, take care. Bye, Bye.