Panel 2: Rethinking Assessment: When Virtual Care is the Vehicle and the Environment of Practice

July 19, 2023

ID10142

To CiteDCA Citation Guide

00:00So we're going to shift gears now and
00:02our second panel is entitled rethinking
00:04Assessment when virtual care is the
00:06vehicle and the environment of practice.
00:09So it's a really interesting challenge
00:10we alluded to at the beginning.
00:12We're very fortunate to have
00:14Doctor Topher Sharp from Stanford
00:16Healthcare to serve as our moderator.
00:18He's the CMIO at Stanford Healthcare where
00:21he and I met actually through our mutual
00:26engagement and investment in telehealth.
00:29He has been a key leader in the
00:31Stanford system in adoption of the EHR.
00:33He runs the Clinical Informatics Fellowship.
00:36He's a general internal medicine physician.
00:39And it's really been at the forefront
00:42of trying to understand how to
00:44engage the participants in a digital
00:48interaction that brings value.
00:50So I'm really delighted to introduce
00:52Topher and then he will introduce
00:53the panel and kick off the session.
00:55Topher,
00:55thanks
00:57so much, Lee.
00:58And from out in California,
01:00I wish you an orange.
01:01Good morning to all of you.
01:04I hope that you're able to hear me.
01:07We are able
01:07to hear me. Yeah, perfectly. Dover,
01:10fantastic. Thanks so much.
01:11So it's really, really exciting to
01:13spend some time with you here this
01:15morning and I'm really just genuinely
01:17excited to bring forward such a
01:20fantastic panel for discussion today.
01:22So as as we mentioned here,
01:24we're going to be looking at assessment
01:26and rethinking what assessment means.
01:28Both in the context of the new competencies
01:31around virtual care as well as when
01:34virtual is that environment of practice.
01:37How does assessment occur in the
01:39most effective ways for the for the
01:41care that we're already delivering,
01:43thinking about all the different aspects
01:45of assessment that are really going to
01:47be requisite in this in this new normal.
01:49Let's proceed to the next slide if we
01:52would and I want to just introduce
01:54the objectives for the session and
01:56then introduce the participants.
01:58So in this session we're going to
02:00really try to give you something
02:01that you can take away, take back,
02:03chew on as well as give us feedback about
02:06and that will start with delineating the
02:09best practices for developing assessment.
02:11So how would,
02:12how would one even start,
02:14how are we starting to do that
02:16locally and nationally?
02:17As we really work through the processes
02:19of understanding and developing
02:20what best practices look like,
02:22then we'll dig in and really list out.
02:26Such that you can pilot assessment
02:28strategies to go through the iterative
02:31learning cycles and identify for telehealth
02:34what those assessment strategies can be.
02:37Third,
02:38we'll describe the facilitators
02:39and the barriers in implementing
02:42these assessment strategies.
02:44This will be really helpful because
02:45we'll dig in and really explore
02:47what it means to make this real,
02:49like what what are the real challenges
02:51and what are the ways in which we
02:53can really break those through.
02:54And then last,
02:55we'd like to spend some time
02:56with the entire group here,
02:58identifying some scholarship
03:00questions in assessment.
03:02How do we really start to advance,
03:05if you will,
03:05the science of assessment as we look
03:07forward into the future in telehealth?
03:10So this will be our agenda here over
03:11the course of this next hour or so.
03:13We're really excited to have the time.
03:15Let's go ahead and I'll introduce
03:17some of the folks here that
03:18are going to be with us today,
03:20which we're super excited about.
03:23So I'd first like to introduce
03:26Dan W Dan is a Professor and an
03:29Associate Chair for Education in
03:32the Department of Pediatrics at
03:34the University of Pennsylvania.
03:36He is as as a matter of fact,
03:38a distinguished endowed chair and
03:40the Director of Medical Education
03:43and the DIO at SHOP.
03:45Dan has two decades of experience
03:48developing educational innovations.
03:50Across the the spectrum of education
03:53from undergraduate through
03:54graduate medical education,
03:56he is very highly published especially in
04:00areas of focus including competency basis,
04:02education and assessment as well
04:04as educational innovations that
04:06reduce handoff related communication
04:09failures and promote patient safety.
04:14So we're looking forward
04:16to hearing from Dan first.
04:17Then next will be Shruti Chandra.
04:21Shruti is an assistant professor
04:23of emergency medicine at
04:25Thomas Jefferson University.
04:27She is the phase three director
04:29for the Medical College there,
04:31so she's responsible for the latter
04:33third of medical of the medical
04:35school curriculum she's become
04:36involved with in telehealth as
04:39both a telehealth practitioner.
04:41And also a developer of the telehealth
04:43educational curriculum at Thomas Jefferson,
04:46where she is the program director
04:48for telehealth and the digital
04:50health education programs.
04:52Next slide, if you would.
04:56Next is Marcy Bolster.
04:57Marcy is the director of the
04:59Rheumatology Fellowship Training
05:00program at the Mass General Hospital,
05:03where she's also an active clinician.
05:05She's been the fellowship program director.
05:08AFO to Program Director since 1999,
05:10first at the Medical University of South
05:13Carolina and then at MGH and has a very
05:16strong interest in medical education.
05:18She participates in the teaching
05:19and training of medical students,
05:21internal medicine residents,
05:22rheumatology fellows.
05:23She is a member of the Women
05:26in Medicine Trainees Council.
05:27And importantly,
05:28Marcy has demonstrated a lot of
05:31leadership at local and national levels.
05:33She has served in a leadership role
05:35in many of the projects with the
05:38ACR that the American College of
05:40Rheumatology A/C GME and the A/B I M.
05:43In a CMGME,
05:45she works to establish a Milestones
05:48Project and the Rheumatology
05:50Milestones 2.0 and she's she's
05:53has been committed to facing gaps
05:55in the rheumatology workforce.
05:58She's a recipient of Rheumatology Research
06:01Foundation Clinician Scholar Grant.
06:03Called Extending Our Reach,
06:05which is about providing telemedicine
06:07training to rheumatology fellows
06:09and providing care to underserved
06:12patient populations.
06:13And last, I want to introduce John Scott.
06:16John, besides being a former colleague,
06:19is also a professor in the Department of
06:22Medicine at the University of Washington.
06:24Where he is in the Department of Division
06:27of Allergy and Infectious Diseases,
06:29John is also the Medical director for
06:33UW Medicines Digital Health Programs.
06:36He has a research focus in viral
06:39hepatitis and in the use of telehealth
06:42technologies to improve patient care.
06:45John in 2009 launched Project
06:48ECHO in Washington state.
06:50Which is an innovative telehealth program,
06:53if you're not aware,
06:54a fascinating program that connects
06:57clinicians that serve rural
06:59populations and underserved areas
07:00with experts in the evaluation and
07:03management of specific conditions.
07:05So it's really a teach the
07:07teach the clinician,
07:09teach the teacher program.
07:10In 2013,
07:11he became the first medical director
07:13for telehealth at University
07:15of Washington and has created
07:17innovative online learning platforms.
07:19Along with in partnership with some
07:21of his other collaborators there.
07:23And finally John leads the Washington
07:25State Telehealth Collaborative
07:27where he's worked with that group
07:28to tackle thorny issues,
07:29including recent law that requires
07:31all clinicians in the state to be
07:34trained on telehealth competencies.
07:36So we're looking forward to
07:37hearing a little more about that.
07:39So we've got just a fantastic panel.
07:41Really super excited to dig in here
07:43and and have them speaking with you.
07:45And with that,
07:46I'd like to turn it over to
07:47our first discussion,
07:48Dan W,
07:49who will bring us into the
07:51discussion of best practices for
07:53developing assessment in telehealth.
07:55Take it away,
07:56Dan,
07:56thank you
07:57so for thank you so much and for that very
08:01generous introduction and good morning
08:04or good afternoon to our participants and
08:06what a great pleasure it is to be here.
08:09I hope that what we'll talk about in
08:12this session will build on the questions
08:15and the topic that preceded this one.
08:18I think that it will,
08:20so I'd like to start by really reviewing
08:23just a few concepts and basic best best
08:28practices for approaching assessment.
08:31I realize some of you may
08:33be very familiar with this.
08:34So please bear with me.
08:36Some of you might not be.
08:37So that's the purpose of going
08:39through some of these concepts.
08:40But to start with,
08:43in terms of assessment,
08:45a key issue is that you need to
08:47start with a competency model.
08:49So you just saw 1A beautifully
08:53developed one for telehealth described
08:55in the previous session and.
08:59There's a really significant amount
09:01of validity evidence that supports
09:03that content validity evidence when
09:05you look at the process that the group
09:07went through and developing that.
09:09So Ioffer my congratulations to
09:11the work that the group did and
09:14the sponsorship by the AA MC.
09:17There are different types
09:19of competency models.
09:19We touched on that a little bit before and
09:22to me when we think about implementing.
09:25Competency models.
09:26The question is what are
09:28we implementing it for?
09:29What's the purpose of that?
09:30What am I going to do with it?
09:32And one chooses the model based
09:35on what the purpose would be.
09:37So what you saw described and what the A/C,
09:41G&amp;E competencies are, for example,
09:43are reductionist models.
09:44So I'm going to break this task,
09:47this global,
09:48more global task of being able to
09:50function effectively in in in during
09:53a telehealth visit, let's say.
09:54I'm going to break it down into its
09:56component parts and I'm going to
09:58try to understand what those are.
10:00I'm going to organize those activities
10:02in domain so I can understand it,
10:04so that I can keep it organized in my head.
10:07And then I'm going to do something with that.
10:10And the things that you can do
10:12without our developed curricula.
10:13You can assess people, you can,
10:15you can set competency standards
10:17based on those sorts of things.
10:19You can use it to help people
10:20who are struggling.
10:21In doing certain activities to try to
10:23drill down on exactly what the problem is,
10:26all of those issues came up in the
10:28questions that were asked before.
10:29So reductionist models have
10:32some great advantages.
10:34First of all,
10:34they're widely used in the US and
10:36Canada and throughout the world.
10:38And and usually it's the assessment
10:40side of that is based on some sort of a
10:43competency scale from novice to expert.
10:46And that's what you saw described.
10:48Previously usually have behavioral
10:50anchors and this and the the model
10:54you saw presented has those and
10:56often times those are at least
10:58people effort is made to try to
11:00make those evidence based.
11:02The challenge of those models
11:03is that they can be
11:05abstract and be very detailed and they
11:08sometimes can lack clinical context and
11:10be hard to apply in that context so.
11:14Models have their have their
11:17advantages and their disadvantages.
11:19Other types of models in competency
11:21are more global models and EPA's are
11:23a really good example of that and
11:25trustable professional activities.
11:26So the EPA's helped inform the development
11:30of of the model that you saw described.
11:35There's been EPA's developed for medical
11:37students as they transition between being
11:39a student and then being a resident.
11:42There have been EPA's developed
11:44in specialty fields.
11:45The American Board of Pediatrics
11:46has been a leader in developing
11:48intrustable professional activities
11:49for the practice of general Pediatrics
11:52and their subspecialties.
11:53There have been other fields
11:55that have also developed them.
11:57Typically what EPA's do is they
12:00synthesize these reductionist,
12:02reductionist
12:05competencies and bring them together
12:08into a smaller, more manageable.
12:11Number of competencies,
12:13if you will, to try to make it
12:19more manageable for practicing
12:21clinicians, for trainees,
12:23for their supervisors and teachers.
12:25The assessment in in,
12:26at least in EPA's,
12:28is based on a supervision scale.
12:29How much supervision does someone need?
12:31That's a very intuitive scale
12:34in graduate medical education.
12:35That is the basis of graduate
12:37medical education You're
12:38trying to move from being in a.
12:40In a highly supervised state
12:41to being able to do something
12:43unsupervised, they're
12:47the the
12:50EPA's are usually,
12:51I notice the typo on my slide,
12:53usually readily observable and
12:55they're related to clinical context
12:57and they're they have this intuitive
13:00nature and they're useful framework
13:03for making both summative decisions
13:05and providing feedback to help.
13:07Help learners progress as they're
13:09moving and trying to learn to develop
13:11this competency and to do something.
13:14Whatever model you use,
13:15it's important to have one.
13:17Often times they'll be a need to have
13:19more than one and that's the really great
13:21thing about EPA's and at least the A/C,
13:24GME competencies and I'm sure
13:25it relates it'll relate to
13:26these telehealth competencies,
13:27is that those are related to each other.
13:29You can understand how they
13:32can integrate together.
13:33Next slide please.
13:37So regardless of the model that
13:40you choose, you can think about
13:43competencies as a journey.
13:45So if you look at this particular
13:48figure was this is a modification
13:51of figure that Olatan Kate
13:53uses when he talks about EPA's.
13:55And Olaf, you don't know know who he is.
13:58He's really the developer
14:00of the concept of EPA's.
14:03He's based at University of
14:04Utrecht in the Netherlands.
14:06And he came up with this concept
14:08and has studied it extensively
14:10over the last decade or more.
14:13But if you think about each of those
14:16competencies that you saw described earlier,
14:19that there's a progression.
14:20So when someone really starts out
14:23trying to do that competency,
14:25they're functioning at a relatively
14:26low level and then they gain skill
14:28over time and there's a slope,
14:30there's a a continuousness to that,
14:33that process.
14:34And which you could describe it as a slope,
14:37a rate at which you actually
14:40achieve that competency.
14:41So what this diagram shows is
14:43on the Y axis is competency or
14:45the level of competency.
14:46So you can think about that
14:48from the novice to expert range.
14:50And then along the X axis is time.
14:53So you're spending some time in GME training,
14:57you're getting feedback and you're
14:58learning during that time and
15:00then at some point you're going
15:02to transition out of that formal
15:03environment into something else.
15:05And on the competency scale in the Y axis,
15:07there's some level at which one
15:09could draw a conclusion that that's
15:11good enough and you're now able to
15:14do that thing in a reasonable way.
15:16But that doesn't mean you don't
15:17continue getting better at it.
15:19Over time,
15:20we're developing more competency
15:21and so these are these evidence
15:24based competency decisions or
15:26thresholds and you can see these
15:29models here with competency 1.
15:31Being achieved at a faster rate and
15:33crosses this competency threshold
15:35faster than competency 2 than competency 3.
15:38And my point here is that assessment
15:41should be designed to deal with
15:44these different purposes and help
15:46inform this whole process as you're
15:48trying to learn to be competent
15:50in a particular area,
15:52so assessment especially is important.
15:54In this area of the GME training
15:56and feedback and learning because
15:58it's going to help guide learning
16:00and then at some point that comp is
16:02going to be important for making a
16:04decision that someone's ready to move on.
16:05It's really clear that you need
16:07to be really clear about what
16:09you're using it for and why,
16:11and that's going to guide the kinds of
16:13assessments that you're going to do and
16:15how rigorous they're going to have to be.
16:17Next slide.
16:21So if you're thinking about an A
16:23general some general strategies
16:25for an approach to assessment.
16:27What I put over here on the right
16:29hand side of the slide is a figure
16:31from a paper that was published
16:34a few years ago from Culture and
16:36the group that was working on the
16:40on the AM A education projects.
16:43About this concept of Master
16:45adaptive learning which is builds
16:47on on concepts of mastery learning,
16:49growth mindset which is really where
16:51education much more broadly than
16:53medical education but definitely
16:55medical education is focusing now this
16:58idea that I'm trying to learn to do
17:00something and I'm approaching that to
17:02try to develop mastery at doing it
17:05as opposed to a performance mindset.
17:07So performance mindset is.
17:10It has, it is sort of centered on this
17:12idea that that learning is a fixed
17:15concept that once I've achieved competency,
17:18I've achieved competency and it
17:19stops there and my mind is fixed.
17:21So, so the pushes is much more towards
17:25the master adaptive learning which
17:27is consistent with adult learning
17:29models and that sort of thing.
17:30So assessment plays an important
17:32role in this because it's providing
17:35data and information into this
17:37model of learning and and growth.
17:40But and then it can also provide
17:42information when when high stakes
17:44decisions have to be made about
17:47whether someone reaches competency.
17:48So one of the things going
17:51through the various strategies,
17:52it's important to remember
17:54what we already know.
17:55And so I would adapt and build off
17:57of assessment tools and assessment
17:59strategies that we already know work.
18:02So you might have to modify and adapt.
18:04But you don't have to reinvent things
18:06and then be guided by competency models,
18:08or a model,
18:09at least one and pick a good one,
18:11Pick a useful one.
18:13And then the validity and impact
18:15of assessment is based on the
18:17conclusions you're going to draw.
18:18So if it's for for formative purposes,
18:21the validity evidence doesn't
18:21have to be so strong,
18:23but you're going to make
18:24a high stakes decision.
18:25For example,
18:26if you were going to develop
18:28a telemedicine board and have
18:30a board certification exam.
18:31Then you'd need to have a lot of
18:33validity evidence if you're going to
18:35set up pass fail standards for that.
18:37And then assessment should support
18:38learning as I just described
18:40and competency decisions.
18:41It's a both and not an either or.
18:44And this idea that there's no
18:46single assessment that's sufficient
18:48use multiple forms of assessment
18:50and that assessment in telehealth
18:52like any other is part of a larger
18:54program of assessment.
18:55So next slide and this is my last slide.
19:00So this idea of a program of
19:02assessment that can assess learning
19:04and and be assessment for learning,
19:06and this is a slide from a work that
19:08Case van der Vluten has published.
19:11If you don't know him,
19:12Case van der Vluten is is probably the
19:16foremost assessment and psychometric
19:19medical educator and probably in the world.
19:22And he's based at the University of
19:24Maastricht, also in the Netherlands.
19:27And he has developed this model of assessment
19:31for and of learning and this idea that
19:35the goal is to have multiple assessments,
19:38typically multiple low stakes
19:40assessments with where any individual
19:43assessment informs learning,
19:45and then in the aggregate those assessments
19:47are used to make summative decisions.
19:49So that when you set up assessment
19:53programs that are low stakes,
19:55you drive people towards mastery learning.
19:58When you set up assessments
19:59that are high stakes,
20:00you drive them towards
20:01performance based learning.
20:02So that's the idea.
20:03So when you think about
20:05developing your assessments,
20:06thinking about what you're using
20:08them for and how you can design them
20:10to achieve both of these goals and
20:12I think I'm going to pass it off,
20:15I think shrewdy,
20:15you're next.
20:20Yes, I am. Thank you very much.
20:22Good morning, good afternoon.
20:24Wherever you are in this world,
20:26I'm going to move on and talk a
20:28little bit more about the details,
20:30the operationalizing of how to
20:33really perform these assessments.
20:35So what I want to talk about first is
20:38how do you really assess A learner's
20:41performance in telehealth and how do
20:44you create these assessment strategies,
20:47so your assessment strategies
20:48can be based on?
20:49Any competency model that you choose to use,
20:51just like Dan mentioned,
20:52and he's mentioned a few of them already,
20:55so you need to just decide what
20:57part of your assessment is being
20:59is being used and what kind of
21:02assessment you want to use.
21:03Telehealth encounters lend pretty well to
21:06both formative and summative assessments,
21:08so you could use your formative
21:12assessments by giving them feedback
21:14versus summative by using checklists,
21:16which we're all familiar with,
21:17or global rating scales.
21:19You know,
21:19checklist prompt raters to attest
21:21to the performance or a mission
21:24of directly observable actions.
21:26Whereas global rating skills,
21:28they typically ask raters to judge
21:30learners overall performance or
21:33to provide global impressions
21:35of performance on subtasks.
21:36So it depends on what you're
21:38looking for to assess and which
21:39ones you might want to use.
21:40Can we go to the previous slide?
21:46I'm sorry, Next slide? From my next slide,
21:48again that missing the slide,
21:50but that's Okay. Let's go back
21:52to the previous slide. Thank you.
21:56Another thing that you want to think
21:58about is you know what are the tools for
22:01assessment that you're going to use?
22:02There are many that are already
22:03used in medical education,
22:04so you can easily adapt
22:06whichever tool you would like.
22:08So how do we really put this into
22:10practice for telehealth assessments?
22:12There are a few points to
22:14take into consideration.
22:14And then keep in mind that whichever
22:17method you might use might work
22:18better for some rather than others,
22:20depending on the type of
22:22setting that you're in.
22:23Whether that be inpatient versus outpatient,
22:25whether that be with students or residents,
22:28whether the students or residents
22:30or the trainee are within the
22:32same space as you versus remotely.
22:34The easiest option to try in my mind is to
22:37model your telehealth assessment workflow.
22:40As closely as possible to your
22:42current in person assessment workflow.
22:45So here are just two options,
22:47very broad options that we can talk
22:49through and see which works best for you.
22:51In workflow A through telehealth
22:53you actually have the ability to
22:55directly observe your trainees,
22:57which is very similar to a bedside
22:59observation of a clinical encounter.
23:02To perform a direct observation,
23:04your platform will require the
23:06ability to have multiple users
23:08on the visit at the same time.
23:10Specifically,
23:10if the student is not in the same space,
23:13and presumably your patients
23:14not in the same space,
23:16you could decide to be visibly
23:19present throughout the visit as you
23:21would if you were there in person.
23:23You can also decide to have your
23:25microphone and video turned off
23:26while observing the student.
23:28Kind of a fly on the wall option.
23:30Which can only really be afforded
23:32to you by telehealth.
23:34Now,
23:34this whole workflow approach requires
23:36more of the time on the clinician's part,
23:39on the supervisor's part,
23:40but it also may be affording you
23:42much more robust supervision than
23:44you currently provide your trainees,
23:46even the live environment.
23:47And how many of us are really watching
23:50our students or our residents perform
23:52their whole entire visit from start
23:55to finish for every visit that
23:57they perform with their students?
23:58Not not so many.
23:59So this does provide you the option
24:02to do a much more robust assessment
24:04in lieu of the entire visit being
24:07directly observed by you.
24:08You can choose to observe only a
24:10portion of the training interviews
24:11such as maybe the assessment and the plan,
24:14and this could be time saving
24:16for the clinician.
24:17Let's move on to the next workflow.
24:19This other model,
24:20it follows what most in person
24:22visit workflows would follow,
24:24which is to have the student complete their
24:27assessment step out of the virtual room.
24:29And then present this patient and the clinic.
24:33The clinic visit to the
24:35supervisor or the faculty member.
24:37The supervisor then then complete their
24:40own assessment in the room while the
24:42trainee can start on the next patient.
24:45This is time saving follows closely many
24:48people's already established workflows.
24:51However, this format will require
24:53consideration of your timing of the feedback.
24:56As to when it is that you're going to
24:58be providing your pre needed feedback,
25:01next slide please.
25:04So this really brings us to our next topic,
25:06which is the timing of your
25:08feedback and evaluation.
25:09Now each person,
25:10each program should think of a few things.
25:13The frequency of providing feedback
25:16you made to provide feedback
25:18after each encounter or after a
25:20block of encounter such as midway
25:22through your day or end of the day.
25:24But you need to ensure that you have
25:27dedicated time with the training
25:29alone to provide this feedback.
25:31You'll also need to build in some time in
25:34case you need to provide feedback on the fly,
25:36in case the student needs to remedy
25:39something before their next encounter.
25:41This is easier when you're in workflow A
25:43when you're with the student the whole time.
25:46However,
25:46it is more difficult when
25:48you're in workflow B.
25:49So workflow B might be more suited
25:51to advanced learners like senior
25:53students or senior residents.
25:55From an overall standpoint,
25:56a decision really needs to be made as to
25:59whether a student is deemed proficient in
26:01telehealth after a certain number of visits,
26:03similar to procedural training?
26:05Or does this require ongoing evaluations?
26:08And I feel like we've touched upon
26:11this topic on the last panel?
26:13Should there be some recertification
26:15requirements and maintenance of competency,
26:17not just for faculty members,
26:19but for ongoing assessment for your trainees,
26:22your students,
26:23and your residents?
26:24Lastly,
26:25it's also helpful to consider
26:27the mode of assessment,
26:29which means how will you be
26:31providing your assessment.
26:32You could do a followup phone
26:34call to the training,
26:35which can be a time efficient and can also
26:37occur closer to the time of the encounter.
26:40And we all know that timely
26:42feedback is probably the most
26:43important thing that you can do.
26:45Video feedback, on the other hand,
26:47has obvious advantages,
26:47such as getting a feel of the
26:49trainee and their body language,
26:51understanding how they're
26:52really receiving this feedback.
26:54But it may be more time consuming.
26:57You could remain on video after
26:59the patient has signed off,
27:01and that could be part of your workflow.
27:04Next slide,
27:06please.
27:07There are some further considerations
27:08that I want to bring to your attention
27:10that we should think about that may
27:11not have perfect answers but are
27:13things that you want to consider
27:14when you just when you figure out
27:16your plans and your workflows
27:18for assessment for your trainees.
27:20Telephone calls which are part of
27:22telehealth can will assess very
27:23different skills than a video visit,
27:25so they can assess things like
27:28communication skills and and
27:31skills of how are you.
27:33Dealing with some technical hiccups,
27:34but you may not be able to assess
27:37your physical exam skills over
27:39telehealth for your training.
27:40Keep in mind also that assessment
27:42with telehealth visit will take
27:44longer since you're not only
27:46assessing their medical performance
27:47but you're also assessing their
27:49telehealth performance during the same
27:51visit. So it would be very helpful to
27:54set goals ahead of time to ascertain
27:56what you're assessing per visit per day.
27:59Is this visit a telehealth
28:01visit and the assessment is.
28:02Purely telehealth related?
28:04Or is this visit a medical visit that you're
28:07going to be using telehealth to assess?
28:10You can also decide to have patient
28:11feedback as part of the assessment,
28:13and you can decide if you want to do
28:15that at the end of the day or at the
28:17end of a certain number of visits
28:19during that the day with your training.
28:22Also keep in mind that a traditional training
28:24may be more proficient in some aspects,
28:27for example in technology and
28:29troubleshooting for telehealth,
28:30than the supervisor themselves.
28:31This bring back the question about
28:34how much faculty development is
28:36required before you can even create
28:39these assessment strategies and
28:40to assess your trainees.
28:42Next, Next slide please.
28:47It would be remiss for us to not
28:49consider that telehealth is useful as
28:51a tool in assessing medical knowledge,
28:53skills and attitudes itself.
28:55So we've talked so far about how we
28:58would be assessing telehealth itself.
28:59But a lot of the topics and then
29:02tips that we've talked about can be
29:05used for assessing medical knowledge
29:07and skills through telehealth.
29:10In the situation where a supervisor
29:12or an examiner is at a remote
29:14site or student cannot be present
29:16for in person clinical visits,
29:18telehealth can be an excellent tool
29:19to use for assessing clinical skills.
29:22In fact, during the COVID pandemic
29:24when your students were not allowed
29:26to be in the clinical environment.
29:28And not allowed to have any face
29:31to FaceTime with their patients.
29:33This was a great way to still assess
29:36their skills and to meet their goals
29:38and objectives for their rotations
29:40on the clinical aspect of things.
29:42Similar to any in person visit,
29:44the supervisors can assess their history,
29:46taking skills, physical exam techniques,
29:48professionalism,
29:49interprofessional communication
29:51skills and a variety of other.
29:55Competencies,
29:56assessments that you might be
29:58doing in any in person visit.
30:00Additionally,
30:00you can also use telehealth to assess
30:03and supervise procedural skills remotely.
30:05So you could set up a procedural lab
30:07where the students and residents
30:09are performing the skills and your
30:12supervisors at a remote location
30:14and can virtually monitor these
30:16procedures and assess and give
30:18feedback to any of these skills.
30:20You may also consider using
30:22telehealth for oski assessments.
30:24Oski's are the objective
30:26structured clinical examinations,
30:28such as in times when there's
30:30a large number of trainees,
30:32but they cannot gather in a small space.
30:34This could be useful for standardized
30:37patients if they cannot be at the
30:39exam site and cannot travel there,
30:41or if your examiner's at a remote location.
30:44So combining virtual assessments
30:46through Oski's is just another use
30:49another another use of telehealth
30:51as a tool for assessment.
30:53That the next slide,
30:54so I'm actually going to move on
30:56to our next speaker. Thank you.
31:08Hello, I'm Marcy Bolster and
31:11this is transition beautifully.
31:13Thank you Shruti and thank you Dan.
31:16I thought I would start with my
31:19section of the discussion is really
31:21to bring forward facilitators and
31:24barriers to assessment and telehealth.
31:27And I thought I would take a step
31:29back and this somewhat addresses the
31:32discussion that ensued during the
31:35first panel discussion that Lee was
31:37leading and that was thinking about
31:40who should be involved in telehealth
31:42and how do we use telehealth and
31:44something that has really helped me
31:47in this transition into telehealth.
31:50Is thinking about it as a way to
31:53enhance rather than replace the care
31:55that we provide to our patients.
31:57And I think what happened in
31:59March for many or most of us is
32:02that telehealth really became the
32:05replacement for in person care.
32:07But now we can take a step back and
32:09figure out the ways that we can use
32:12this tool to enhance the care that
32:14we're delivering to our patients.
32:16And so as we're thinking about assessments.
32:20How do we enhance our assessments
32:23of trainees to not only foster
32:25their growth in telemedicine,
32:27but also to enhance their ability to take
32:30care of patients across all settings?
32:35Next slide,
32:38I came up with some considerations
32:41that are general considerations
32:43for facilitators of assessment.
32:45But I think this is also a great
32:48opportunity to engage our participants,
32:50both panelists and audience participants,
32:53in experiences, thoughts,
32:55evidence that others may have in
32:59terms of facilitating assessment and
33:02also identifying barriers that exist.
33:05And at some point,
33:06we may even find that some of
33:08the barriers that are current
33:10barriers can be overcome and could
33:14actually become facilitators.
33:15And taking a step back to think about the
33:20facilitators for assessment in general,
33:22as Dan started with remember
33:25which you already know and already
33:28use and as medical educators,
33:31we have been working on achieving
33:35excellence in assessment,
33:37the delivery of assessment,
33:39the timely delivery of assessment and the
33:42delivery of assessment to enhance the.
33:44Achievement of competencies and the
33:46achievement of excellence in patient care.
33:49So all of those facets are
33:52really the same in telehealth,
33:55but we need to consider some of the
33:59modifications that could improve our
34:01ways of assessing in this yet new
34:05setting and so similar to other times,
34:09for assessment and it's important to
34:11foster a safe space for learning.
34:14And Shruti discussed the timing of
34:18assessment and it I think requires
34:20even a little bit more recognition of
34:23the environment because we're not at
34:26the elbow necessarily with the trainee.
34:29And so finding the time and space
34:32where the assessor or preceptor
34:35and the trainee can have, you know,
34:38a thoughtful approach to the assessment,
34:40I think it's important.
34:44We want to consider ways that we can
34:47enhance learning and that will help to
34:51promote achieving other outcome metrics,
34:54whether those are within
34:56the core competencies.
34:57Also thinking about outcome metrics
35:00in in terms of patient care metrics,
35:03thinking about, as Dan discussed the EPA's,
35:07the untrustable professional
35:09activities and the achievement of.
35:11Being able to practice independently.
35:15Another general consideration for a
35:18facilitator in assessment is labeling.
35:20And this is something that we do in
35:23the general setting of in person
35:26assessments and that is labeling
35:28for the trainee that we're going
35:30to provide some feedback that this
35:32is time for assessment because that
35:35way the trainee can really anchor on
35:37what feedback is being provided and.
35:42Consider specific examples and how to
35:45move to the next level of competency.
35:50And then, as Shruti alluded to,
35:52designating an optimal window and
35:54this is part of fostering a safe
35:57space for learning that also takes
35:59into account what's the timing of
36:01the assessment and to facilitate.
36:03Oops,
36:03sorry
36:06to facilitate. Providing assessment
36:09immediately after the virtual visit
36:11when the faculty member and trainee
36:14may remain on the call or on the video
36:17encounter after the patient leaves.
36:19It could be done as a post clinic debriefing,
36:24and there are also important roles for a
36:28broader sense of assessment that could
36:30occur at the end of a time period,
36:33such as a rotation. Next slide.
36:41Now to think about a few specifics
36:43in terms of facilitating assessment.
36:46First of all, as a reminder up in the
36:49upper right hand corner is to be open,
36:52to be open about what you have
36:54to offer in terms of assessment,
36:56but also to be open to what the trainee
37:00may provide during the assessment period.
37:03To really facilitate what what you'd
37:06like to share with the trainee and to
37:09be able to recognize what the trainee
37:13has experienced in the past and to focus
37:16on where the trainees goals may be.
37:20And that will be important for providing
37:24meaningful and more lasting assessment.
37:29It's also helpful to.
37:33Set goals prior to the visit,
37:36prior to the patient encounter and that
37:39can be a facilitator for assessment.
37:42Setting goals could be you know
37:44the huddle that can occur before
37:46the patient visit or before the
37:48half day of clinical encounters and
37:53providing A workflow that will be
37:56effective for whether the training
37:58will see the patient first and then.
38:02Contact the unit.
38:04By what means will the faculty
38:06member be contacted and bringing in
38:09the patient from the waiting room.
38:10Bringing in the faculty member from
38:12the waiting room and then including
38:15in that framework of how the patient
38:17visit will go is providing a a
38:19space or a time to the trainee that
38:21says after the patient visit why
38:23don't we regroup and we can talk
38:25about how things went and so that.
38:26Provides again part of the framework.
38:29It can end up being somewhat time
38:32saving because everyone knows
38:34what the expectations are.
38:37Other specifics are really focusing
38:40on what the preceptors goals might
38:42be as well as the trainees goals
38:45for the patient care encounter
38:47and the trainees education.
38:51Other specifics that can be helpful
38:53with force be to provide examples
38:55when talking about the trainees
38:58performance and level of competence
39:00in terms of specific history.
39:02Taking examples and where nuances of
39:04history could have been delineated more
39:07clearly that can be really presented
39:10to the trainee as modeling where the
39:13faculty member elicits different nuances.
39:15During that portion of the encounter,
39:18but then bringing the trainee back
39:21to highlight what occurred that
39:23was different in the preceptors
39:25portion of the patient encounter.
39:27Providing specific feedback for
39:29physical examination techniques or even
39:32demonstrating those when the faculty
39:35member is participating in the encounter.
39:37And then again touching back to how
39:40those techniques might have been
39:42different or elicited some different.
39:45Nuance of the fiscal exam and then
39:50professionalism and interpersonal
39:51communication.
39:52So often providers with students,
39:55trainees,
39:56faculty members believe that they have,
40:00you know,
40:01the perfect setting for the communication
40:03and the things are going well.
40:05And as of you know,
40:07the third person at the faculty
40:10member can certainly.
40:11Have you know a view on where
40:14there might be setbacks or where
40:16things could have gone differently
40:18to enhance the patient encounter?
40:21And then other facilitators for
40:23assessment would actually be having the
40:26trainee either shadow the preceptor
40:29so that the preceptor can demonstrate?
40:32And in that way,
40:33provide feedback to the trainee
40:34or having the demonstration be
40:36part of the virtual visit.
40:37And that's really the work,
40:39the 2nd workflow that Trudy
40:41alluded to in terms of the trainee
40:45performing the patient encounter
40:47and having the preceptor join later.
40:49And in that way the preceptor can
40:53model different aspects in a way to
40:56provide feedback to the trainee and then.
40:59Ultimately,
40:59I think in in being open or
41:01finally I should say in being open,
41:04it's important to inquire from the
41:06trainee how there could be ways that
41:09the faculty members participation in
41:12the encounter could have facilitated the
41:15learning and the experience for the trainee.
41:18That's part of being open.
41:19It's also part of what?
41:23Lee really talked about earlier in the
41:25day in terms of we're all learning
41:28and the trainees may have a quicker
41:31uptick on how how quickly they are
41:34becoming competent in in certain
41:36areas and there may be identified
41:38needs that the trainee has that the
41:40faculty member hadn't really even thought,
41:43thought about addressing.
41:45Next slide.
41:49Well, here's a list of barriers.
41:51And these are things that
41:54our group thought about.
41:56And there are many limitations and they
41:58can vary by institution to institution and
42:01community practice to community practice.
42:04And so I will identify a few barriers.
42:07And as I said, this list could change
42:10dramatically in a month or in six
42:12months or in a year because of the
42:15exponential growth we're all experiencing.
42:17In telehealth.
42:18And that will in fact affect our
42:21ways of assessing our learners.
42:24And I've divided the barriers into the
42:27limitations of the encounter itself,
42:29the faculty member,
42:31the technology,
42:33the setting and the timing,
42:35and the learning curve of assessment and
42:39thinking about limitations of the encounter.
42:42The fiscal examination is limited.
42:44There's some things that we can do in person
42:47much better than we can ever encounter on a,
42:50you know, a video visit.
42:52And so our ability to assess our
42:55learners is much more limited
42:57than if we are at the elbow.
43:00There's also the limitation,
43:01the loss of what we can observe in
43:05our trainees in terms of the loss
43:07of touch when we're not sitting.
43:10At the corner of the desk next
43:12to the patient,
43:13and the patient is tearful and in person.
43:16The expression of empathy can be
43:18so different than on a video visit,
43:22and the patient's emotions aren't
43:24going to necessarily go away just
43:26because it's a video visit In terms
43:28of barriers for the faculty member.
43:31The faculty member may not have
43:33incredible competency in telemedicine.
43:35The faculty member might be learning
43:37just like the trainee is learning.
43:39Or the faculty member may have
43:41a great degree of competency but
43:44lack confidence and his or her
43:46own abilities with telemedicine
43:48and this can therefore provide a
43:50barrier to assessing our trainees.
43:53Other barriers for assessment have
43:55to do with failing of our technology
43:58if there's a three-way phone.
44:003 way video visit and it fails
44:01and we go to a phone visit and
44:03the ability to assess is obviously
44:04much different than if we were in
44:06a video visit and all of that's
44:09different than if we were in person.
44:11And there may be the technology may also
44:15not support a video meeting for feedback.
44:18And you know I think Trudy talked about when,
44:22when and where does this feedback
44:25occur and for.
44:27Formal assessment or even on
44:29the fly assessment,
44:30it's so nice to be in the
44:31room with the trainee,
44:33be able to read how things
44:35are going for the trainee,
44:37both in terms of comfort with the encounter
44:39as well as comfort with the assessment.
44:42And so if technology files us and we're not
44:45able to do this by video and it's only,
44:48you know,
44:48a phone call,
44:49then I think that that provides
44:53further barriers.
44:54And and that really leads to what I
44:56was already describing in terms of the
44:59setting for the assessment being provided.
45:01We've talked about the timing.
45:04If feedback can be provided
45:06in close proximity to the
45:07patient encounter, that's ideal.
45:09If it can't, that can be a barrier in
45:13terms of the anchoring of learning
45:15and what's gained by the trainee.
45:18And this is impacted by
45:19whether it's a busy clinic,
45:21whether the trainee and or the faculty
45:24member are participating in what's
45:26called the hybrid clinic where there's
45:28some virtual visits and there's some
45:30in person visits and the workflow is
45:32kind of crazy and it makes it harder to
45:35take that time aside for assessment.
45:36And then they're also competing schedules
45:39that somebody has to run off to go
45:42do something different or leave from.
45:44Patient encounters to zoom, meetings.
45:46And so really setting that time to
45:49be able to assess is important.
45:52And then there's the learning curve
45:54We are clearly learning as we go and
45:58what we know today about assessment.
46:02May be very different.
46:03We learned so much already today about
46:06the competencies developed by the AA
46:08MC and this will elevate all of us to
46:12be able to provide better assessment
46:14in the telehealth setting and we will
46:18continue to recognize new competencies.
46:21Next line.
46:25In closing, I would say that our
46:28goal is to enhance our assessments
46:31of trainees to not only foster
46:34their growth in telemedicine,
46:35but to enhance their ability to take
46:38care of patients across all settings.
46:40And assessment has just a huge part
46:44in the the growth that our trainees
46:47and that the educators that our
46:51faculty will achieve. Thank you.
47:00Well, thank you, Marcy.
47:02Thank you everyone.
47:03The thank you to the the panelists
47:05who've been our discussions.
47:06This has just been fantastic.
47:08I'm going to summarize briefly the
47:11slide that we're sharing now and then
47:13really would like to open up to Q&amp;A.
47:16So please I want to really invite you
47:20to put your comments and questions into
47:23the question and answer for the group.
47:27As we sum up and I I want to
47:30kind of acknowledge that there is
47:31there's so much to work from here.
47:34We also recognize that there are
47:36some really big opportunities
47:38to enhance the evidence base and
47:41address the open questions that are
47:44are really unfolding when we think
47:46about what it means to educate
47:49in the setting of telehealth and.
47:53These are some reiterations of some
47:55of what that you've heard here as
47:57well as building some others beyond.
47:59But please don't think of this
48:01list as a comprehensive list.
48:02This is sort of a a quick brainstorm.
48:04And the one of the opportunities from
48:07this fantastic group that we have here,
48:10including all the members of this,
48:12of this symposium,
48:13is to really source across all of
48:15us to think about what some of
48:17those remaining questions,
48:18what what are those burning questions are
48:20that we ought to think about in the setting.
48:23Of telehealth.
48:25You know one one question here is as
48:27was raised by Marcy is you know what
48:29are the limitations of the encounter
48:31that are going to be affecting
48:33and how can we anticipate for and
48:36mitigate for those limitations of
48:38this new encounter modality in the
48:40setting of education and competency.
48:45And and assessment,
48:46you know how can how can the
48:48telehealth experience complement
48:49more conventional approaches And I
48:51think this really grows upon Marcy
48:54your your wonderful declaration
48:55that really we should not only
48:57enhance our assessment of trainees,
48:58but we should enhance their ability
49:01across all settings of care.
49:04The questions around understanding our
49:07patients have certainly shifted as we
49:10are engaging with patients in their
49:12homes in other settings over telehealth.
49:14And as Marcy noted,
49:16sometimes over technology that's
49:18less than ideal when the technology
49:21for whatever reason may be a
49:23barrier to that access point.
49:25So we end up talking on the phone instead of
49:28having a full interactive video interaction.
49:30You can see as well others here
49:32how to enrich the care provided.
49:34In the end,
49:35the learning assessment that
49:36occurs in a phone visit,
49:38a very unique area that to date probably
49:41many of us have not spent a lot of time.
49:45On the phone with our trainees
49:47with a patient to be able to know
49:50what those engagements really
49:51mean and how to really augment,
49:53evaluate and assess and provide the right
49:55feedback and learning environment in them.
49:57As well as some of the others here
49:59that are mentioned that I think
50:01are areas that would be terrific
50:03to open up into the panel here.
50:04And with that,
50:05I'd like to turn us into the the last.
50:08I think we have about 10 minutes
50:10for Q&amp;A and
50:13I'm. And I don't see any immediate
50:15questions in the question and answer.
50:17So I'd love to kick off a couple
50:18and then look for others to come in,
50:20if that's all right.
50:21I'd like to start with one,
50:24which is thinking about
50:27high stakes and low stakes.
50:30Then the the framework that
50:31you've talked about, you know,
50:33high stakes learning,
50:34low stakes learning and I'd like
50:37to extend that to ask you about.
50:40Now since we've said we're
50:42all in this together,
50:43largely we're seeing that the
50:45learning curve for trainees and the
50:47learning curve for the educators,
50:49for the supervisors may be closer or aligned.
50:53How do we think about high stakes and
50:55low stakes learning for both parties
50:58in the setting where both occur?
51:00And how do we,
51:01how do we educate to both parts of that?
51:03And then maybe Dan,
51:03after you get this a little bit of A-frame,
51:05I'd love to ask John, Dr.
51:07Scott, if you wouldn't mind jumping in,
51:09especially thinking a little
51:10bit about if we're,
51:11if we're trying to assess clinical
51:13clinicians competencies across the board,
51:15not just trainees.
51:16Again,
51:16how do we think in terms of high stakes,
51:19low stakes and really trying to make
51:21sure that we're doing the right thing
51:22for all of us in this new setting.
51:24Dan, you mind kicking us off?
51:28Really, really good question.
51:29Reminds me of that I did a lot of work on
51:34handoffs and working as part of the group
51:36that developed the ipass handoff program.
51:38And one of the things we discovered there
51:41was that although attending physicians
51:43thought they gave patient handoffs,
51:45well they in reality didn't.
51:48And they were now put into a position where
51:51they are having to observe and help teach.
51:53Trainees to do it in a more effective way
51:57when they were challenged themselves.
51:59I would say first of all,
52:01the competency framework helps a lot 1 that
52:04both trainees and supervisors can buy into.
52:07So understanding what those elements are,
52:10we found that faculty.
52:11One of the ways that faculty
52:13learned around about handoffs most
52:16effectively was to observe trainees
52:18doing them and to give them feedback.
52:21And to really so I would probably
52:24take that be one of the approaches
52:26I would use and then if you in
52:29in trying to push towards more
52:34frequent low stakes assessments
52:36would be ideal in this particular
52:38situation because no given assessment
52:40carries much weight and but we
52:42can take those same assessments
52:44and use them in an aggregate way.
52:46If you have a large enough number
52:48of them to get a highly reliable
52:49estimate of someone's skill.
52:51So going in that direction
52:52provides some safety.
52:54And then really
52:57Shruti talked about it some and Marcy as
52:59well about how how does that feedback
53:01happen and how do you frame it,
53:03How do you set that up?
53:05And the kind of openness in dialogue
53:07and and the conversation that's had
53:10setting up this process can really help.
53:13I'd be real interested in what John thinks
53:15and others think about this as well.
53:21That's your cue to unmute John.
53:24OK, great. So maybe I just want to
53:27share some comments on that last
53:29slide to for in your first point
53:31was limitations of telemedicine.
53:33I might even flip it the other way.
53:34And what is actually the value at and I
53:37think you know talking to my colleagues
53:38is that it's kind of like bringing
53:41back the old Marcus Welby home visit.
53:43So you really can see people
53:45in their living environment.
53:46If you have an elderly patient,
53:47you might be able to identify
53:49like tripping hazards,
53:51you can often enlist family
53:52members and kind of improving
53:53inherence and things like that.
53:55So I think there there's going to
53:57be pros and cons and so that's going
53:59to be a very important scholarship.
54:01The other thing I would really
54:02encourage all of us to look at is
54:04the importance of implicit bias.
54:06So when we are exposed to our
54:09patients home environment,
54:10it might actually trigger some
54:11of those stereotypes and it might
54:13actually lead to a worse interaction.
54:15So there's kind of there's going to
54:17kind of double edged sword with that.
54:19In terms of what we're doing at UW Medicine,
54:23I I wanted to come back to a point
54:25that Marcy and I think even John Co
54:28had made in the prior panel and that
54:30was around just assuring that our attendings,
54:33our senior faculty know how to do.
54:35Telemedicine.
54:36So we've really beefed up our Qi
54:39activities and I really feel like
54:41Qi is feeding back our education.
54:43So it's like it's basically
54:45our feedback loop.
54:45So I wanted to tell you a little
54:46bit about what we're doing.
54:47The 1st is we're doing a lot of surveys.
54:49We've done, we're doing surveys of patients,
54:51we've done surveys of our providers and
54:54actually and then our front desk staff.
54:57And so we're kind of getting that
55:00information and we're we're already
55:01kind of hearing some of the some
55:04concerns around professionalism.
55:05Some of our our docs did work
55:07from home from day one.
55:09So they never really did a
55:10telemedicine visit in the clinic
55:12where we they have the support.
55:13And so we we've heard about some
55:16faculty members who you know there's
55:17a kid in the background or maybe they
55:20you know are are not comfortable with
55:21the meeting and things like that.
55:22So it's really important to get
55:25that data and feed it back.
55:27We also have an anonymous patient
55:30safety reporting tool.
55:31So anyone can kind of put this on,
55:33it's a little icon on all of our desktops.
55:36And then the last aspect of our Qi
55:39program is a proactive, sorry about that.
55:42Welcome to Harborview.
55:47So the last aspect is a
55:49proactive chart review.
55:50So we're we're reviewing about
55:5310% of all the the charts.
55:55And I think a key question is what is
55:58appropriate for telems and what's not
56:00really putting up those guardrails?
56:02So we've heard,
56:03we've seen some things where maybe a
56:05patient got over diuresed and showed
56:06up in our emergency department.
56:09Another patient who had some eye
56:12problems was seen twice by telemedicine
56:14and really had delaying care.
56:16So we're trying to report that back
56:18and and making sure that that is
56:20incorporating our educational process.
56:22You know, Topher, I just wanted to just
56:25throw in three random comments that
56:27occurred to me during your comments
56:28and the session. The first was.
56:30I really love that focus from the
56:31beginning about a learning culture versus
56:33a performative or performance culture.
56:35You know, it's it's not about getting an A,
56:38you know, on telemedicine.
56:39It's really about having everyone
56:41engage in the learning process and
56:43Orient themselves as a learner.
56:44Even when they're a teacher they're
56:46they're always a learner and this
56:47is an area where they're being
56:49put on the front burner for that.
56:51The second was what John just mentioned
56:52about clinical appropriateness.
56:53We don't really have good standards
56:54yet for our physicians.
56:55It's kind of like empiric therapy,
56:58learn as you go.
56:58Well, I guess I shouldn't.
57:00Evaluate that burn over telemedicine
57:01anymore because the guy got septic,
57:03you know, we want to be ahead of that.
57:06But the last is around another
57:08component of observation bias.
57:10I think we don't think routinely
57:12about what we don't see.
57:14So there's lots of things that you
57:16see in a telemedicine encounter
57:18that you don't see but are still
57:20happening in the inperson encounters.
57:22And so we have almost like this sort
57:24of don't look, don't tell approach.
57:26To what's going on, you know,
57:28behind the four walls of the clinic door.
57:30So I agree with John,
57:32we really want to be thinking about
57:34how does the in person visit need to
57:36be deconstructed and reassembled,
57:38What pieces are going to be missing,
57:39what new pieces are going to be added.
57:41And then we don't want to boil
57:43the ocean just in telehealth.
57:45If we find a problem in telehealth,
57:46it's there in person as well.
57:48We have to have a closed loop
57:50feedback that reminds us, oh,
57:52we just don't know about this problem.
57:54But this is definitely a problem.
57:55We just have avoided addressing it
57:57because we were never confronted by it.
58:01You know, these are great points.
58:03And actually one of the areas that
58:04I was kind of struck by thinking
58:06about this is that often when we
58:08need to learn something afresh,
58:10we need to take more time in doing it.
58:13So Shruti, you mentioned this,
58:15I I think you you said this there
58:17might it might take more time on
58:19the supervisors part while you
58:20while you commented on various
58:22kind of efficiencies and workflows.
58:24I'm wondering if really we all need to
58:27think about taking more time in order
58:30to be the right type of supervisors
58:32and the right type of learners as
58:33a part of the overall learning
58:35environment during this transition.
58:36Do you have comments about like what
58:38that what that might mean and and and
58:40Marcia is also thinking about you know
58:42facilitators and barriers because.
58:43I I feel and I hear always that time
58:45is a barrier to education in the way
58:48that we all really want to perform it.
58:52I absolutely agree with you to for that
58:55the only way to effectively evaluate
58:58and assess your learners is to be
59:02proficient in what you're trying to
59:04evaluate and and we are coming to this
59:07point where we're all new learners and
59:09we're all new at this telehealth thing.
59:12No matter how long we might be
59:15using it and and being part of
59:17this environment because we're all
59:19new at assessing telehealth too.
59:21So a lot of faculty development should
59:23go into our faculty before we start
59:26to unleash them upon our our trainees,
59:29our students and our residents to be able
59:31to assess and evaluate them appropriately.
59:33I mean and no matter what new scale or
59:36tool or procedure or device that comes in.
59:39Most of our departments are doing
59:41some sort of factor development
59:43before they go out and use it.
59:45Similarly, if you're trying to use
59:47new rubrics to assess telehealth,
59:49new competencies to assess telehealth,
59:51new anything,
59:51it's going to require a lot of
59:53faculty development ahead of time.
59:55And that's where a lot of our time
59:57should go into and a lot of our efforts
59:59should go into to ensure that we're
01:00:01doing this appropriately and we're
01:00:02not doing a disservice to our trainees
01:00:04when we're go out and assess them.
01:00:07So Topher, you've got one minute
01:00:09left in your session that you can
01:00:11decide if you want to yield to the
01:00:13rheumatologist in corner four or if
01:00:14you want to close it out yourself,
01:00:18you're muted. Topher, if
01:00:21I am the rheumatologist in the corner, yes,
01:00:25You know, it's hard for rheumatologists
01:00:26to limit their talk to less than a minute.
01:00:27But I'm going to just add one other
01:00:29thing that builds on what Shruti said
01:00:31and then turn it back over to Topher
01:00:33and I think actually Dan said earlier.
01:00:36Frequent assessments are important and
01:00:38it's important for us as faculty members.
01:00:42And I would just add the direct
01:00:44observation that can occur like peer
01:00:47direct observation can be very valuable.
01:00:50And this is a setting where I think we
01:00:52could all learn so much from having
01:00:55somebody watch us and that will help us
01:00:58then and reflecting with our trainings.
01:01:01Great. Marcy, thank you for
01:01:03offering tremendous closing words.
01:01:04Lee, we yield back.
01:01:05I I wish that there was more time
01:01:08to hear from all of our fantastic
01:01:10panelists to be able to share more here.
01:01:11But this has just been
01:01:13a fantastic discussion.
01:01:13I want to thank you all and thank you,
01:01:15Lee, for the opportunity.
01:01:16Oh, you're very welcome.
01:01:17Again, I have to say I'm shocked,
01:01:19but we've we've gone for almost 2 1/2 hours.
01:01:22It feels like 5 minutes and we
01:01:23have not lost a single attendee.
01:01:25So we must be doing something right or
01:01:27they've all gotten up and gone fishing
01:01:28but just left the connection up.
01:01:30We don't really know.
01:01:32Let me encourage folks to
01:01:33take a 10 minute break.
01:01:34We will start sharply at 2:30.
01:01:36Feel free to get a do a bio break,
01:01:38answer the phone,
01:01:39whatever you need to do.
01:01:41And again, want to strongly encourage
01:01:45the audience members to please ask
01:01:47questions in the in the Q&amp;A panel.
01:01:49So we'll keep this slide up.
01:01:52We'll see you back in 10 minutes.