Yale Psychiatry Grand Rounds: December 16, 2022

Transcript

• 00:00Um.
• 00:04Thank you so much Seth and John
• 00:06for inviting me and hosting me.
• 00:09Finding out that I'm the last
• 00:11grand round speaker of the year,
• 00:12I feel like I'm standing in
• 00:14between you and your holidays.
• 00:16Hopefully it doesn't feel that way to you.
• 00:19But very excited to be here today
• 00:21and talk about our research.
• 00:23Let me. Without any.
• 00:31Further in do we'll get.
• 00:37We'll get started. It it it is still
• 00:40early out here on the West Coast,
• 00:42so you'll have to put up with me being
• 00:45lit by the glow of my computer monitor.
• 00:48So. Very excited to be able to
• 00:50be here and talk about our work,
• 00:54which is focused on AI broadly.
• 00:57And we'll unpack exactly what that means
• 00:59and what we do to try and understand
• 01:02really at a fundamental level what
• 01:04happens in counseling and psychotherapy.
• 01:06How is it that we can capture
• 01:09some of those active ingredients
• 01:11using new technology so that we
• 01:14can understand something about?
• 01:16What is in the pill of
• 01:17psychotherapy out in the real world?
• 01:22Hold on a second.
• 01:24Before we jump in, I I do want to
• 01:28acknowledge our research support.
• 01:30Which has come really in two phases.
• 01:32There has been foundational
• 01:34research at the universities.
• 01:36This is really been about 15 years
• 01:39of trajectory at this point and that
• 01:42research is continuing at the startup
• 01:44that we founded that Seth had mentioned.
• 01:47And just to disclose,
• 01:49I am a cofounder and have an
• 01:52equity stake in that company.
• 01:55Alright, so where are we headed today?
• 01:57What am I hoping to cover?
• 01:59I want to talk a little
• 02:01bit about the problem.
• 02:02What is the problem that
• 02:03we're trying to solve?
• 02:04And then spend a bit of time at
• 02:06at at least an intuitive level
• 02:08trying to provide an understanding
• 02:11of how the AI technology works.
• 02:13How is it that we can go from
• 02:16fundamentally a conversation
• 02:17to something about the quality
• 02:20kind of actionable information
• 02:23fidelity and competence?
• 02:25Of psychotherapy and counseling.
• 02:26And then we'll shift and we'll look
• 02:28at some of the technologies that we
• 02:30are developing and studying right now.
• 02:37SO1 slide hitting an issue that we
• 02:39all are intimately familiar with,
• 02:41behavioral health problems
• 02:43are massive and disabling.
• 02:45This figure of 20% of Americans
• 02:47is now a couple years old.
• 02:49I can only imagine,
• 02:50courtesy of the pandemic,
• 02:52that that is higher.
• 02:54If anything,
• 02:55my colleagues in the Institute for
• 02:57Health Metrics and evaluation here
• 02:59at the University of Washington have
• 03:01been conducting the global burden of
• 03:03disease study for a number of years.
• 03:05And one of the findings?
• 03:06That was provocative when it first came out,
• 03:09was how large a proportion of psychiatric
• 03:13and psychological conditions account
• 03:15for in terms of the global burden of disease.
• 03:19So again to say something that we all know,
• 03:22there is a huge need for us to have
• 03:25effective and widely available treatments.
• 03:28There are two fundamental problems
• 03:30that we bump into on a daily basis.
• 03:34One is access,
• 03:34and this tends to be front and center,
• 03:37I think in our minds for good reasons,
• 03:39this is being.
• 03:43Exacerbated by the workforce shortage and
• 03:45how do we train up a new workforce members?
• 03:50And then the second issue is quality.
• 03:52And quality can often I think fly under the
• 03:55radar to a certain extent behind access,
• 03:58but let me make the case that
• 04:00it is pretty significant.
• 04:02That,
• 04:03as best we can tell,
• 04:04there's approximately 100 million counseling,
• 04:07psychotherapy,
• 04:07behavioral health oriented
• 04:09intervention conversations each year.
• 04:12And we basically don't know what happens
• 04:15in any of them for the most part.
• 04:18When we do get glimpses into the
• 04:21actual conversations that are
• 04:23counseling and psychotherapy,
• 04:25we can find incredible variability.
• 04:27And so let me unpack what
• 04:29we're looking at here.
• 04:30My colleague John Bayer,
• 04:32who's at the VA Puget Sound
• 04:34on several years ago,
• 04:36did a motivational
• 04:38interviewing training study,
• 04:39partnering with a variety
• 04:41of community agencies,
• 04:42substance use agencies here
• 04:44in the Puget Sound region.
• 04:46As part of that,
• 04:48we created 200 providers,
• 04:49got six recordings of
• 04:52their actual counseling,
• 04:53and then spent the time effort resources
• 04:57to actually fidelity code those.
• 05:00And so this is just the empathy rating
• 05:02that comes out of the motivational
• 05:04interviewing treatment integrity system,
• 05:06their fidelity coding system.
• 05:07And so each one of those purple
• 05:10dots is the average empathy
• 05:12for each one of 200 providers,
• 05:14most of whom have about 6 sessions.
• 05:16Going into that,
• 05:18so based on what we know from this scale,
• 05:21we have here at the upper end
• 05:23of this scale some superstars,
• 05:25these are clinicians that are
• 05:28out practicing in substance use
• 05:30settings who are demonstrating
• 05:31a deep understanding of their
• 05:33clients worldview and capturing
• 05:35something about the meaning of what
• 05:37is being said by their clients.
• 05:39On the other hand,
• 05:41down here we have a group of no
• 05:43stars I guess who are demonstrating
• 05:45no interest in their clients
• 05:47worldview and little to no attention
• 05:50to what the client says.
• 05:51And if you are like me when
• 05:53I first saw this thinking,
• 05:54how is that possible?
• 05:56If these are actually behavioral health
• 05:58providers, these sessions tend to
• 06:00sound a little bit like Doctor Phil.
• 06:03You know, when are you going to get real?
• 06:04When are you going to come to grips with
• 06:06the fact that substance use is ruining
• 06:08your life and it's causing problems?
• 06:09And so there really is a sense that
• 06:11they are not listening to the client.
• 06:12They have a message that
• 06:14they're hammering home.
• 06:15And based on what we know of
• 06:18the very broad literature on the
• 06:20association of empathy and outcomes,
• 06:23this is really toxic treatment.
• 06:26But the main point that I'm trying
• 06:28to make here is not that there's
• 06:30some toxic treatment happening,
• 06:31but just that there is incredible
• 06:33variability and we don't have
• 06:35any line of sight into this.
• 06:36This is data that has only come out
• 06:39of a well funded NIH research study.
• 06:42And so in those well funded
• 06:44NIH research studies,
• 06:45we use the traditional evaluation method
• 06:48for counseling and psychotherapy,
• 06:50which is behavioral coding or human coding.
• 06:52We don't usually do it in real
• 06:55time like this picture.
• 06:56We record sessions and then have
• 06:58a team of experts evaluate them.
• 07:00But it is slow and it is expensive and
• 07:03it really is not used in the real world
• 07:06outside of well funded NIH research studies.
• 07:09If we had AI or or some alternative
• 07:13rapid means of assessing quality,
• 07:15there's a variety of uses that we
• 07:18could use it for performance based
• 07:22feedback and training for supervision.
• 07:25One of the best ways we'll we'll
• 07:27look at an example of using this
• 07:29in a in a few minutes,
• 07:30but one of the best ways to learn
• 07:32is to get specific feedback on
• 07:34new skills that you're learning.
• 07:38Similar to the image,
• 07:40the figure that we just saw.
• 07:43Having some means of quality and
• 07:45sure quality assurance or quality
• 07:47improvement within service delivery.
• 07:51We think a little more commercially,
• 07:54payers are writing checks for services
• 07:56of unknown quality at this point.
• 07:58So payers could have something to
• 08:01know what they're paying for and
• 08:03could potentially be the basis for a
• 08:05type of value based care arrangement.
• 08:10Finally, there are in in many ways we
• 08:14still don't know exactly how it is.
• 08:17The conversations of of counseling and
• 08:19psychotherapy lead to behavior change.
• 08:22And so being able to open the black box
• 08:25in this sense could help us understand
• 08:27how is it that those conversations,
• 08:30those intimate engagements that
• 08:32we have with our clients lead
• 08:35to sustained behavior change.
• 08:39So let, let's tip into.
• 08:41So over the last 15 years, I have been,
• 08:45I am a clinical psychologist by background.
• 08:47I do have an interest in data science,
• 08:50but this work has really been
• 08:53enabled by deep and sustained
• 08:56collaborations with technical
• 08:58experts across machine learning,
• 09:01natural language processing,
• 09:03speech engineering.
• 09:05And so we'll we'll dip into.
• 09:08Within this next section,
• 09:10I'm hoping to give kind of
• 09:11an intuitive understanding
• 09:13of how the technologies work,
• 09:15and if we want to dip into more details,
• 09:17happy to do that in the Q&A.
• 09:22So let me start with just thinking
• 09:24through at at a really basic if we were
• 09:27to to trying to describe to a layperson,
• 09:30you know, what are the raw data,
• 09:32what are the basic building blocks that
• 09:35go into psychotherapy or counseling.
• 09:38So first and foremost it's words,
• 09:40whether that is you know
• 09:42increasingly now telehealth.
• 09:43But it could be in person,
• 09:45it could be telehealth,
• 09:46could be on a phone, could be video,
• 09:48could be text or chat based text interaction.
• 09:51But it is a conversation,
• 09:53so words are one of the basic ingredients.
• 09:57For everything except text based chat,
• 10:02interaction, tone and other types
• 10:05of paralinguistic information.
• 10:07So there's tone prosody things like
• 10:10linguistic disfluencies when someone,
• 10:12someone,
• 10:13someone maybe perseverates on a certain word,
• 10:16and that's indicative of cognitive load.
• 10:18So all of the types of information
• 10:21about how something is said versus just
• 10:24what is said in the words is important.
• 10:27If it's in person or video,
• 10:29there's different types of
• 10:31nonverbal information.
• 10:31That could be posture,
• 10:33facial emotion,
• 10:34gestures.
• 10:37And then finally, it's not simply
• 10:39each one of these components,
• 10:41but the dynamic way in which they
• 10:43unfold in the interaction itself.
• 10:46So the we can't just take
• 10:47a statement such as, wow,
• 10:49I cannot imagine how difficult
• 10:51that must have been to lose your
• 10:53kids and on its own say whether
• 10:55that is an appropriate or good
• 10:57high quality intervention.
• 10:59We need to know what's the context,
• 11:00how would, where is that being said?
• 11:04So then let's switch over and think about.
• 11:06I mean the challenge of measuring
• 11:09counseling and psychotherapy is
• 11:11fundamentally that it is a conversation.
• 11:14It's very unstructured.
• 11:15We think about other types
• 11:17of data that we collect,
• 11:19whether that's lab tests or
• 11:22questionnaire data like the PHQ 9.
• 11:25There is inherently nothing
• 11:27numeric about a conversation.
• 11:29So how do we actually
• 11:31quantify this information?
• 11:35Historically, natural language
• 11:37processing use what's called ngrams,
• 11:40and that's really a fancy way of saying.
• 11:44Quite literally.
• 11:45They would dummy code create indicator
• 11:48variables for unique words, for vocabulary,
• 11:51or for short common phrases.
• 11:53So two word and three word phrases.
• 11:57And if you're thinking like,
• 11:58how is that even possible?
• 12:00If you have a basic understanding
• 12:01of a regression model,
• 12:02wouldn't that mean that there are thousands
• 12:05and thousands of predictors in these models?
• 12:07Yes, that that is exactly right.
• 12:10That is a good intuition.
• 12:12Increasingly, these models are now
• 12:14using something called word embeddings,
• 12:17and this is the idea that
• 12:19when we see a word there,
• 12:21it has a certain meaning.
• 12:23And and these word embeddings are ways of
• 12:26trying to get at a meaning so that it's
• 12:29not just a word with an indicator of binary.
• 12:33Yes, no, this word showed up, but it's
• 12:35implying something about the meaning.
• 12:37I won't try to go in to explain that, though.
• 12:40Happy to go into the weeds if
• 12:42that were of interest later.
• 12:43But so there's there.
• 12:45There are ways of quantifying words.
• 12:48Similarly,
• 12:49there are a variety of speech
• 12:52signal processing methods so that
• 12:54we can estimate acoustic features.
• 12:57So these are things like tone,
• 13:00the vocal arousal that we can hear,
• 13:03and in someone's voice.
• 13:04When someone is excited and the pitch
• 13:07or the tone of their voice goes up,
• 13:08we can measure that reliably.
• 13:11And in addition,
• 13:12we can also measure something called jitter.
• 13:15This is when someone is really upset
• 13:16and we say that their voice is shaking.
• 13:19That is the very extreme form of jitter,
• 13:21but we can measure that over a broad range.
• 13:23So again,
• 13:24point being that there are speech
• 13:27signal processing methods for
• 13:30quantifying these types of information.
• 13:33Similarly,
• 13:33there is an area of machine learning
• 13:36called computer vision that is the
• 13:39reason if you use Google Photos and
• 13:41you can search Google Photos for
• 13:43finding your dog or what have you,
• 13:45that is computer vision is the
• 13:48AI engine that's enabling that.
• 13:50Although our team has some expertise in that,
• 13:52that's not a focus of our current
• 13:55work as we have felt that language
• 13:57and words is really the lowest common
• 14:00denominator across all the different.
• 14:03Medium of counseling and psychotherapy.
• 14:07Finally,
• 14:08there are a variety of techniques
• 14:10both machine learning and natural
• 14:12language processing,
• 14:13but also outside of that dynamic
• 14:16systems models for understanding how
• 14:19is it that interactions unfold over time.
• 14:22So just a couple again,
• 14:24my goal here is really to provide
• 14:26some sense for what is this process
• 14:28and how does this work and so let's
• 14:30take an example of 1 specific example.
• 14:32So our work started.
• 14:35Is motivational interviewing.
• 14:37In hindsight,
• 14:37that seemed like an incredibly wise choice.
• 14:39The reality was, of course,
• 14:41it was a bit happenstance.
• 14:43Was collaborating with some
• 14:45colleagues who were using MRI and
• 14:48had recordings from some RCT's,
• 14:50that that was the very first
• 14:52grant in this work.
• 14:54But MI is a fantastic place
• 14:56to start because it
• 14:58is very linguistic focused.
• 15:01So am I is interested in things like.
• 15:05Or is a clinician asking an open-ended
• 15:08question versus a close ended question,
• 15:10so really tightly tied to the language itself
• 15:14in a way that is fundamentally different
• 15:16than say cognitive behavioral therapy where
• 15:19they're interested in assessing how well
• 15:21the clinician set in the agenda. And yes,
• 15:23there we we can know that from the words,
• 15:27but it's at a kind of higher level,
• 15:29it's more of a psychological
• 15:30construct that's in the words.
• 15:32So am I was a great place
• 15:34for us to start this work?
• 15:36And so we have this brief little
• 15:38snippet of transcript here.
• 15:40You know, client says,
• 15:41I wouldn't mind coming here for treatment,
• 15:43but I don't want to go to one of
• 15:44those places where everyone sits
• 15:46around crying and complaining all day.
• 15:48The counselor says you don't want to do that,
• 15:50so you're kind of wondering
• 15:51what it would be like here.
• 15:54So this was a an example that we used
• 15:57in in one of our early research papers,
• 16:00where the goal was can we
• 16:03automatically identify when a
• 16:04therapist is making reflections,
• 16:07when they're providing a brief summary
• 16:10and reflecting back to the client whether
• 16:14they are understanding them correctly?
• 16:17And so let's just use this to unpack.
• 16:20How do we actually go from a transcript
• 16:24of words to a predictive model?
• 16:27So again, as we talked about, um,
• 16:29one of the traditional ways is that
• 16:31we would use what's called Ngram
• 16:34features where literally it's just
• 16:36identifying there's particular
• 16:38words that are in this statement.
• 16:40And also common two or three word phrases.
• 16:44So again we would basically be quantifying
• 16:48those in types of indicator variables.
• 16:52This is also a conversation,
• 16:53so something is happening over
• 16:55time and so the local context,
• 16:57especially for trying to understand
• 16:59something like a reflection,
• 17:00which is inherently something being said
• 17:03back in response to a previous statement.
• 17:07So something about the context so
• 17:09we can look at words in the local
• 17:12context before or after.
• 17:13There's also a little bit of metadata,
• 17:16and when we use metadata in
• 17:19natural language processing,
• 17:21it refers to anything that is
• 17:23not the words themselves.
• 17:25And in this case,
• 17:26what we minimally know is that
• 17:28there are two different speakers
• 17:30and they have different roles.
• 17:32So is this the client or is this
• 17:34the therapist who's speaking?
• 17:38Finally, we can create other types
• 17:40of features to include as predictors,
• 17:43and because a reflection is inherently
• 17:46capturing something about the,
• 17:48there should be some similarity
• 17:49with what the client has just said.
• 17:51We can identify other types
• 17:53of similarity features,
• 17:54whether those are parts of speech
• 17:56such as are using an adverb or a
• 17:59direct match in terms of words.
• 18:01So all of these would be ways
• 18:04that we could quantify and and,
• 18:06you know, in a statistic sense.
• 18:08Create a a set of predictors
• 18:11for our prediction equation,
• 18:12trying to identify a reflection.
• 18:17One of the other things that as
• 18:20I got into this work and began
• 18:23to collaborate with computer
• 18:24scientists and speech engineers is.
• 18:27I do enjoy data science and there was
• 18:30a period early in my career where I
• 18:32was a lot of my time was being spent
• 18:35really as an applied biostatistician.
• 18:37As we got into this work,
• 18:39I quickly realized there's a whole
• 18:41set of models and methodologies
• 18:44that I was never exposed to.
• 18:47And so part of the inter interdisciplinary
• 18:50work is really that translation,
• 18:52being able to form a foundation of knowledge,
• 18:56both clinical knowledge,
• 18:57so the computer scientist and the speech
• 19:00engineers needed to learn something
• 19:02about motivational interviewing,
• 19:04but also for the rest of the team
• 19:06to understand something about the
• 19:07models that are being applied.
• 19:09And so if you look at these different models,
• 19:13latent Dirichlet allocation,
• 19:14conditional random field,
• 19:16recursive neural networks.
• 19:18And think I have never heard of any of
• 19:20those and I at least had some stats.
• 19:21You are not alone and that's
• 19:24something about the the work here
• 19:26able to bridge those gaps.
• 19:30So the initial phases of our research
• 19:34focused strongly on this idea of can we
• 19:37go from the word spoken in a session
• 19:41to reliably estimating fidelity codes.
• 19:46And we over the course of about 8 to 10
• 19:49years, we have probably 25 or 30 publications
• 19:52capturing different aspects of this,
• 19:54looking at words themselves,
• 19:56looking at paralinguistic information
• 19:58tone itself, combining them,
• 20:00different types of models.
• 20:02And and let me,
• 20:03so let me show you a couple results and let
• 20:06me tell you what we're looking at here.
• 20:08So the traditional method for
• 20:11estimating a fidelity code is to
• 20:14have a team of raters learn a
• 20:16well validated clinical system.
• 20:18So here the motivational interviewing
• 20:21treatment integrity system or the
• 20:23motivational interviewing skills code
• 20:25system and then they make their ratings.
• 20:29But even well trained humans do not
• 20:32agree with each other perfectly.
• 20:34So we call that inter rater reliability
• 20:37and so that's an important piece.
• 20:39For training a computer to be
• 20:41able to do this,
• 20:43which is that interrater reliability
• 20:45of functionally sets a ceiling for us.
• 20:48And so the goal here is really can we
• 20:51develop an AI algorithm that will be as
• 20:54accurate as the most accurate human.
• 20:57And So what we're looking at there on
• 21:00the X axis each one of these labels,
• 21:02advice, giving, affirmation,
• 21:04confront these are specific fidelity
• 21:06codes within the motivational
• 21:09interviewing system.
• 21:10Either things that you should do,
• 21:12such as asking open questions
• 21:14and making reflections,
• 21:15or things that you should not do,
• 21:18such as confronting your client
• 21:20or giving up giving them advice.
• 21:22And that X axis is asking,
• 21:25out of the reliability of the human raters,
• 21:28how reliable is the computer estimate,
• 21:31and so at 100% the computer
• 21:36is estimating providing.
• 21:38Fidelity codes that are identical
• 21:40to our most reliable human coders,
• 21:44and so we can see that over time
• 21:46this is not where we started,
• 21:47but over time we have been able to
• 21:50develop AI algorithms that would
• 21:52start with a recording and generate
• 21:55codes that are highly reliable and
• 21:57very similar to expert human coders.
• 22:00The the one other thing that I'll
• 22:02mention here is that this graph
• 22:05and all of our results use the.
• 22:08Additional methods within machine
• 22:09learning to evaluate models,
• 22:11which is we take the whole data and
• 22:13we cut it up into a couple pieces
• 22:16and there's a set of data that
• 22:18we use to train models.
• 22:19And then when we are completely done
• 22:21with the training of those models,
• 22:23then there is a separate piece of
• 22:25data that they never saw in training.
• 22:27That is our test set or evaluation set.
• 22:29And so these numbers and every time
• 22:32we evaluate them and come out of
• 22:35a test set which and I hope and.
• 22:38In our Q&A,
• 22:39we can get into a little bit of
• 22:41the conversation around what is the
• 22:43data that trains models and then
• 22:45where is that model being applied,
• 22:47because potential AI bias is
• 22:50inherent in those types of
• 22:52questions. OK. So that was for
• 22:55motivational interviewing.
• 22:56More recently, we have done at a parallel
• 22:59set of work with cognitive behavioral
• 23:02therapy that's focused on the CTRS.
• 23:05And similarly over time,
• 23:06this was not the where we started,
• 23:09but over time we've been able to
• 23:12develop models that reliably replicate
• 23:14what human experts will do in
• 23:17terms of generating fidelity codes.
• 23:22And just to highlight another aspect
• 23:25of this work, which is I I have
• 23:28primarily been talking about this
• 23:31one slice around prediction models,
• 23:33but the reality is the the entire,
• 23:36what we call the pipeline starts with
• 23:40a recording or that spoken language
• 23:44and there's an incredibly important
• 23:46and perhaps the most complicated part
• 23:49of what we do is that a speech signal.
• 23:52Processing tasks.
• 23:53So from a recording,
• 23:55can you tease apart uniquely the
• 24:00multiple different speakers and can
• 24:02you identify automatically who the
• 24:04therapist is and who the client is?
• 24:06And can you then generate a
• 24:08highly reliable speech to text,
• 24:10transcript and and so I never
• 24:13would have imagined,
• 24:15as I was getting my PhD
• 24:17in clinical psychology,
• 24:18that I would be collaborating
• 24:20at points on methods for.
• 24:22Lattice scoring in speech
• 24:24to text transcription,
• 24:25but that has been part of the work,
• 24:28is that to help move this forward I have
• 24:32needed to move into technical areas.
• 24:34And my technical colleagues have actually
• 24:37gone to motivational interviewing
• 24:38workshops taught by Bill Miller.
• 24:40And I think that's part of the
• 24:42magic that has made this work,
• 24:43is that you have a collaborative
• 24:45team that's really willing to get
• 24:47outside of their comfort zone pretty
• 24:49dramatically in certain cases.
• 24:53All right. Just to give a snapshot,
• 24:56so the that AI pipeline that we
• 24:59were just talking about at the
• 25:01moment generates around 54 metrics.
• 25:04And so as we saw for both CBT and
• 25:06for motivational interviewing,
• 25:09we generate gold standard fidelity metrics.
• 25:12These are not systems that we made-up,
• 25:15but we went to the literature and said,
• 25:17OK, CBT researchers,
• 25:18am I clinical developers, what all,
• 25:21what is the gold standard?
• 25:23And there's also some other
• 25:24things that we have kind of baked
• 25:27into the pipeline over time.
• 25:29So there are some content codes,
• 25:31so we can have the goal here was
• 25:33really to provide a line of sight
• 25:35into what's this conversation about?
• 25:37Is it about at a high level,
• 25:40is it assessment or therapy
• 25:42or case management?
• 25:44And then what's the focus
• 25:45of the conversation?
• 25:46Is it about mood problems or
• 25:49trauma or suicide work problems,
• 25:51intimate partner problems?
• 25:53So we can capture something about
• 25:56really what's the conversation
• 25:58about and then I'll we'll talk a
• 26:00little bit about this at the end,
• 26:01but we have some exciting
• 26:04developments in particular around
• 26:06suicide risk assessment.
• 26:07And automatically identifying emotions.
• 26:12OK. So we, I had started off a couple
• 26:15minutes ago showing a graph that looks
• 26:18a lot like this that was courtesy
• 26:20of my colleague John Bair to give
• 26:23a sense for the power of the AI.
• 26:26So that was approximately 900
• 26:29sessions that took John and his
• 26:32team about a year to generate.
• 26:35Now that we have moved this
• 26:37technology outside of the university,
• 26:39we have an opportunity to work with.
• 26:42And partnerships customers and
• 26:46in partnering with a large
• 26:50digital telehealth company,
• 26:52we got access to a a million sessions
• 26:56on more than 5000 providers.
• 26:59And so here it's hard to see but
• 27:01there are actually 5000 purple dots.
• 27:04And there were summarizing
• 27:06approximately 1,000,000 sessions
• 27:07on that same empathy scale.
• 27:10We see a similar pattern here,
• 27:12but in addition because we were
• 27:15partnering with a a real World
• 27:18Service delivery provider,
• 27:20we also got access to some KPI's.
• 27:23And so here we can say not only
• 27:26something about the empathy but
• 27:28we can also look at if you saw
• 27:31providers who are highly empathic.
• 27:34Turns out that your clients
• 27:36were much more satisfied.
• 27:38That's Net Promoter score,
• 27:40which is functionally a zero to
• 27:4210 score of how satisfied you are.
• 27:45You're much more satisfied than if
• 27:47you saw folks who are less empathic.
• 27:50It's not rocket science.
• 27:51If someone is really good at
• 27:53paying attention to you and trying
• 27:55to understand your worldview,
• 27:57it's not surprising that their
• 27:59clients will be more satisfied.
• 28:00But we also see this with other effects.
• 28:05So this what we're calling
• 28:07active listening here, is,
• 28:08for those of you who know,
• 28:10motivational interviewing.
• 28:11It is a summary of ORS,
• 28:14how out of all of the language
• 28:17from the therapist,
• 28:18how much of that is open-ended questions,
• 28:22affirmations, reflections and summaries.
• 28:25And again,
• 28:27it is those really low level
• 28:30micro counseling skills.
• 28:31And what we see is that if you see
• 28:34a therapist who is really good at
• 28:37listening at those kind of basic
• 28:40active listening skills.
• 28:41You are much more likely to get what
• 28:43we would consider a full dose of treatment.
• 28:46Conversely,
• 28:47if you see someone who does not do
• 28:49that and is probably giving advice,
• 28:52giving lots of information and
• 28:55maybe even confronting,
• 28:57then you don't stay around
• 28:58in treatment very long.
• 29:00So the the key point here is that
• 29:02both there is a level of validation
• 29:04that's coming as this gets out
• 29:07into large real-world data.
• 29:08It's also demonstrating that these
• 29:11tools can to a certain extent begin
• 29:14to open up the the black box that
• 29:17is therapy in the real world and
• 29:20provide a line of sight and some
• 29:22reliable indicators of what's happening.
• 29:25So let's now kind of shift gears
• 29:27a bit and we'll look at some
• 29:30of the specific technologies.
• 29:32So really everything that I've
• 29:34talked about thus far is kind of
• 29:37describing the engine of the car.
• 29:39And then now let's look at some
• 29:41different technologies and ways
• 29:43that it's getting deployed and
• 29:45some related research as
• 29:47well.
• 29:49And so kind of just repeating what I said,
• 29:52that that university based research
• 29:54really laid the kind of AI engine,
• 29:57the AI foundation that now is getting
• 29:59evaluated in a variety of technologies.
• 30:05So let's take a look at how we
• 30:08might use this for training.
• 30:11One of the things that I love about
• 30:12Bill Miller and the MIT community is
• 30:15that they're incredible empiricists.
• 30:16And so Bill was one of the first people
• 30:19to do fairly rigorous training studies.
• 30:22And what he found is that the ways
• 30:25that we traditionally do training,
• 30:27which is 1/2 day or a full day
• 30:31workshop don't actually work very well.
• 30:33And when when I say they don't work
• 30:35very well, I mean they don't have
• 30:38durable effects on provider behavior.
• 30:40And so he, he is raising this
• 30:43question of how,
• 30:44how is it then that we could train to
• 30:47get broader and more durable effects?
• 30:50This is particularly critical right
• 30:52now because we have such a workforce
• 30:56shortage and so being able to train
• 30:59rapidly and highly effective ways
• 31:02it would be incredibly useful.
• 31:05Fortunately,
• 31:05we haven't yet really moved past
• 31:09our traditional methods.
• 31:11We now might do them online,
• 31:14but our professional training
• 31:15often includes a lot of content
• 31:18and so that's either slides and
• 31:20written content will have lectures.
• 31:22You can see some examples which
• 31:25would be good,
• 31:26but in terms of practice probably limited
• 31:29to role plays and that's probably a very,
• 31:32very small part part of the training.
• 31:35And in terms of assessment or
• 31:37demonstrating what has been learned,
• 31:38we're probably limited to some
• 31:41some type of knowledge quiz.
• 31:43And so at the end of traditional training,
• 31:47oftentimes what we can measure are
• 31:49things like what content was offered,
• 31:52how many providers access the content,
• 31:54to what degree, and again,
• 31:57maybe something about the demonstration
• 31:59of of knowledge or attitudes.
• 32:01But critically,
• 32:02we have not had good ways of actually
• 32:06assessing skills or skill development.
• 32:09If we look at other skills,
• 32:12and I think we can make a very strong case
• 32:15that counseling and psychotherapy is a skill,
• 32:18behavioral skill,
• 32:19much like other types of
• 32:21skills that we would learn.
• 32:22A really key component is practice,
• 32:25lots of practice, ideally with feedback.
• 32:29And this is some of the ways in
• 32:31which I think things break down,
• 32:33which is if you are learning to play tennis,
• 32:36you know whether you're shot,
• 32:37went in or out.
• 32:38And I don't know that that that there
• 32:41is a strong parallel at that level.
• 32:43So feedback often is going to
• 32:46require a coach or outside input.
• 32:49The other thing that's interesting and
• 32:51has motivated some of our training work
• 32:54is that we practice small components.
• 32:57So professional musicians
• 32:58still practice scales.
• 33:00Professional athletes will practice
• 33:02their backhand over and over.
• 33:05And it's it's interesting to to think about,
• 33:07as far as I know,
• 33:09we don't do that within
• 33:10counseling and psychotherapy.
• 33:11We don't.
• 33:12Practice.
• 33:13We might do it a little bit
• 33:16in our degree based training.
• 33:19And uh,
• 33:20but we don't.
• 33:23We we certainly don't do
• 33:25that in a professional way.
• 33:27We don't practice.
• 33:28I'm going to just practice
• 33:29making reflections or asking
• 33:31good open-ended questions.
• 33:35And sorry, I had a little thing pop up.
• 33:39And, and this fits with what we know from
• 33:41the learning sciences community community.
• 33:44So Danny Kahneman said this in his
• 33:47book thinking fast and slow that.
• 33:49The acquisition of skills requires
• 33:51a regular environment and adequate
• 33:54opportunity to practice and rapid
• 33:56and unequivocal feedback about the
• 33:58correctness of thoughts and actions.
• 34:01And intuitively,
• 34:02that makes complete sense for how that
• 34:04is going to drive good skill acquisition.
• 34:08And man, those are hard conditions
• 34:10to meet in terms of training,
• 34:13counseling, and psychotherapy.
• 34:16So if we then say OK, if that if
• 34:19those are the optimal conditions for
• 34:21skilled development and we wanted
• 34:24to design A training technology,
• 34:26these would be some of
• 34:28the design requirements.
• 34:29There would be a heavy focus on practice,
• 34:32many unique practice opportunities.
• 34:34Varying skills, varying content and feedback.
• 34:39In particular, immediate feedback
• 34:41on the correctness of thoughts
• 34:43and actions according to Kahneman,
• 34:46and being able to track that overtime.
• 34:51Excuse me?
• 34:54So this is these are exactly some of
• 34:57the design requirements and features
• 34:59that we have built into several
• 35:02different training technologies.
• 35:06And so I want to give you a quick
• 35:09snapshot of our MIT training platform.
• 35:13So not surprisingly,
• 35:14we it's designed around different skills,
• 35:17so there's different skill modules.
• 35:20And you know, similar to other
• 35:22types of training, we are going
• 35:24to have some expert introduction.
• 35:26So we'll just hear.
• 35:28A quick intro from Terry Moyers.
• 35:31One big reason for burnout in
• 35:34clinicians is the burden of
• 35:36trying to convince people to
• 35:37change when they don't want to.
• 35:39When you see that as your job,
• 35:41it can be exhausting to spend all
• 35:43your time trying to persuade clients
• 35:45to do what they're fighting against.
• 35:49So again, this would be similar to
• 35:51what we find in other trainings.
• 35:53However, the total video
• 35:57piece here is 3 minutes long,
• 35:59so we're going to give you a relatively
• 36:03light touch in terms of that content.
• 36:06We're going to give you some
• 36:08examples of asking good questions.
• 36:13But then the, the innovation,
• 36:14the thing that we really
• 36:16spent the time working on was
• 36:19developing practice and feedback.
• 36:21And so in this training tool what we
• 36:24have are a variety of brief clinical
• 36:28vignettes from standardized patients
• 36:30with actors portraying patients.
• 36:32And so we'll just take a quick listen.
• 36:35My dad was a drunk and I always
• 36:37thought I would never be like him.
• 36:41But lately, all I do is
• 36:42act like him at his worst.
• 36:47I get mean when I drink. Just like he did.
• 36:54And again, this is not a
• 36:56a long kind of 10 minute,
• 36:59this is 43 seconds and that's what
• 37:02these were designed as are like 30
• 37:04seconds to one minute long and a
• 37:07variety of them where the point is that
• 37:10it would give you opportunity to practice.
• 37:13So we'll look at a couple different
• 37:16practice examples and the types
• 37:17of feedback then that is given.
• 37:19So when you drink a lot it
• 37:21impairs how your brain works,
• 37:22it makes your prefrontal cortex.
• 37:24Part of your brain and the front
• 37:26of your head work less effectively,
• 37:28so you end up making really
• 37:31impulsive decisions like getting
• 37:32into fights when you don't want to.
• 37:42And so there's an opportunity to practice.
• 37:44So you, you. We've just heard our
• 37:47our trainee listen to Gabriella kind
• 37:49of talk about her drinking problems.
• 37:52Our trainee went with a
• 37:54kind of psychoeducation.
• 37:56Let me teach you something.
• 37:58And so the system immediately
• 38:00transcribes what is said and then
• 38:02tags it within the motivational
• 38:05interviewing fidelity coding system.
• 38:07And so through an MRI fidelity lens,
• 38:10that was giving information psychoeducation.
• 38:13And so then we get a brief
• 38:16encouragement and kind of a redirection.
• 38:19It looks like you gave information,
• 38:20let's try asking a question.
• 38:23And so we'll look at just one
• 38:24or two more examples of this.
• 38:29If you want to make better
• 38:31decisions and get in fewer fights,
• 38:32you have to consider reducing your drinking.
• 38:35That's going to help your brain work
• 38:37better your prefrontal cortex and
• 38:39allow you to make better decisions.
• 38:41So you might potentially get in
• 38:43fewer fights with your partner.
• 38:45Maybe just talk things out instead?
• 38:55So again, same idea. Unfortunately,
• 38:57our trainee has not quite gotten it,
• 38:59so she's now providing advice.
• 39:02You know, you should,
• 39:03you should stop doing this,
• 39:05you know, kind of hung up on
• 39:08the prefrontal cortex here, Umm.
• 39:10And then finally,
• 39:11example decisions and get in fewer fights,
• 39:15you have to consider reducing your drinking.
• 39:18That's going to help your brain work better.
• 39:21One second. OK.
• 39:27Have you just considered
• 39:28trying to drink less?
• 39:34And we're getting closer,
• 39:35but the goal here is to ask
• 39:38an open-ended question,
• 39:39right one in which our our client
• 39:41is going to kind of open up and
• 39:44tell their narrative, their story.
• 39:47Have you just considered trying to
• 39:50drink less is a yes or no question and
• 39:53so that's going to close them down.
• 39:56So just a couple examples of how we have.
• 40:01Tried to have you just considered that?
• 40:03There we go.
• 40:06A couple of examples of how we're trying
• 40:09to use AI to provide that practice
• 40:12opportunities and immediate feedback.
• 40:14Whereas traditionally if we get
• 40:17feedback it would be, you know,
• 40:19having an entire session and then
• 40:21getting some high level feedback on it.
• 40:23The idea here is can we provide
• 40:26small training examples where
• 40:28practice is coupled with specific
• 40:30feedback in relatively small bytes?
• 40:33And then there are dashboards for
• 40:36providing summative feedback.
• 40:37How well are you learning the skills
• 40:40and for then completing additional
• 40:43practice opportunities over time?
• 40:47There is a larger RCT of this
• 40:50training that is underway currently.
• 40:52The pilot work that laid the
• 40:55foundation for this showed that
• 40:58the immediate feedback after each
• 41:02statement not only led to larger gains.
• 41:06So it's our red bars down here in our graph,
• 41:10but at a later testing period
• 41:13where there was no feedback.
• 41:16We also saw better retention
• 41:19and it increases overtime.
• 41:24So that was a quick snapshot on training.
• 41:27Let's shift gears and think about
• 41:30supervision and quality monitoring at scale.
• 41:35And so here we have been thinking about
• 41:38that AI pipeline and how we could use it
• 41:41within a direct clinical service setting.
• 41:45And so a couple features that we have
• 41:48built into that technology is that at
• 41:52a foundational level, we believe that.
• 41:56Providing a direct contact with the skills
• 41:59of counseling and psychotherapy are
• 42:02going to enhance reflection and learning.
• 42:05So, you know, as opposed to supervision,
• 42:08that begins with a supervisor asking
• 42:10what do you want to talk about or
• 42:13tell me what happened last week?
• 42:15If we can enhance contact with the
• 42:17review of the actual skills and practice,
• 42:20regardless of whether we have any AI
• 42:24fidelity, that's going to enhance
• 42:26people's reflection and learning.
• 42:28And so one piece of it is providing
• 42:30easy access and easy ways to interact
• 42:33with the session recording itself.
• 42:35So there's an automated speech to
• 42:38text transcript,
• 42:39it's searchable.
• 42:40Those little kind of orange bubbles
• 42:42that you see at the top,
• 42:43those are our AI identified content
• 42:47codes kind of what what's happening
• 42:50in this session.
• 42:51And then you'll notice that we also provide
• 42:55a little tags for kind of good good behavior,
• 43:00if you will.
• 43:02So particularly empathic moments
• 43:04within the session get tagged as well.
• 43:10Each session then gets a
• 43:12formal fidelity assessment.
• 43:14So here we're providing kind of a
• 43:16high level summary of the quality of
• 43:20motivational interviewing and then some
• 43:22additional traditional fidelity metrics.
• 43:25And and so this provides both.
• 43:28We can drill down really deeply.
• 43:31Again, MI is is helpful because it
• 43:34actually will uniquely code every
• 43:36statement from the therapist in a session
• 43:39and so our AI will uniquely code every
• 43:42statement from a therapist in a session.
• 43:46And then we've also built tools
• 43:48to try and support really more
• 43:51of a quality assurance view.
• 43:53If we now have the ability with that
• 43:57pipeline to take every session and
• 44:00generate fidelity metrics or quality metrics,
• 44:04how then do we interact
• 44:05with all of that data?
• 44:07If you are in a large agency,
• 44:09there's thousands of sessions a month,
• 44:12if not per week.
• 44:13And so we have built in this kind of
• 44:16suite of tools that would allow either
• 44:19a supervisor or a clinic manager to
• 44:21be able to have really a population
• 44:23health view of the quality of services
• 44:26being provided within their clinic.
• 44:28And so this,
• 44:29this GIF is showing that you can select
• 44:32a particular individual quality metric,
• 44:34you can select a time frame and then you'll
• 44:37see summaries of all your providers.
• 44:40And for Andy,
• 44:41any individual provider,
• 44:42you can select them and see something.
• 44:44Up their caseload and drill down
• 44:46all the way to individual sessions.
• 44:52This technology also is getting
• 44:54evaluated in a current RCT.
• 44:56Actually I think I've got a
• 44:58few pieces of information about
• 45:00that on the next slide in.
• 45:02In preparation for that work,
• 45:06we did some user centered design
• 45:09with publicly funded agencies
• 45:11in the Philadelphia area.
• 45:13This is in collaboration with Tori
• 45:15Creed at the University of Pennsylvania.
• 45:18And so we spent time talking
• 45:20with therapists and leadership,
• 45:22showing them examples of the
• 45:24technology getting their input both
• 45:26about implementation feasibility,
• 45:28how how easy or challenging would
• 45:30it be to implement these types of
• 45:33technologies within their setting
• 45:35and and also measuring kind of
• 45:37implementation readiness and in
• 45:39particular after we spent time with
• 45:41them and showing them the existing
• 45:44technology all kind of across
• 45:46the board the acceptability of.
• 45:48Appropriateness and feasibility increased
• 45:50for both therapists and leadership.
• 45:56We are just starting recruitment for a
• 46:00step wedge design where we will across
• 46:045 actually due to workforce shortages,
• 46:07we anticipate it'll be more like
• 46:097 or 8 clinics to get up to the
• 46:12number of providers that we need.
• 46:13But we will randomize clinics to
• 46:17a sequential essentially turning
• 46:19on of the technology to support
• 46:21supervision and quality monitoring in.
• 46:26Almost 1900 clients and where we
• 46:29will also be assessing PHQ and GAD
• 46:32on a weekly basis to assess both
• 46:35is there an overall effect as well
• 46:37as what are the particular ways in
• 46:39which supervisors and clinicians
• 46:41use the technology that then lead
• 46:44to improved client outcomes.
• 46:45So there's a a mediational
• 46:47hypothesis hypothesis here as well.
• 46:53All right. I am just got a few more
• 46:55slides and before wrapping up and so
• 46:57want to just kind of talk about a couple
• 47:00things that are in the works right now.
• 47:05As probably everyone here knows,
• 47:07there is both a an epidemic of suicide and
• 47:13suicidality and that the federal government
• 47:16has been put in has put in place 988,
• 47:20which is a new crisis care call line.
• 47:23That is where the idea
• 47:25is that this mimics 911.
• 47:27It's a simple 3 digit number
• 47:30anywhere in the US that can be
• 47:33used to access crisis services.
• 47:35As part of this rollout,
• 47:37which SAMPSA is coordinating,
• 47:39they are mandating some quality assurance.
• 47:43I mean, obviously these are some of the most.
• 47:50Severe interactions within
• 47:52behavioral healthcare.
• 47:54As as part of the grant that
• 47:56we wrote on this topic,
• 47:57I learned that in crisis
• 48:00calls in 1% of the calls,
• 48:02a suicide is taking place during the call.
• 48:05So these are incredibly important
• 48:09conversations. And.
• 48:13But it we're in that same situation
• 48:15of how do you assess the quality
• 48:18of these crisis care calls?
• 48:20Well, the traditional method is you record
• 48:23them and then they are reviewed manually.
• 48:26And so we have been working with a partner
• 48:30to lay the lay the foundation for and
• 48:33we wrote a grant together to develop an
• 48:37AI assisted suicide risk assessment.
• 48:39So these are some of the dimensions.
• 48:43Of the quality assessment tool that um
• 48:46Samsa and their partners have put together
• 48:49and are the focus of our grant work.
• 48:52So we're we're partnering with protocol
• 48:55who is provides 988 services and
• 48:58is also one of the Samsung funded
• 49:01national backup Centers for 988.
• 49:04And so as part of our pilot
• 49:06work with protocol,
• 49:06we after getting all of the
• 49:10appropriate IRB in place,
• 49:12we took some of their existing crisis calls
• 49:16and put them through our current pipeline.
• 49:18And one of the things that we
• 49:21can do right now is identify when
• 49:23suicide conversations are occurring.
• 49:25And so as just sort of a proof of concept,
• 49:27we were able to demonstrate that we
• 49:30could differentiate both is there
• 49:31suicide content in this call or not,
• 49:34as well as how much?
• 49:35Of a focus,
• 49:36was it so that X axis is really
• 49:39kind of what proportion of the call
• 49:42was focused on suicide?
• 49:44And then also within that kind of
• 49:47transcript review phase of the technology,
• 49:50suicide is one of those items that
• 49:52gets tagged kind of session content
• 49:54that gets identified immediately.
• 49:56And so it's possible to scope into
• 50:00exactly where in the conversation
• 50:03suicide is being discussed.
• 50:04And so as part of what we hope for,
• 50:07we don't yet have the grant,
• 50:08but as what we hope for will be our,
• 50:10our next grant will be developing
• 50:12some AI technology.
• 50:14To automatically identify whether
• 50:16or not suicide risk assessment
• 50:18is occurring from the call taker
• 50:20and the quality of that.
• 50:24Last thing that that I'll mention,
• 50:26another kind of exciting thing that
• 50:29is kind of right in the middle of.
• 50:32You know one of the real world
• 50:35challenges is implementation and I
• 50:39think within a training setting it's
• 50:41you know easier to sell AI based
• 50:43feedback for learning new skills.
• 50:45But to you know kind of selling clinicians
• 50:48in the real world on quality assessment
• 50:51is a little less straightforward.
• 50:54And so as we have spent time with clinicians,
• 50:57I'm talking with them and user
• 51:00centered design interviews,
• 51:01you know one of the things that.
• 51:03We consistently heard from them
• 51:05was we hate documentation.
• 51:07If there's any way your technology
• 51:09could help us with documentation,
• 51:11we would be more excited about this.
• 51:14And so we have been in the process
• 51:18of doing an initial version 1.0 of
• 51:22a automated clinical documentation.
• 51:24In particular,
• 51:25we have done the same basic process
• 51:28that I described at the beginning,
• 51:30which is with a clinical partner
• 51:32we were able to.
• 51:33Get access to 40,000 clinical notes.
• 51:38And then we have been training AI,
• 51:40AI models that well,
• 51:41here's the recording of the session
• 51:44and then here is what the human
• 51:45said is a good summary of it.
• 51:47And then we can train the AI to
• 51:50learn that mapping of a recording
• 51:52to a session summary.
• 51:54And so this is the type of summary
• 51:58currently that the system can generate.
• 52:02So it is if you think about.
• 52:04Adapt note or a soap note.
• 52:07It provides that basic initial
• 52:10discussion summary of what
• 52:12actually occurred in the session.
• 52:14And we are still in the process
• 52:17of evaluating this internally
• 52:18and with some partners.
• 52:20But the goal here really is that
• 52:22it would be a tool that would help
• 52:25and support providers with not so
• 52:27much their clinical work per se,
• 52:29but necessary documentation that
• 52:31goes along with that clinical work.
• 52:35And there's some,
• 52:36some other features that we've
• 52:38built into it you can include.
• 52:41The kind of canned phrases that you
• 52:43might want to include with regularity.
• 52:47So let me let me wrap up there.
• 52:48There is a lot of work still to be done.
• 52:50We have a bunch of these technologies
• 52:53are currently getting assessed in RCT's.
• 52:56But it does feel like, you know,
• 52:59particularly with AI, there can be so,
• 53:02so much breathless excitement
• 53:04about AI and what I could do.
• 53:07And having spent the last
• 53:0915 years in the trenches,
• 53:11I feel like we're getting to the
• 53:13place where there are some practical
• 53:15tools and we can see some practical
• 53:17applications of how AI could really
• 53:20support behavioral healthcare.
• 53:23So let me wrap up there and happy
• 53:26to take any and all questions.
• 53:29Feel free to ask or if they're.
• 53:33In the chat, I'll try and open the chat.