Yale Psychiatry Grand Rounds - March 5, 2021

March 05, 2021

Information

"A Novel Computational Framework for the Role of Dopamine in Learning and Memory"

Erin S. Calipari, PhD, Assistant Professor of Pharmacology, Vanderbilt School of Medicine

ID6251

To CiteDCA Citation Guide

00:00Because my introduction is much less
00:02important than Doctor Calipari's talk,
00:04which will be eagerly awaiting.
00:07So it's it's my absolute pleasure
00:10to introduce Doctor Erin Calipari
00:13who's today's grand round speaker.
00:16She is someone who has really been
00:19a pioneer and incredibly prolific
00:22scientists already in the area of
00:25addiction and using a rodent models to
00:30understand the neurobiological basis
00:32of important constructs underlying
00:35addiction that are relevant to.
00:38Human subjects she did her
00:40undergraduate degree at the University
00:42of Massachusetts in Amherst,
00:44where she has got her BS in both psychology
00:47and biology and already began working
00:50with Doctor Gerald Meyer using rodent models.
00:53And then she did her graduate work at Wake
00:57Forest School of Medicine with Sarah Jones,
01:00where she began to.
01:03It really immerse herself and neurochemistry,
01:06an other aspects of in vivo manipulations
01:09and measurements related to behavior.
01:12She then went on as a postdoctoral fellow
01:15and then a instructor at the Mount Sinai's
01:19Icahn School of Medicine at Mount Sinai,
01:23where she worked with Eric Nestler.
01:26And there she began to develop
01:28a number of lines of research,
01:31including examination of.
01:33Particularly molecular consequences
01:35of drug addiction,
01:37and particularly intracellular
01:39signaling as a consequence of drug
01:43addiction that may maintain long term
01:46structural and behavioral adaptations
01:49to drugs of abuse and also was really
01:52key in a number of important papers
01:55on sex specific effects of addictive,
01:58addictive substances like cocaine and
02:01those experiments are particularly
02:04notable because they.
02:05Involved many, many levels of evaluation,
02:07not only in rodents but also in
02:10human subjects.
02:11She has won a number of awards already.
02:14I want to note a few notable ones.
02:17In particular she is an awardee of the DP.
02:21One Avenir Award in genetics,
02:23and epigenetics from the National
02:25Institute on Drug Abuse,
02:27and that's someone who the director
02:29of Naida pulls out as having research
02:32that is extremely innovative and
02:34at the edge of the.
02:36Molecular basis of addiction.
02:38She's an associate member of the AC NP.
02:41Oh I should have noted that she's
02:43also an assistant professor in
02:45the department's particularly in
02:46the Department of Pharmacology
02:48at Vanderbilt University,
02:50but also has appointments in
02:52the departments of Psychiatry
02:53and Behavioral Sciences.
02:55Of course relevant to our Department
02:57and the Department of Molecular
02:59Physiology and Biophysics.
03:00So again, even with her appointments,
03:03you can see how her work spans areas of
03:06investigation from the very molecular
03:08and cellular through the pharmacological too.
03:11The area of drug addiction
03:13relevant to psychiatric illness.
03:15So back to her awards.
03:17Just because they're so notable she is,
03:20has been awarded a Whitehall Foundation
03:23research grant and are said Young
03:25Investigator Award and a K99 to
03:28R00 pathway to Independence Award,
03:30and even back in her days
03:33as a graduate student.
03:34The Knighted Director's Award,
03:36which acknowledged her innovative
03:38work from the beginning,
03:40an addiction.
03:41She's an editorial boards relevant
03:43to this Department.
03:44Both the editorial board of
03:46Neuropsychopharmacology and
03:46the Journal of Neuroscience,
03:48and she's been incredibly
03:49prolific at every stage of her
03:52career, showing how which I love that
03:54not only does she have great ideas,
03:57but she carries them through all the way
03:59to really publishing beautiful papers,
04:02and it's been exciting to watch
04:04the evolution of the work.
04:06Do you want to say something that
04:08is not exactly science related,
04:10as she was recounting during?
04:12The first discussion she does have one area
04:15that she's interested in outside of science,
04:17or that she's able to devote time
04:19to as an assistant professor,
04:21and that is she is a powerlifter and
04:24I have seen a video of her doing.
04:27Plus, which I'm very, very envious of,
04:29but I do want to highlight that her most
04:32recent achievement in powerlifting is
04:34that she has just deadlifted 300 pounds.
04:36Now if that isn't an achievement,
04:38I don't know what is.
04:39So thank you very much for being
04:42with us here.
04:42Erin Dr.
04:43Calipari,
04:43it is a pleasure to welcome you
04:45to the Department and I'm looking
04:47forward to hearing we're all looking
04:49forward to hearing your talk.
04:51Thank you so
04:52much. That was so nice.
04:53And yeah, I know I tried it.
04:55You know the pandemic has
04:57made me start to try to find.
04:59Things to occupy my time that aren't
05:01just sitting at home and working and
05:03so that's that's what I've been doing.
05:06But I'm I'm actually really excited
05:07to present for a number of reasons.
05:09One, because I get to see some people
05:12outside of my immediate family in my lab.
05:14But to this, this work is some kind
05:17of work that's been developing in
05:19my lab of the last couple of years.
05:22That focuses on kind of what
05:24dopamine is doing,
05:25and so I think you know for somebody who
05:28studies addiction and psychiatric disease.
05:31The reason this is so important
05:33is because domains at the core
05:35of a lot of these disorders,
05:37and you know specifically addiction
05:39where you see deficits in dopamine,
05:41and I think it's really important
05:43to understand for people what
05:45those deficits mean.
05:46You know if dopamine is encoding
05:48reward and reduction in dopamine
05:50may mean something very different.
05:52Then if it don't mean is encoding
05:54some other aspect of learned behavior,
05:56which I'm probably going to show you today.
06:00So you know,
06:00I'll kind of show some stuff in the weeds,
06:03but also kind of tide into big
06:05picture and so please like stop
06:07me if things aren't clear or you
06:10have thoughts or comments as we go.
06:12OK,
06:13so the focus of my lab is really
06:15understanding if you can see my slides right.
06:18OK,
06:18good is understanding how neural
06:20experience or how neural circuits
06:22integrate experiences to drive
06:23behavior and so you know in life
06:26or in animals you know we have
06:28experiences that have negative outcomes.
06:30Things that have positive outcomes
06:32and what happens is that these
06:34experiences change the way our brain
06:36response to stimuli in the future to
06:38increase the probability of behaviors
06:40that result in good outcomes.
06:41And decrease the probability of
06:43behaviors that result in negative outcomes.
06:46And the reason I'm so interested
06:49in this is because this is kind of,
06:52you know,
06:53the fundamental core of how we
06:55make decisions.
06:56But it's also dysregulated,
06:57and almost every psychiatric disease state,
07:00and so you know for somebody
07:02who studies addiction,
07:04you know drug associated stimuli,
07:06or overvalued relative to
07:07negative consequences.
07:08An alternative reinforcers depression.
07:10You have reduced motivation.
07:11Valuation of rewards.
07:12Reward learning or things like
07:14anxiety and stress disorders where
07:16these negative outcomes may be
07:18overgeneralized to neutral cues and contexts,
07:20which is,
07:21you know things like PTSD,
07:23and so you know this kind of
07:25fundamental process by which we
07:27attribute value to things that our
07:29environment is really a core of
07:31how we should be thinking about
07:33treating people with disorders
07:35where this is dysregulated.
07:37So you know, I'm, you know,
07:39going way back to the simple,
07:41you know half of my lab studies.
07:43You know how to drug change the brain,
07:45but the other half of my
07:47labs that he's just
07:48kind of. How do these same systems
07:50work in a normal situation?
07:51And so the first thing is kind of,
07:54you know, how do we learn to
07:56make these adaptive decisions?
07:57And so we use things in our
07:59Virat environment to do this.
08:01So maybe there's contextual cues
08:02that help you figure out when things
08:05are dangerous and when they aren't,
08:06and so if you have something like the
08:09sound of a helicopter in a hometown
08:11may be very different than the cell of
08:14sound of a helicopter in a war zone.
08:16We have things like discrete cues,
08:18which would be the sound of a
08:19helicopter itself with drug addiction.
08:21It's these cues that are associated
08:23with the drug taking experience,
08:25and so we learn to associate those.
08:27And then there's also, you know.
08:29What we do in response to these stimuli,
08:31and so you know my background is
08:33really focused on reinforcement
08:34learning and how these stimuli in
08:36the environment drive animals to
08:37make decisions in different context.
08:39So what are they going to do?
08:41Are they going to, you know,
08:43are they reinforce?
08:43Are they going to do something or
08:45they going to avoid something?
08:47And how we can understand the neural
08:50circuitry that underlies this
08:51decision to kind of seek out or avoid
08:54different things in the environment?
08:56And so you know,
08:57I think this is a really important thing,
08:59right?
09:00Is that we learn to make predictions and
09:02so our actions have some sort of outcome.
09:05It changes the state of our
09:07environment and basically what we
09:09do is we learn to do something and
09:11so this is kind of guiding these.
09:13These associations aren't just there,
09:14they're guiding how we navigate
09:16an environment. And so.
09:18How do we do this?
09:21Well,
09:21we need to encode the value or
09:24salience salience is kind of like how
09:26attention grabbing something is of
09:28unexpected outcomes and so you get ice cream.
09:31That's great.
09:32It's awesome.
09:33We need to know whether it's good
09:35or bad and how good or bad it is.
09:38How attention, yes, are you advancing slides?
09:40Yes, sorry,
09:41this has happened to me before and
09:44I have no idea why it does this.
09:47Let me try this again.
09:49It's OK if you guys followed them.
09:51Yeah, I thought the intro was all very clear,
09:54but I thought maybe you were dancing.
09:56We didn't know. Yeah, this is
09:57this is happened to me before. I have no.
10:01Idea when or why this does this?
10:04Let me try this one more time.
10:06So where I made it 'cause it's on,
10:09zoom on the shorter side because
10:11I don't think people love watching
10:13zoom for two hours. So OK.
10:15So now if I move the slides they move OK.
10:19Well, there were pictures you guys
10:21have experience with all of this stuff,
10:23so that's fine.
10:24So now we're into the important
10:25bit so it's good you saw this. OK,
10:28so nothing like this is like pandemic level.
10:30Like everything,
10:30something has to go wrong every time.
10:32Otherwise, like you know, it's not.
10:34It's not real, so OK,
10:35so you have to encode some information.
10:37We need to know whether it's good
10:39or bad and how intense it is.
10:41Is this something we should
10:43really pay attention to?
10:44Or is this something that's not as important?
10:46We need to make predictions
10:48about when that's going to occur.
10:50And so you know you have an ice cream truck.
10:52You predict whether the ice
10:54cream will be there or not.
10:55But not only do we need to make predictions,
10:58we need to be able to update
11:00these when they change.
11:01So when something no longer is associated,
11:03we need to be able to adapt.
11:05If the update this so that we can change
11:07our behavior when the environment is
11:09not the same as we learned previously,
11:11and so this is a really, really critical
11:14aspect of learning and behavior.
11:15So I'm going to kind of go.
11:17There's going to be some computation in here,
11:19but what I'll tell you is most of
11:21it is is more of a framework for
11:23how people think about how these
11:25these computations are being done,
11:27and if you don't care about the computation,
11:29which I've met, many people who say,
11:31oh,
11:31whatever we've used these to
11:32design experiments,
11:33and so it's not like you
11:34need to know the math.
11:36It's more of a kind of framework
11:38for how we designed experiments.
11:40So this kind of prediction
11:42based learning was formalized.
11:43You know,
11:44originally by Rescorla Wagner in 1972,
11:46and there's been a bunch of kind
11:48of adaptations of this and allow
11:51the model to do other things.
11:53But really,
11:53what this is is it's a mathematical
11:56model that allows us to kind
11:58of formalize how animals learn,
12:00and So what happens in this model
12:02is that if you have something
12:04like an unexpected outcome,
12:06that is an error in prediction,
12:08you predicted nothing, something was there.
12:10You made an error and what happens
12:13overtime is your prediction gets
12:15better and then there's less error.
12:17So essentially what happens is the
12:20associative strength or how well
12:22you how well something predicts
12:24something goes up and the error in
12:26that prediction goes down and so
12:28basically the way the model works is
12:31that as you learn the prediction of
12:34that Q and the outcome increases.
12:36But any errors you make go down
12:39and so essentially
12:40you get this. Increase in the
12:42predictive response and a decrease in
12:44the error or the mistakes from that,
12:46and so this is kind of how animals learn.
12:49It can map learning rates in
12:51a lot of different contexts.
12:53You know learning about
12:54Accuen award extinction.
12:55All of these and so people have really
12:58been searching for what is a circuit
13:01in the brain that does this math.
13:03And that's been a kind of really big
13:06focus of specifically the dopamine field.
13:09And other fields too.
13:10I think a lot of people are
13:11starting to see that these kinds of
13:13computations are done in a variety
13:15of circuits across the brain.
13:17So the dopamine system is important
13:19for any a lot of reasons.
13:21These neurons you know that we focus on
13:24originate in the ventral tegmental area,
13:27so we're focusing on more reward
13:29associated circuits rather than things
13:31that are associated with motor.
13:33So we're looking in for this
13:35particular project.
13:36The nucleus accumbens,
13:37the core region,
13:38and So what these dopamine neurons
13:41are really important for survival.
13:43Lesioning them present prevents
13:44this kind of associative learning.
13:46An also reinforcement learning.
13:49And the thing people been kind of really
13:52focus on is that this domain neurons
13:54respond in a fashion that mimics this.
13:57This mathematical model I just showed you,
13:59and so essentially this kind of
14:01originated and within a lots of other
14:04people have shown these patterns.
14:05So this originative with Wolfram Schultz
14:07and I'm just showing the the original.
14:10But people within the domain
14:12field have done this with all
14:14kinds of other approaches as well.
14:16But essentially what they see
14:18is this kind of same math.
14:20We went hoping that this didn't
14:23just move because OK,
14:24the slides are still advancing, right?
14:27Yeah, OK, OK.
14:28So essentially what happens is
14:29you have an unexpected reward.
14:32Dopamine firing goes up.
14:34You predict that reward dopamine firing
14:37now goes up to the queue that predicts it,
14:40but not the reward,
14:41because the prediction of that
14:43reward is basically perfect.
14:44And now if the reward is omitted,
14:47what happens is the dopamine
14:49response to the queue goes up,
14:51but there is now a decrease in that
14:54domain response when it's omitted,
14:56signaling the negative error that,
14:58uh, from that prediction.
14:59And so this is they.
15:02You know, originally this,
15:03this first paper.
15:04They said, wow,
15:05that looks a lot like reward prediction,
15:07error learning,
15:08and so this is kind of formed the
15:10basis of the domain field dopamine
15:13does reward prediction learning.
15:15So here's the kind of maybe issue with that.
15:19If you do stress work,
15:21anything else you know the domain
15:23does not only respond to rewards
15:26and reward predictions,
15:28it's involved in things like punishment,
15:30which is an aversive learning parameter.
15:33Motivation, fear, safety transitions,
15:35aversive learning.
15:37All kinds of there's been a
15:38lot of Association,
15:39aversive learning and these
15:40fields have been kind of.
15:42It was a separate,
15:43but there's kind of the reward.
15:44Prediction people and then the
15:46people who studied anxiety,
15:47depression looking at Microdialysis,
15:48showing that dopamine does
15:49go up to aversive stimuli,
15:50and so these kind of have been a
15:52little bit at odds with each other.
15:55But they kind of are,
15:56you know, different fields,
15:57so people haven't really looked
15:59at them in the same context.
16:01And so essentially,
16:02I think some of the disconnect also
16:04comes from this kind of fundamental
16:06process of about domain neurons.
16:08That's actually my favorite
16:09part of GOP neurons.
16:11Many studies have looked
16:12at VTA cell body firing.
16:14They use electrophysiology.
16:15They say we don't need it goes up.
16:18It goes down.
16:19And there's this inference that
16:21that means dopamine release.
16:23That is,
16:23projection targets is going to be the
16:26same as what the firing looks like.
16:29But dopamine terminals are so cool
16:31because they're regulated at the
16:33terminal level by **** synaptic.
16:34So things that are regulated by Domi
16:37itself but also header, synaptic,
16:39regulators things like glutamate,
16:40GABA.
16:41A favorite of this Department,
16:43acetylcholine and so these.
16:44These things actually can elicit
16:46dopamine release from the terminals,
16:47independent of cymatic firing.
16:49And so if you want to understand
16:51what dopamine release another
16:52projection target is doing,
16:54you need to actually record
16:56dopamine and the ultimate wrists.
16:57You have a few of those as well,
17:00have been doing this for a really long time,
17:03but there's a lot of kind of
17:06limitations to voltammetry and we'll
17:08kind of talk about those as we go,
17:10but our goal was really too.
17:13Record dopamine,
17:13but be able to dissociate
17:15these kind of things.
17:16People have seen in the aversive
17:18field with the things people
17:20have seen in the reward fields.
17:22Why is dopamine look like it's doing
17:24both of these at the same time?
17:27So my background isn't reinforcement
17:29learning and what we did is we we
17:31like to develop behavioral tasks to
17:33parse the things that we're interested.
17:35So we developed this task,
17:37which is not really the task
17:38itself is an innovative behavioral
17:40pharmacology and reinforcement learning.
17:42People have been doing this for years.
17:45Essentially what we do is we
17:46have a queue that comes on.
17:48In one phase,
17:49that tells animals if they know
17:50spoke during this Q and in
17:52this example is white noise.
17:54But we counterbalance and change
17:55that they will get sucrose.
17:57This is like normal positive reinforcement.
17:59You know you treat your teacher dog,
18:01that's it. They get a reward.
18:03What we taught the animals in
18:05the other face is that a separate
18:07queue comes on and they have
18:09the same behavioral response.
18:10They know spoke,
18:11but they know spoke to prevent a
18:13series of shocks from being delivered,
18:15so it's called negative reinforcement.
18:18The reason that using these is so cool
18:20is because they have the exact same action.
18:23So if dopamine just encodes
18:24the motivated response,
18:25these will look the same.
18:28They have the same outcome value.
18:30The outcome is positive.
18:31Avoiding something negative is positive.
18:32Getting something positive is positive,
18:34but there's different stimuli maintaining
18:36these behavioral events and so
18:38essentially what we've associated here is
18:40the kind of motivated action from this
18:42stimulus value in the outcome value,
18:43and the question is in this sounds
18:46more complicated than it is,
18:47and I'm going to tell you the story
18:50is that we're going to be able to see
18:52if dopamine responds to just rewards
18:55if it's just involved in motivation,
18:57or if it's doing something maybe slightly.
19:00It's a more complicated,
19:01but it's actually simpler.
19:03We need a way to record dopamine
19:06during this task.
19:07Aversive foot shocks are electrical signals.
19:09All of the previous domain recording
19:11techniques on fast time scales
19:13were used on electrical systems,
19:14and so the problem with this is all
19:17the voltammetry techniques people
19:18use before you couldn't record
19:20responses to foot shocks,
19:22and So what we've been using is
19:25a fluorescent dopamine sensor.
19:26This one is called delight.
19:28It was developed at UC Davis by
19:30Lindsay Angela and what this is is
19:33it's a modified D1 dopamine receptor
19:35that when it binds to dopamine it.
19:38Laura says.
19:38And so this fluorescent sensor is
19:40really great because we can inject
19:42it in with a virus into the brain.
19:44We build fiber, photometry, systems.
19:46I know a lot of people are using these,
19:49but what it allows us to do is in awake
19:51and behaving animals during these
19:53discrete aspects of this behavioral task,
19:55is record fluctuations in joking that
19:57happened through this kind of fluorescent
19:59response that isn't interfering?
20:00Electrical signals and also the
20:02great thing about these optical
20:03sensors is they have really great
20:05signal to noise and so you can get
20:07single trial responses which with
20:08a lot of voltammetry in the past,
20:10which is my backgrounds you didn't get,
20:12you had to average responses and
20:14what I'll show you is a lot of what
20:16we see is these really rapid changes
20:18in dopamine that are happening on
20:19the trial by trial basis that we
20:21think are really critical for
20:23this behavioral response.
20:26So we started with kind
20:27of what everyone is done.
20:29Before this you know it's always good
20:30when you start with like new tools to
20:33make sure you see what everybody else's.
20:35And So what we did is we recorded domain
20:37responses during the pre training session.
20:39The first time the animals had been
20:41in these operating chambers and post
20:42training after the animals had learned
20:44and so not surprising animals learn to
20:46know spoke during a queue for sucrose we
20:48can change the length of the queue to
20:50make the task more or less difficult.
20:52We kind of did this so we had some
20:55dynamic range of whether they did the.
20:57Miss trials or not,
20:58and so we did some machine learning.
20:59I won't show you then.
21:01This was actually a really
21:02great tool for that,
21:03but what we do is we see kind of the
21:05same thing everyone else is seen.
21:07Early on,
21:08when the animals go into the sucrose port,
21:11so red in here this is more
21:13domain response over trials.
21:14When they go into the Super sport,
21:16you get this robust domain
21:18response to the sucrose.
21:20Overtraining this signal
21:21moves back to the cube.
21:23That's very predictive.
21:23So now you get this really robust
21:26domain response to the queue,
21:27but not as much of a domain
21:30response to the sucrose.
21:32Great, it looks just like that equation.
21:34I showed you.
21:35Dopamine goes up to the Q goes
21:37down to the error signal.
21:40All is well in the reward domain
21:43does reward based learning field.
21:45But then we moved on to this
21:47other behavioral task.
21:48So again,
21:49the animals can know spoke
21:50during this Q and they do it to
21:53avoid a series of foot shocks.
21:54So if they don't press during
21:56the Q they get shocked.
21:58There's a series of shocks they
21:59can press anytime during this
22:01series to terminate the shocks,
22:02and so we did the same thing
22:04we recorded early in Leanne.
22:06Learning animals actually learn.
22:07When I started my lab a bunch of behavior.
22:10People told me that animals
22:11will never do this.
22:12Mice do this really great.
22:14They'll learn really rapidly
22:15to press on the nose,
22:17poke that active know spoke
22:18to avoid the shocks.
22:19And they actually at the end
22:21of these trials are doing this,
22:23that they are not getting shots at all,
22:25and so they learn this very fast.
22:27And it's actually really robust task
22:29for generating motivated behavior.
22:30Um?
22:31OK,
22:31so the first thing we saw which goes
22:34in the face of dopamine encoding
22:36rewards is that you get this really
22:39robust positive response to a foot shock.
22:43So dopamine goes up when
22:45aversive stimuli are encountered.
22:47Um,
22:48other people have seen this,
22:49but what's really interesting is I'll
22:51remind you of what that model looked like.
22:53If dopamine is doing reward based learning.
22:56We didn't get this robust
22:58response to the Q like we did with
23:02sucrose overlearning.
23:03We did get a decrease in the response.
23:05We have the safety queue
23:07that came on when the
23:08animals avoided the negative consequences.
23:10So at the end of the trial.
23:13It did go down over learning like
23:15you'd expect of an error signal,
23:17but here's the problem.
23:18The safety queue domain response
23:19was the biggest on the first
23:21trial they ever encountered it
23:22before they could know its value,
23:24and so we were kind of a little
23:26bit hesitant about that.
23:27But we said OK, but maybe this looks kind of.
23:30Maybe it fits.
23:31But then what we found was that Adobe
23:34response to the foot shot during
23:36these trials was not only positive,
23:38it actually increases.
23:40Animals got better at the task and so.
23:43We've looked and we said,
23:45OK, this doesn't really fit.
23:46People had seen this safety Q response
23:48before and said look doping doesn't
23:50work of RP in aversive contexts.
23:52But we looked at the other parameters
23:55that this doesn't really make sense.
23:57So now we have this situation
24:00where dopamine responses in the
24:02nucleus accumbens track these
24:05prediction error based computations.
24:07But only in contexts that are reward
24:10based and so everyone has really kind
24:13of design these experiments to parse
24:15weather domain does RP not does all
24:18of what doping does fit these computations.
24:21So we had a problem.
24:24This reward based Association model
24:26was just too simplistic to account
24:28for what domain was doing in the same
24:31animals in his behavioral tasks.
24:33And what we started looking through
24:35the literature is a lot of people.
24:37What they did is once we have this RP
24:39hypothesis reward prediction error.
24:41Apophysis people started saying OK,
24:42well reward fish based dictionary does.
24:44This. Does dopamine look like this?
24:46And the issue with that is that
24:48dopamine does a lot of stuff that
24:50reward prediction error cannot do,
24:52and so we ended up doing all
24:54this broad prediction error.
24:55Modeling those that math cannot
24:56do negative reinforcement.
24:57So we ended up at this problem where
24:59if we wanted to understand what domain
25:01was doing from a computational model.
25:03These models didn't even make
25:06the computations we needed.
25:07So.
25:08We decided to I have a
25:10postdoc who's fantastic,
25:12who's a computational psychologist.
25:13I will not take credit for
25:15developing the model.
25:16This is not my backgrounds,
25:18but we've had a really great
25:20synergistic relationship.
25:21And So what we did is we created a
25:23complex model of learning and memory.
25:26You have the theoretical components of
25:28this model that are developed from site.
25:30Many years of psychology research
25:32and then what we can do is we can
25:35record domain responses and many many
25:37Contacts and we can map the domain responses.
25:40On to the parameters of this model
25:42that we know is capable of modeling
25:45the behavioral outputs we have.
25:47So I'm not going to go into
25:49details like super details,
25:50but I'm really happy for people if
25:52they have questions to talk more
25:54essentially with the model does is
25:56it models the behavioral responses,
25:58the outcomes,
25:59the predictions like people have done before.
26:01We actually have those prediction
26:02based learning algorithms.
26:03But one thing it has that's actually
26:05was been has been found over years.
26:08So you really critical component of learning.
26:10Is this perceived salience term?
26:11And so this is kind of you know how
26:14attention grabbing something is and
26:15so it's really highly influenced by
26:17things in the environment like novelty.
26:20The first time you experience an unexpected
26:22aversive foot shock or something,
26:23it's it's more attention grabbing
26:25the 10th time you present it,
26:26and so this that with this term does
26:28is it influences how we learn based
26:31on these other factors that are
26:33not included in these other models
26:34and what it does is it's able to
26:37make really accurate predictions of
26:39what animals will do in the future.
26:42Again, we use this model to figure out
26:44what experiments would dissociate these
26:45different factors from one another.
26:47So after I show you this,
26:48you can ignore the model stuff
26:50if you want and just look at the
26:52experiments we run to parse.
26:54Kind of what's going on.
26:56The really important thing about this
26:58model is the simulations from the model.
27:00The behavioral output
27:01simulations are in grey,
27:03and the behavioral data itself
27:04is in blue and you can see it can
27:07start to model these things that
27:09couldn't be modeled before,
27:11so things like the animals to train
27:13to know spoke for sucrose and then
27:15we have we introduce an aversive
27:17foot shock all the sudden.
27:19The animals responding goes down.
27:20That's what the model suggests.
27:22We can model how animals will learn
27:24and negative reinforcement Contacts
27:26the removal of an aversive stimulus.
27:28And so we're now able to model the
27:31behavioral outcomes of these more
27:33complex behaviors but use these same
27:35kind of computations to not punch to me so.
27:38What I'm going to show you is that
27:40dopamine does not track reward
27:42prediction error.
27:42It tracks perceived salience,
27:44which is just kind of like how
27:46attention grabbing a stimulus is.
27:48Perceived salience is the perception
27:51of housing question.
27:52Yes,
27:53just a quick question.
27:55How different is this model from
27:58Pierce Hall Macintosh model of?
28:02It's it's actually very similar,
28:04so it includes the.
28:05So the thing about the Pearsall Macintosh
28:07models for the people who don't know is
28:09they include these attentional terms,
28:11which are really actually important
28:12for things like Leighton ambition,
28:14things that these other models can't do.
28:16It includes the same kind of
28:18computational terms, and this perceived
28:19salience term is essentially kind
28:21of what was added to that model.
28:23It's based on a neural net model,
28:25so it's a little bit different in the math,
28:27but it's the same kind of idea,
28:30and that's it that the
28:31attentional value of things.
28:32Are going to influence the associative
28:34strength and how animals learn,
28:35and so that's that's what we're adding
28:38that allows us to do these things.
28:40And then on top of that, we've added these.
28:42They operate probabilistic responses because
28:44pavlovi and learning is not probabilistic,
28:45and so we've added that to that.
28:47To that kind of framework.
28:49So it's very, very similar to that.
28:51Thank you for that question.
28:52I don't go into too much detail 'cause
28:54people don't always know about these models,
28:57but that's actually kind of the
28:58framework by which we're using it.
29:00More appears Hall that the Macintosh,
29:02though.
29:04OK, so we can model the behavior we need,
29:07so so again,
29:08this perceived salience is kind of
29:10like a driving animals attention
29:11towards States and so it's really
29:13highly affected by the in physical
29:15intensity of a stimulus and the
29:17novelty and environment that
29:18changes how you attend to stimulate.
29:21And So what we did is we started.
29:24We said OK,
29:25if it's tracking just the saliency,
29:27it should increase with physical
29:29intensity of a stimulus.
29:30Whether it's positive or negative.
29:32And so this is a increasing
29:34series of foot shocks on the left.
29:37That is,
29:38the simulations from that perceived
29:40salience term of our model,
29:41and on the right is the actual
29:44dopamine recorded responses to
29:46increasing intensities of foot shocks
29:47and what you see is that dopamine
29:50increases with foot shock intensity.
29:52It also increases when we
29:55increase the volume of sucrose.
29:57So a better than expected appetitive.
30:00Reward or the volume of quiet Night,
30:02which is a bitter taste scent,
30:04and so everyone always asks.
30:06And it's a great question.
30:07Foot shocks are kind of
30:09weird aversive stimuli,
30:10so for many of these tasks we have
30:12worked in other things that are aversive.
30:15But not, you know,
30:16painful.
30:16We can argue all day about whether
30:18foot shock is pain or something else,
30:20but we see the same pattern with
30:22other types of aversive stimulation,
30:24and so this is ruling out simple,
30:26rewarding coding by dopamine,
30:28because dopamine in the incumbents
30:29is going up to both appetitive
30:31and aversive stimuli.
30:32So it cannot be just rewards.
30:36So the other thing that influences
30:39perceived salience is novelty.
30:40So how much experience
30:42you have with something.
30:43So what we did is we took foot
30:46shocks of the same intensity.
30:49Intensity is not changed and we repeated
30:51them in a series on a fixed interval,
30:54so every every 60 seconds
30:56the animals got a foot shot.
30:59What we found is that doping goes down
31:02to the foot shots even though the
31:04intensity of the foot shock stays.
31:07Constant, so it's also not just
31:10encoding only the intensity,
31:12it's encoding other aspects
31:14like novelty as well,
31:16and so you get decreased domain
31:19responses to repeated exposures
31:22of stimulate both aversive.
31:24And neutral simulate,
31:25and so this is an auditory tone,
31:28so we've done this with tones,
31:30lights, and white noise.
31:32This is an example from white noise,
31:34but what you see which is
31:37important is the first exposure
31:39of a stimulus that is neutral,
31:41elicits dopamine and repeated exposures
31:44of that stimulus go down overtime.
31:47So you're getting decreased dopamine
31:49when animals are exposed repeatedly
31:51to stimuli in the environment,
31:53regardless of whether they have
31:55positive value, negative value,
31:57or are what we think of as neutral.
32:01I'm.
32:02So another aspect that I think,
32:05and this is really important experiment
32:07because it really rules out this
32:09reward prediction error based learning.
32:11So what we did is we trained animals to
32:13know spoke during a discriminative cue.
32:16So auditory cue comes on.
32:17If they respond they get sucrose.
32:21Without any signal,
32:22we now switch it to the same auditory cue.
32:25If they press,
32:27they get shocked,
32:28so we're switching that the Q is
32:30the same and it now represents
32:33a worse than expected outcome.
32:36And so.
32:36What should happen in a task
32:38like this is that the animals
32:41in this grezar simulation,
32:42the animal should reduce
32:44their behavioral responding.
32:46They do not surprisingly,
32:47this is a traditional kind
32:48of punishment task.
32:49Animals will reduce their behavior.
32:52But what our model predicts,
32:54this perceived salience model.
32:55Is that because there's unexpected
32:57information and it's novel?
32:59There should be increase
33:01in dopamine to this Q.
33:03Prediction error responding with
33:05say it should be decreased because
33:07it's a worse than expected outcome.
33:10A reward prediction error, excuse me.
33:12So what we're going to look
33:14at here is this Q,
33:16which is the last Q that predicted sucrose,
33:18so this is before the animals
33:20got gotten shocked.
33:21So this Q still has that predict sucrose,
33:24and then we're going to look at the next
33:27Q right after the first foot shock.
33:30And what we find is first,
33:33this foot shock causes a
33:35positive domain response.
33:36So it doesn't matter what kind of task,
33:40but shocks are being presented and
33:42they're always words resulting
33:44in positive domain responses.
33:46What the dopamine response to this
33:48discriminative cue actually goes up,
33:50even though it's it represents a
33:53worse than expected outcomes and
33:55so dopamine is increasing anytime
33:58information is novel or salient
34:00to the animal.
34:01And it's increasing even if
34:03the outcome is worse than
34:05expected or better than expected,
34:07and one of the key aspects of this
34:10experiment is dopamine is going up,
34:12even though the animal's
34:14behavior is going down,
34:15so increases in dopamine don't just
34:17mean motivated behavior or approach
34:19because we're getting increases in
34:21dopamine here that correlate with animals
34:24inhibiting a behavioral response,
34:25and so this kind of saliency.
34:27What it'll do, is it helps animals make
34:31adaptive updating of responses were.
34:33List of what the context of those responses
34:36are or the behavioral response necessary.
34:39So I'm gonna show I think
34:41I've one more experiment,
34:43so this experiment is
34:45actually really important,
34:46because it's kind of shows how much these
34:49kind of novel salients events that don't
34:53necessarily acquire value R to animals.
34:56So what we did is we did an experiment
34:59where we associated ECU with a foot shock,
35:02so just fear conditioning.
35:04But what we did is on some of
35:07the trials we just put a random
35:10irrelevant house light on.
35:12And what the model predicts is that because
35:14there is novelty in the environment,
35:16there will be an increase in dopamine
35:18response on these trials where novel
35:20information is added even though previous
35:23work has shown that novel irrelevant
35:25information will not acquire value.
35:27So the Q that we're adding this random
35:29irrelevant light won't acquire value
35:31because the animals already associated
35:34the previous Q with the foot shot.
35:36And So what we find is on trials
35:38where we add this novel light.
35:41There is a very large increase
35:43in the domain response,
35:44even though that novel light
35:46won't acquire value itself.
35:47And So what this does is it rules
35:50out the simple Attribution of some
35:52sort of balance to a queue or
35:55associative strength of that Q.
35:57It means that you're getting
35:59increases in dopamine responses.
36:00They don't necessarily correspond with
36:02the animal making Association between
36:05that queue and the outcome in some cases.
36:11The last thing, which again for
36:12people who are are really deep
36:14in the prediction based field.
36:16One thing that people can
36:17say at this point is, well,
36:19maybe doping is doing prediction error
36:21but not reward prediction error.
36:22So it's going up every time
36:24there is an error in prediction.
36:26This is actually really good good thought
36:28and we thought this too and we said OK.
36:30Well let's let's see if that is the
36:33case and we were kind of Gnostic here.
36:35We were saying, OK,
36:36let's just figure out what it does.
36:38We're not trying to.
36:40Pusha theory we're saying, well,
36:41how does the data fit together?
36:44So saliency or perceived salience?
36:46What it would suggest is that when
36:48you have a stimulus like a foot shock,
36:52you should have the biggest opening response,
36:55because when the stimulus is
36:56present and there it's the most
36:59salient 'cause most intense.
37:00But if you have a prediction of
37:03that during extinction or omission,
37:05there should still be a positive
37:08doping response,
37:08but it should be lower than when the
37:11stimulus is physically there a prediction.
37:14Error hypothesis would be that
37:16when you have an omission,
37:17this response should be higher
37:19than when the stimulus is there
37:21because it signals an error.
37:25We did this experiment.
37:27What we found was that there's a
37:29positive domain response at the time
37:31of the predicted foot shock when
37:33it's omitted, so it's not there,
37:35but the response of the foot shock
37:38itself is higher than when it's omitted.
37:40And so this also rules out
37:43other competing theories,
37:44which is that domain does
37:45prediction error learning,
37:46but it's not reward based.
37:50So. And I showed you a lot of stuff and,
37:54well, I kind of rule kind of come
37:56back and say like why should you care?
37:59So essentially what we did is we did
38:01a number of experiments to rule out
38:03these kind of competing factors of what
38:05dopamine is doing in learning and memory.
38:07Don't mean release is doing.
38:08I'm not saying the VTA cell
38:10bodies don't do this.
38:11Maybe they do,
38:11but there's integration of information
38:13at the level of the terminal that
38:15dictates how doing is actually
38:16releasing these brain regions and
38:17what I am saying is that dopamine
38:19release in the nucleus accumbens,
38:21core Maps on true perceived.
38:22Salience not prediction error or value.
38:27Our models are modeled behavior,
38:28we just use it to generate experiments
38:30we should run to parse different
38:32aspects of domain encoding,
38:33and So what you can do is use
38:35these kind of predictions to make
38:37experiments with other circuits as well,
38:39which I think is kind of an interesting
38:42way to approach the question.
38:45But what I'm showing you is that
38:47even in reinforcement context,
38:48pavlovi in context,
38:49this isn't a value based prediction signal,
38:51and these same signals are
38:53there in punishment tasks in
38:54negative reinforcement tasks,
38:55and so it's actually really
38:57interesting that you're seeing
38:58this kind of dopamine signal.
39:00It's very critical in driving behaviors,
39:02just not in the way that I
39:04think we predicted before.
39:07So why should we care?
39:08I think understanding what domain is
39:11doing is really important for disease,
39:13and so if you want to understand what
39:15dopamine is doing and what deficits in
39:18dopamine in a patient mean for that patient,
39:21it really requires a kind of
39:23holistic understanding.
39:24What domains doing across contexts and
39:26internal States and things like that,
39:28and so you know when you have a model and
39:32you say doesken dopamine fit this model,
39:35the answer might be yes.
39:37But it kind of leaves out that aspect of.
39:39But what is domain doing in other
39:42contexts of the model can't fit and
39:44so understanding the components that
39:46dry of these behaviors is really
39:49critical to understanding this.
39:51But I think that maybe more
39:53important thing for for kind of
39:55human health is is from, you know,
39:57my primary field which is
39:59addiction an it's understand.
40:00The difference between a dopamine
40:02signal that signals were worn
40:04and what a salience signal does.
40:05So a reward signal.
40:07If you have a deficits in a reward signal,
40:09you may say you know we don't
40:11want to increase those and people
40:13suffering from substance use disorder
40:15because if we do that may increase
40:17the rewarding value of stimuli
40:19in the environment like drugs.
40:21But the issue is with the salience signal.
40:24If you have deficits,
40:26it's going to slow the rate of learning
40:29for everything in the environment,
40:31so it could explain why people
40:34are compulsive because they don't
40:36respond to negative outcomes.
40:38It would explain why they have trouble
40:41learning. The adaptive alternatives.
40:42Are there an?
40:43It would explain why extinguishing
40:46drug associations is much slower,
40:48and so in if it's a saliency signal,
40:51we may want to.
40:52Increased opening so that people can
40:54learn adaptively in all of these contexts.
40:57And so again,
40:58I'm not saying that like you know,
41:00this is the end all be all dopamine is
41:02in lots of projection targets and it
41:04does lots of things in different areas.
41:07And we're in one single area,
41:08but I think kind of taking a step
41:10back and thinking about what these
41:12domains signatures really mean
41:14and what those deficits would
41:15mean to a behaving individual,
41:17as I think it's really important
41:19component of conceptualizing
41:20what these you know psychiatric
41:22deficits mean to people and how
41:24to best treat them.
41:25Anyway, so with that I'll
41:27end with thanking my lab,
41:29so Ganesh clue his background
41:30is in computational psychology,
41:31so he's like the modeler and he's
41:34really like driven this you know.
41:35Together he's a Pavlovian guy.
41:37I was a reinforcement person.
41:39I think this was like one of
41:40those projects that was this great
41:42synergism between two people who
41:44just we're really excited about.
41:46Kind of figuring out what's going on.
41:48Jennifer, Zachary and Patrick and
41:50Stephanie were are grad students
41:52that were working on this project
41:54and put a lot of time into it.
41:56Cody Siciliano and Lindsay Ann Lynn
41:58was was really nice and was helping us
42:01get the delight these optical sensors
42:03up and running in the lab fairly
42:06early in Cody's at optical engineer.
42:07So he helps a lot at Vanderbilt with
42:10getting these working correctly,
42:12validating them an my funding and I
42:14can take any questions you may have.
42:17So thank you so much.
42:22Thank you so much Aaron.
42:23If anybody doesn't want to pipe
42:25up with the just asking questions,
42:27please put them in the chat and
42:30I can read them out for Aaron.
42:33Like can I start with one?
42:35Go for it. So in terms of those
42:38projections you were discussing
42:40and you're looking in the core,
42:42do you think that just with predictions
42:44of what the core versus shell of the
42:47incumbents does in activations of the
42:49core versus Shell does to behavior?
42:51Do you think that one might be
42:54more important than the other in
42:56the prediction error?
42:57This is actually really
42:59great question an I should
43:00have pasted this.
43:01Actually I can do it now.
43:04I we have shell data so so.
43:06The answer is probably yes in some contexts,
43:09although when we we started to
43:12look through the shell data.
43:15So I just wanted to side when
43:17we started to look through the
43:20new data or the shell data.
43:22It did not look like what
43:24I would expect either.
43:28Basically. It still doesn't
43:31look like prediction error.
43:33So we still get a positive
43:36domain response to the shock.
43:38We get some scaling with stimulus intensity.
43:41Not quite as much.
43:44People have shown decreases in dopamine
43:46and to aversive stimuli and we have
43:49been working out why that would be when
43:51we aren't seeing them and we think.
43:53And so basically we did show
43:55decreases in domain in some contexts.
43:57Dopamine goes down when animals don't
44:00have to do anything or they have to wait.
44:02So what we did is we design
44:05this other experiment that I'm
44:06like really excited about.
44:08What we did is we trained animals to know
44:11smoker sucrose and then we switched the
44:13contingency so that they had to with.
44:16Hold a response and wait to get sucrose.
44:18So this is kind of like the same reciprocal
44:20thing to fear conditioning right?
44:22You have a queue.
44:23The animal just waits to get shocked.
44:25There's nothing they can
44:26do during that period.
44:27They wait,
44:28we see decreases in dopamine to
44:30fear conditioning Q and two AQ,
44:31where the animal gets sucrose at the end,
44:34but they have to wait to do it.
44:37And So what we think is happening is a
44:39lot of these decreases in domain people
44:42have seen or not necessarily just value.
44:45They have to do with what animals
44:47are doing and what novelty in
44:49salience do in an environment
44:51is they increase exploration.
44:53So if you need to decrease exploration
44:55and just wait for something to happen,
44:58domain goes down.
44:59So we get doping reductions even when
45:02the outcome is positive when the task
45:04design mimics that of the aversive
45:06tasks where people have seen reductions.
45:09And a lot of the like you know,
45:11there's a lot of great work from
45:13like Mitchell White Men looking at
45:15aversive like wine in the mouth.
45:17It's unavoidable.
45:17The animals are just waiting there as well.
45:20And so I think there's also
45:21differences in relative,
45:22you know,
45:23in his designs he has positive and
45:25negative stimuli in the same task,
45:27which are a little bit different
45:28than having an animal behaves.
45:30So obviously there's there is some
45:31sort of value based computation,
45:33but we think they're done and
45:35really specific context.
45:38Thank you.
45:39I think the other question go ahead.
45:44But they are zoom etiquette.
45:46Go ahead, I'll jump in after you.
45:49OK, thanks for bringing talker
45:50and that was really really cool.
45:52So one thing that you can do with
45:55these models is sort of see if they
45:57can predict particular phenomena
45:58and one of the ideas I think that's
46:01becoming more and more prevalent
46:02about a sort of teleological idea
46:04about what the dopamine system
46:06might be for is not rewards,
46:08not punishments as you've argued,
46:09but actually the causal
46:10structure of the world.
46:12Yeah, and so does your model
46:14predict things like sensory
46:15preconditioning where?
46:16There was
46:16no value at all.
46:18Initially you use that information
46:19later to imbue or impede value.
46:21Yes, yes, so actually this is one
46:23of the other powers of this model is
46:25it can do sensory preconditioning,
46:28Layton addition,
46:28so these are two things that that even
46:31the temporal difference models cannot do,
46:33and the problem is dopamine does them.
46:36So if dopamine does these
46:37in a computational model,
46:39can't that cannot be the
46:41computation domain is doing,
46:42and so we we have our next at
46:45once we get this out the door.
46:47I'm trying to find my.
46:49We haven't started doing sensory
46:50preconditioning yet because of the fact that
46:53it's a little bit more of a pain in mice,
46:56and I think we're going to
46:58have to switch to rats.
47:00Mice aren't like the best set
47:02like attending to things so,
47:04so lame.
47:05Inhibition is something that
47:06our model does do.
47:08So late inhibition is actually
47:10this really interesting novelty
47:11based learning constructs were
47:13essentially pre exposed stimuli,
47:14acquire values slower than simulated.
47:16Have not been pre exposed so familiar
47:18stimuli take longer to because you're
47:20basically unlearning the no Association.
47:23So a lot of these different models
47:25you know we brought this up earlier.
47:28I was asked about Pierce Hall Macintosh.
47:30Like all of these models have
47:33added these components to do this
47:35and our model does this and the
47:37sensory preconditioning.
47:39I showed you the dobine goes
47:41down to repeated shocks.
47:42Um?
47:42We can get pre exposed stimuli to
47:45have less associative value and
47:47what's really interesting is that
47:50the dopamine response to these pre
47:53exposed stimuli is much lower and it
47:56also tracks over the pre exposure period.
47:59So these these kind of non value based
48:01learning constructs were previous
48:03experience is changing the way that
48:05stimuli can drive future behavior
48:07or sensory preconditioning were two
48:10irrelevant simulate form associations
48:12that can then be associated later.
48:14Our model does it,
48:15and dopamine still Maps onto that
48:17perceived salience term in those contexts.
48:19And this is actually why we were so set on.
48:21You know,
48:22the first experiments really well
48:23it could be other things too,
48:25and then we started going
48:26into these latent addition and
48:27sensory preconditioning ideas,
48:29because those really can't be other things.
48:31I mean,
48:31it could be there's other components of it,
48:34but I think it is more strong
48:35with the other stuff.
48:37It does too, that that's what it's doing.
48:39But it's that's a great, I think.
48:41Those are like the killer like knife
48:43in the coffin experiments, right?
48:44Because they just.
48:45Those other models cannot do them,
48:47so yeah, that's a great great point.
48:51I think we had a question,
48:52thanks. Those are really great talk here
48:54and thanks for taking us through all that.
48:56And you have touched on my question
48:58a little bit because you started.
49:00I mean, even with the default
49:02Pomeranians when he starts to
49:03look at nucleus accumbens shell.
49:04But one thing I was curious about with your
49:07model is how how global do you think it is?
49:10And you hinted as we talked about the
49:12VTA component as well. So you know,
49:14going back to some of the Bromberg,
49:16Martin work about different VTA
49:17neurons responding differently
49:18to balance versus silence.
49:19You think this Maps onto multiple regions
49:21using this sub regions within the.
49:23The common score is a global number
49:25as a discrete to specific places.
49:27Yeah, so this is a great question,
49:29so I'm lucky to have married very well in
49:32my life and my partner is a two photon guide.
49:35When he does,
49:36he does 2 photon imaging through grin lenses,
49:39cranial windows, you name it and
49:40so one thing we're working on now.
49:43The really power of of these kind
49:45of optical imaging approaches is
49:47that you can record as small as
49:49you want or as big as you want,
49:51and so depending on your.
49:53Microscope and resolution so
49:54we're moving into these.
49:55Either I love slice work.
49:57This is like my background so
49:59we're moving into these kind of.
50:00In vivo in slice imaging approaches
50:02to understand better domain
50:04regulation across big and small areas.
50:06Because the thing about domain
50:07neurons that so you know,
50:09like kind of weird about
50:11them is their projection.
50:12Like you know, Arborization is insane.
50:14If you fill a single domain on the PTA
50:17and look at the field that it populates,
50:20it's like half the straight up.
50:22But then if you look at these specific
50:24release sites on these neurons,
50:26it's not releasing dopamine
50:27at everywhere every time.
50:29It's depending on all these different things.
50:31So this.
50:32Release structure is so complicated and
50:33I think part of the reason people have
50:36been so like oh volume transmission
50:38is our ability to really look at these
50:40granularities between these components.
50:42And so we're starting to go,
50:44you know, start big.
50:45We're just saying,
50:46OK,
50:46if we do image in a bigger field or
50:49with multiple sites at the same time,
50:51are we seeing differences?
50:52We do see differences so I think there
50:55are differences in these VTA neurons in
50:57what what they're doing in different areas.
50:59So I don't think like all dopamine is.
51:02This I think don't mean to the core is this,
51:05but it also makes sense that domain
51:06in the core that's been tide more to
51:08instrumental responding than like the shell.
51:10That's like these acquisition.
51:11And like Valeant space kind of learning
51:13would look like a perceived salience term,
51:15right?
51:15That makes way more sense for something
51:17that is involved in punishment,
51:18negative reinforcement,
51:19positive reinforcement,
51:19which are the same motivated responses,
51:21independent violence.
51:22So we're doing some more stuff in the
51:24shell. You know, I'm not sold that the
51:26shell doesn't do value because I don't think
51:28foot shocks are the best way to do stuff.
51:30I think foot shocks are weird.
51:32Stimulus that are really powerful initially,
51:34but we didn't really evolve
51:36to respond to foot shocks,
51:37so we're starting to go in
51:39more with things like Quy 9.
51:41We've been developing.
51:42You can make them liquor lic ometer hot,
51:45so we've been doing like thermal,
51:47like not pain, but thermal sensitivity
51:49curves so that you can look at thermal
51:51stimuli that reduced responding.
51:53But without this, like foot shot component,
51:55so we're trying to parse this out.
51:57I'm not sold that,
51:58it's just like every dopamine responses
52:00that I think it's more complicated,
52:02but I think we need better resolution
52:05techniques to really parse that.
52:07And hopefully over the next I don't know.
52:09However long my career last
52:10will see will start to get it.
52:11Some of those questions and
52:12other people are doing that too.
52:14I mean,
52:14there's some really great work
52:15coming out where people are
52:16using those like single synapse.
52:18You know,
52:18you know Uncaging and Eli and
52:19all kinds of crazy stuff,
52:21so I'm excited to see where the field goes.
52:23Yeah,
52:23that's really excited how
52:24great you are looking
52:25at. I'm glad you're looking
52:26into it. Sounds like
52:27you thought about it already. Definitely
52:28I went a little bit. Now
52:29the question is just like do we
52:31have the tools and then the next
52:33thing is do we have the month of
52:34money and the people that do it?
52:36And so it's like you know.
52:38You you see what you can do and
52:39with the resources you have so.
52:44There is a question in the chat by
52:47from Denise, Baghdad and Denise.
52:48Do you want to read it out or
52:50would you like me to ask it?
52:53Good morning, great talk,
52:56so I just wanted to understand something.
53:00Maybe it's not a great question.
53:03Let me just say it.
53:06So nicotine reinforcement is
53:09generally considered as positive
53:11reinforcement enforcement.
53:12However, it's also discussed
53:14about like the weather.
53:17Nicotine reinforcement is
53:18negative reinforcement.
53:20Becausw of nicotine withdrawal.
53:22The compost.
53:23Open intake and taking is actually
53:27contributes to nicotine reinforcement,
53:29so I am interested in whether your
53:33model could dissect the positive or
53:36negative reinforcement for the nicotine.
53:39It's I know it's your.
53:44Shock, but this is the one molecule.
53:47You know, so could have both.
53:49So how we put the fact
53:51that's a great question.
53:53So one of the things in the addiction
53:55field is that there are all these
53:57series of negative reinforcement, right?
53:59Like opioid withdrawal alcohol,
54:01withdrawal.
54:01All of these are negative
54:03reinforcement concepts,
54:03but no one actually does
54:05negative reinforcement.
54:06We make the inference that is negative
54:08reinforcement from the fact that it
54:10causes withdrawn animals are taking
54:12it with during the withdrawal period.
54:15It's OK,
54:15so it's a hard question.
54:17I think the first step would
54:19be to look at how you know if
54:22negative reinforcement processes,
54:23like for avoiding shocks,
54:25are changed after nicotine.
54:26So one of the things that we're
54:28working with with Cody Siciliano is
54:30looking at how alcohol changes animals
54:32motivation for negative reinforcers,
54:34and so that's like the first step.
54:37I think this is like it's
54:39a hard thing to parse.
54:41Nicotine is also, I know it's like I do.
54:45A cholinergic regulation
54:46of dopamine terminals.
54:47So nicotine is like in my mind,
54:49but like we don't do nicotine
54:51reinforcement stuff.
54:51It's also this really interesting
54:53molecule because it regulates
54:54like how dopamine is released
54:56in a really interesting way.
54:57That's not just like up,
54:59it's changing like phasic responses
55:00to stimuli in the environment.
55:02And so thinking about the
55:04interaction between those,
55:05it's like we're doing some work
55:07with sex differences in that
55:08system is like so much more
55:10complicated than I want it to be.
55:12Like with cocaine,
55:13it's like it binds to the transporter.
55:15Show me goes up.
55:17Can we reduce that nicotine is like Oh
55:19well in some cases domain goes down
55:22some cases it goes up and so it's
55:24just such a complicated question.
55:25I think the behavioral stuff we do
55:27can start to parse how processes and
55:29animals are changing by exposure,
55:31and I think that's the first step
55:33and then the next step is trying.
55:35We're trying to develop task
55:36to figure out how to do this.
55:38So one thing we've been thinking
55:41about is doing.
55:42Old school drag discrimination so
55:44animals will actually press before
55:46like to tell you an internal state
55:48is X or Y and what we want to do
55:50is we've been thinking about doing
55:52this with optogenetics, right?
55:53Does an optical stimulation of a
55:55circuit substitute for X drug X
55:57state and I think that some of these
55:58withdrawal effects you could see if,
56:00like nicotine withdrawal,
56:01substituted for some of these other things,
56:03and if that was a critical component
56:05of reinforcement isn't hard question.
56:07I think that that's it can
56:09start answering that question,
56:10but I've been thinking about this a lot
56:12and I'm not sure how to specifically.
56:14Parse when an animal is doing something
56:17for two things at the same time.
56:18What component is what I?
56:20I wish I had better answer.
56:22I'm excited about the question,
56:23but I don't have the answer for you.
56:29Is there time for one more Marina
56:32or do we directly is that Beth?
56:34No, it's less less sorry.
56:37I had one too. OK first Liz then Jane.
56:42So Aaron, that was such a beautiful
56:43talk and I love all the different
56:46behavioral experiments that
56:47were inspired by your model.
56:48And one of the powers of this
56:50model is obviously you could
56:52take that salience term out.
56:53And guess how behavior would be
56:55altered by it in the future.
56:57So I'm curious whether you're going
56:59to start looking at blocking these
57:00signals and seeing whether they
57:02match the expectations that the
57:04model would make in particular,
57:05that when you were showing the responses
57:07to the light cue during conditioning,
57:09which shouldn't be involving any learning
57:11like what's the point of that signal?
57:14Our behavior could come from it.
57:15I'm
57:16so excited to just ask me this one.
57:18OK, so there's.
57:21I'm lazy and I don't want to.
57:22Maybe I'll do this why I
57:24should have put these in here.
57:25I didn't think people were going to have.
57:27Not that I didn't think you
57:28would have great questions,
57:29but I didn't think you guys were to ask
57:32questions that I had like specific data for.
57:34OK, so two things. First thing first.
57:37Wow, that looks terrible.
57:39We did do experiments to
57:40eliminate this signal.
57:41You can see I'm like really crafty with this.
57:44This is not my OK so we First things first.
57:47Yes, I'll tell you what I think
57:49that signal is doing and then two.
57:51Well, two.
57:52I'll show you the optic Jenner.
57:54Other we're not.
57:55We're almost done with the
57:56Histology so take this with it.
57:58This is preliminary preliminary ish.
58:00We did two experiments where we
58:02inhibited the signal using what our
58:04model would predict as the condition
58:06response that would dissociate
58:07it from these other components.
58:09Injected Halo rhodopsin in TH
58:11positive neurons in the VTA and
58:13then inhibited the terminal.
58:14So we're only inhibiting
58:15dopamine releasing terminals.
58:16Any comments?
58:18We either inhibited during a Q
58:22predicting fear conditioning.
58:25Or we inhibited or?
58:27We know we stimulated during.
58:28Sorry this is my fault.
58:30We stimulated during these are
58:32two different things.
58:33We stimulated during a fear conditioning
58:36Q or we stimulated channelrhodopsin
58:38during an emitted but expected shock.
58:41If you stimulate and this gets
58:43your questions during a Q,
58:45that's a fair condition.
58:46Q You actually get less freezing,
58:49so this is the opposite of what you
58:51would expect from associative strength,
58:53but it kind of person.
58:55We basically points to this question when
58:58there's novel stimuli in the environment,
59:00you increase exploration.
59:01All of our data,
59:03like the novel Q.
59:04All of our data show that
59:06when you add novelty,
59:08you increase dopamine and the increased
59:10dopamine is associated with more exploration.
59:12And less freezing.
59:13And so these novelty terms,
59:15what they're doing is they're
59:17helping animals to adaptively
59:18learn by increasing exploration.
59:20And like here, decreasing freezing.
59:22So in the same animal,
59:23the queue that wasn't stimulated has just
59:26as much freezing is the wifey group,
59:28and when it's stimulated they freeze less
59:31and that's what our model would predict.
59:33The other thing we did is we show
59:36that we can prevent extinction,
59:38freezing extinction by
59:39stimulating dopamine to the Q,
59:41and so we both prevent extinction.
59:43We prevent extinction by basically
59:45like increasing the salience of that
59:47event so that it doesn't go away,
59:49which is not the same as you'd expect
59:52by these prediction error terms.
59:54So what we think is happening
59:55is that these novel stimuli
59:57are increasing dopamine that
59:58increase in dopamine promotes and.
01:00:00Exploration term,
01:00:01we also did some like deep lab cut based,
01:00:04you know,
01:00:05machine learning algorithms to
01:00:06look at orienting responses.
01:00:08It's not associated with
01:00:09general motor activity,
01:00:10it's associated with orientation
01:00:12towards the novel stimulus.
01:00:13And so we think that these these
01:00:15this saliency term in the Commons
01:00:17is changing the way the animals are
01:00:19interacting with the environment
01:00:21rather than just the associated value.
01:00:23But the problem is these are such
01:00:26complex things to dissociate that.
01:00:28I understand why people solve
01:00:29the data before instead.
01:00:31Oh, it's that balance goes up to rewards,
01:00:33goes down to a fear conditioning Q
01:00:35that looks like violence to me too.
01:00:37You only start to see that
01:00:38it can't be balanced.
01:00:39When you do these kind of really
01:00:41in the weeds like someone showed
01:00:43this in 1950 in psychology,
01:00:44we're gonna do this again with optogenetics
01:00:46kinds of experiments which I don't know.
01:00:48I think those are the fun experiments,
01:00:50but does that answer your question?
01:00:51Yes, thank you, awesome.
01:00:53I think James next and then Rick.
01:00:55Hi that was a great talk.
01:00:59So my question is.
01:01:02You talk about increasing dopamine,
01:01:05and in most of your experiments where
01:01:08you're actually measuring dopamine,
01:01:10you're looking at.
01:01:12Using Delight an if you go back to the
01:01:16not the old psychology experiments,
01:01:19but the 80s dopamine literature.
01:01:21People think about tonic versus phasic,
01:01:24dopamine and a lot of your experiments
01:01:26seem to be focused more on what
01:01:30the phasic dopamine signal is.
01:01:32An I'm wondering whether you have
01:01:35some way that you can simultaneously
01:01:38look at dopamine tone because it
01:01:40may be that things like novelty.
01:01:43Might actually be linked to some
01:01:45of those and you get you know
01:01:47interactions between the two,
01:01:49and you know when you're you're seeing
01:01:51your increase with the delight.
01:01:53What baseline is that on?
01:01:54This is,
01:01:55I'm like you guys
01:01:56are like making my day.
01:01:58I have like I'm this is
01:02:00such a great question.
01:02:02So this is like our first like thing.
01:02:04We're getting out the door.
01:02:06We have a bunch of extra data where what
01:02:09we've been doing and this is the thing.
01:02:12OK, so voltammetry.
01:02:13Is background subtracted?
01:02:14So you can't really get both tonan,
01:02:17phasic stuff in the same experiment.
01:02:19So what people historically done as
01:02:21they said, microanalysis histone,
01:02:23gigha Ruth and voltammetry is
01:02:24is phasic an my the 80s domain.
01:02:27Literatures like where I started my
01:02:29career so I'm very excited about that.
01:02:31Do you like this kind of nice because
01:02:34you have some photobleaching but you
01:02:36can control for that and you don't
01:02:39know what the the problem with it
01:02:41is that you don't know the number.
01:02:44So with microdialysis you get an amount.
01:02:46With voltammetry you calibrate your probe,
01:02:47you have an estimated amount with delight.
01:02:51I haven't found a great way to
01:02:53figure out what the number is,
01:02:54but you can look at relative
01:02:57changes over the session.
01:02:58I don't have this data up and is easier
01:03:01way because we're putting it into
01:03:02something so we have a manuscript that
01:03:05we're getting together and its focus
01:03:07on novelty based changes and joking signals,
01:03:10and so it's focused on Lane in addition,
01:03:12But what we're looking at is
01:03:14the phasic response relative
01:03:15to longer changes in dopamine,
01:03:17and I don't want to call it tone
01:03:20because it's over like 10 minutes.
01:03:22Not like ours,
01:03:23but it's definitely not what you
01:03:25would call a phasic fast response.
01:03:27What novelty in the environment does?
01:03:29Is it increases that phasic response?
01:03:31But then it does it.
01:03:32The baseline is much higher.
01:03:34So what you have is this shift and
01:03:36what we think is happening is that
01:03:38the novelty is changing the state of
01:03:41the system so that if the next thing
01:03:43that's encountered in that situation
01:03:45the domain response will be bigger.
01:03:47Now one of the things I've always
01:03:49been interested in over my career
01:03:51is what matters for the animal that
01:03:53change from baseline or the peak.
01:03:55And so we're trying to get into that.
01:03:57Now to say like, OK, that increasing.
01:04:00Slide does that just increase the peak?
01:04:03Or does that actually still amplify
01:04:05more this signal to noise and so these
01:04:09novelty terms are definitely changing.
01:04:11What I would call.
01:04:13I don't want to,
01:04:15but they are definitely changing
01:04:16these slower baseline fluctuations in
01:04:18dopamine over longer periods of time,
01:04:20and we think that's really important
01:04:22for the effects of novelty on
01:04:24other types of learning,
01:04:25and so this dopamine,
01:04:26perceived salience,
01:04:27perceived failings is influenced by novelty,
01:04:29so anything that changes novelty
01:04:31will change this as well,
01:04:32and so it's really important in
01:04:34these novelty based learning
01:04:36things on these both slow and fast
01:04:38timescales in a way that is,
01:04:40I think,
01:04:40consistent with what people have seen
01:04:42with microdialysis and voltammetry,
01:04:44but in a more you are able to more granularly
01:04:46relate them with this, But again,
01:04:48you don't have the amount, so I kind of,
01:04:50you know, it's hard because there's no
01:04:52like calibration like you can't say,
01:04:54oh, this is the amount,
01:04:55and I think that's where I hesitate
01:04:57a little bit to make these really
01:04:59strong conclusions about like what.
01:05:00It is, I know that it's changed from
01:05:02the minute one of this session.
01:05:04The question is like exactly what is that?
01:05:06But there are those changes and I think
01:05:08that's a great point that people.
01:05:10I think a lot of people have gone into
01:05:12these kind of optical measurements and they.
01:05:15That was ignored,
01:05:16but they're not necessarily rooted in these,
01:05:19like microdialysis.
01:05:19Will Tanistry fields,
01:05:21and so they don't understand that
01:05:23there has been a ton of work parsing
01:05:26what these tonic changes mean?
01:05:28How tonic dopamine is regulated relative
01:05:31to like fast release like the domain
01:05:33transporters? My favorite protein because
01:05:35I guess the question I guess I'm asking is,
01:05:39would you see some more reward evidence
01:05:42of more reward prediction error if
01:05:45you somehow subtracted what the?
01:05:47Basil Change is an are you missing some
01:05:50of that? Because your tonic salience?
01:05:52I mean, you started out talking a lot about
01:05:55how dopamine could be doing this and that,
01:05:58but it also could be doing all of it.
01:06:01You know it's not mutually exclusive.
01:06:04It could be.
01:06:05It could be let me.
01:06:07I do have a data that answers
01:06:09that question because actually a
01:06:11really astute reviewer asked us.
01:06:13Now I need to find it. It wasn't me.
01:06:17They they they asked us saying,
01:06:18oh, there it is, we did this,
01:06:21so this is actually a really,
01:06:22really good question.
01:06:23Why is this coming up?
01:06:25Oh I'm like not looking at the right file.
01:06:27Maybe if I can get it in like
01:06:302 seconds I'm going to write,
01:06:31But basically we did what you
01:06:33your question is a good one.
01:06:35I think one of the things that they ask,
01:06:37which is a great question.
01:06:39We'd already been thinking about
01:06:40this so we were like, OK, cool was.
01:06:43If these changes in baseline are the
01:06:45reason that we don't see changes.
01:06:47In in a, let's see file new release.
01:06:51All these will be in here now. Insert.
01:06:55Side OK. OK, So what we did here?
01:07:01Is we had that repeated shock
01:07:04experiment and their question
01:07:05was kind of what yours is like.
01:07:07Well, maybe the difference is the baseline,
01:07:09like the baseline is changing overtime
01:07:11and so we calculated the shocks in
01:07:14the original stuff I showed from like
01:07:16a 2 second window before the event
01:07:18and then we went back and calculated
01:07:20all the shocks from a global baseline
01:07:23at the very beginning of each trial.
01:07:25And what we found is that
01:07:27the data is correlated,
01:07:28like very highly correlated.
01:07:29And if we look at the baseline change.
01:07:32Over that trial it is not changing,
01:07:34so it can't explain all of it.
01:07:36Like I understand,
01:07:37I do agree that there is a lot of
01:07:39baseline stuff that's this where we
01:07:41are like kind of taking out here that I
01:07:43think does paint a more complex picture,
01:07:45but I think for a lot of these
01:07:47things that would say OK,
01:07:48this is definitely not our PE.
01:07:50The baseline is not the factor
01:07:52that's driving all of it,
01:07:53but I think that's something actually
01:07:54that people should be thinking
01:07:56about in these papers and everyone
01:07:57does imaging now and they're all
01:07:59doing everything is baseline,
01:08:00But there's like slower changes
01:08:01that are totally left out of that.
01:08:03They're going to change the way
01:08:05you interpret the data.
01:08:06But that's I mean it's a great question.
01:08:08It will look into this more
01:08:09'cause I think maybe it does.
01:08:11All of that is totally reasonable.
01:08:13Explanation and it might be like nice at
01:08:15a different synapses and the same area,
01:08:16and so like how to parse those.
01:08:18I think it's important.
01:08:21Last question from Rick.
01:08:23The outstanding thank you.
01:08:25Very much so this is I'm not
01:08:27sure exactly this question,
01:08:28but since there's so much regulation
01:08:30and release at terminals Strike and dip
01:08:33compared to mean cell body activity,
01:08:34do you think that other neurotransmitters
01:08:36responsible for the specific
01:08:38components of your learning model
01:08:39and like dopamine release the end
01:08:41result and integration of those?
01:08:43And can you speak this one?
01:08:45I can think of right off the bat.
01:08:47So one thing we think is really
01:08:49important for this is acetylcholine
01:08:52regulation of dopamine release.
01:08:54We we have started to do some of this
01:08:56where we're starting on the slice level
01:08:58trying to workout these parameters
01:09:00because it's like calling regulation
01:09:02of domain releases actually really
01:09:04kind of cool regulatory mechanism
01:09:06because it's one of the few that really
01:09:09robustly releases domain from terminals,
01:09:11totally independent of the semantic activity.
01:09:13And you can even get this in
01:09:15isolated terminals and slices.
01:09:17And so we think that there is probably,
01:09:19you know, maybe it's the.
01:09:21Maybe it's both questions.
01:09:23Maybe Domain was released.
01:09:24In response to RP signals,
01:09:26but also attentional signals
01:09:27that are coming from these,
01:09:29maybe like that's the first thing I think
01:09:32of acetylcholine is like attention arousal,
01:09:34but here's the thing.
01:09:36Those are all integrated.
01:09:37What the domain signature finally is,
01:09:39and so I think that it is
01:09:41probably both in some contexts,
01:09:43right?
01:09:44It's probably both acetylcholine
01:09:45regulated domain release and RPE
01:09:47regulated cymatic activity that leads
01:09:49to this kind of saliency based term.
01:09:51One thing I'll tell you some
01:09:53ants about it as we see pretty
01:09:56interesting sex differences.
01:09:57In terminal regulatory
01:09:58mechanisms through the A4 beta,
01:10:00two containing nicotinic receptors
01:10:02that are on those terminals,
01:10:04and So what we're trying to do first
01:10:06is outline the mechanism or how this
01:10:09is different and then take that
01:10:11model into this and say do those
01:10:14differences predict differences
01:10:15in these learning parameters
01:10:16because it's really hard to get it.
01:10:19Don't mean terminal regulation by
01:10:21acetylcholine in vivo because it's
01:10:22your calling regulates other cell
01:10:24types that also regulate dopamine release.
01:10:26So it regulates GABA release.
01:10:28That regulates terminals and so,
01:10:30like without some sort of like biological
01:10:33system that you know is different.
01:10:35It's really hard to isolate just
01:10:37the effects of cetyl choline from
01:10:40the effects of acetylcholine media
01:10:43dopamine terminal regulation.
01:10:44I hope tools come along.
01:10:45There's like darts and stuff
01:10:47which are awesome.
01:10:48Those are coming and like we'll see.
01:10:49But like anyway I'm like this is a that's
01:10:51a great question I'm excited about so.
01:10:55Well thank you were coming up to 11:30
01:10:57and we peppered you with questions
01:10:59we really appreciate the great talk
01:11:01and the time that you spent with us.
01:11:03Thank you to everyone for your attention
01:11:05and we hope to hear more about the
01:11:08next steps in the research. Yeah,
01:11:09thank you guys so much.
01:11:11These questions were awesome man.
01:11:12I had a great time
01:11:14chatting with everybody so
01:11:15thanks. Thanks Erin.