CPM_Dome_default

Name: CPM_Dome_default
Uploaded: 2026-07-28T13:05:47.89Z
Duration: 1 h 3 min 10 s

July 28, 2026

Information

ID: 14363
To Cite: DCA Citation Guide

Download Transcript

00:00Thank you so much for
00:01joining today, and I'm just
00:03going to immediately
00:04pass to
00:06Alyssa because our MAPS group
00:09is just in supporting role
00:10here, and the whole idea
00:12and organization
00:14came from
00:15doctor Ku, and this is
00:16your show. So thank you.
00:18Great. Thank you, Helen. So
00:20it's a pleasure to introduce
00:21doctor Lynette Dohm today. Lynette
00:24is currently a postdoctoral researcher
00:27in Tobias Hauser's lab at
00:29University of Tubingen, and he's
00:31also a guest scientist at
00:33the Max Planck Institute of
00:34Biological Cybernetics in Tubingen.
00:37He completed his PhD at
00:39the University of Plymouth in
00:40the UK
00:42under the supervision
00:43of Andy Wills,
00:45And his research focuses
00:47on modeling human heterogeneity,
00:50with a specific interest in
00:52how intelligent systems
00:54fail and also effect centric
00:56research on what intelligent systems
00:59fail and then with a
01:00focus on non uniform generalizations
01:02of learned experience.
01:05And beyond that, Leonard has
01:06also created innovative frameworks for
01:08large scale model comparisons
01:10and most recently
01:12developed a modeling toolbox for
01:14computational psychiatry and cognitive behavioral
01:17tasks,
01:18in Tobias Haus' lab.
01:20And as an early adopter
01:21of this,
01:22toolbox myself, I can personally
01:24vouch for how incredibly useful
01:26it is.
01:27And, yeah, we're very excited
01:28to hear more about it
01:29today.
01:31So with that, I'll hand
01:32over to Leonard.
01:35Thank you very much, Alisa.
01:37So I'm just gonna share
01:38my screen. Hopefully, nothing's gonna
01:40break.
01:43Amazing. It seems like it's
01:44shared. Do you guys see
01:45it?
01:46Mhmm. Yeah. Great. So hi.
01:49I'm Leonard. Yeah. And I'm
01:51joining from Germany, Tuebingen.
01:53And I'm gonna talk about
01:55this science software,
01:56CPM, that we built to
01:59kinda help you and support
02:00you
02:01getting your
02:02theory theory and modeling done
02:04in computational psychiatry.
02:05Now,
02:07in the talk, I will,
02:08of course, introduce some of
02:10our rationale
02:12and going to
02:15I'm going to,
02:17walk you through how to
02:18interact with the toolbox and
02:19then
02:20emphasize some features that might
02:22be like,
02:23that I think are very
02:24prominent
02:25in our designs.
02:27So computational modeling is in
02:29the name, of course, so
02:30I'm going to start off
02:31by
02:33giving some rationale of why
02:35we are using computational models
02:36to begin with. My favorite
02:38take on this is that
02:39models are
02:41essentially
02:42implementations
02:43of specific theories, psychological theories
02:45that we have about how
02:46humans work.
02:48So models are
02:49essentially just ways to remove
02:51the theory from the brain
02:52of its authors. I love
02:54it because it means that
02:55those are independent.
02:57So no matter who you
02:58are, no matter the prestige,
03:00the model is very specifically
03:02says something should happen so
03:04we can
03:05clear directly compare it with
03:07observations
03:08that we make in our
03:09experiments
03:10and what is either wrong
03:12or right.
03:13And if it's, like,
03:15adequately right, we can generate
03:17hypotheses from these models.
03:18And we can do these
03:19things because the models are
03:21essentially ambiguity reduction machines.
03:24They are make us
03:26like, force us to be
03:27very precise about
03:29what we think psychological processes
03:31are, how they work so
03:33that they can make very
03:34specific and precise predictions that
03:37we can, as I said,
03:38compare with human behavior.
03:39And this,
03:40and these two features, I
03:42think, one of the more
03:43important ones for computational psychiatry
03:46because we are interested in
03:48how cognitive impairment can be
03:49translated into symptoms. Models
03:52have mechanisms
03:53that are very clearly defined.
03:55So we can map any
03:57kind of, like, change
03:58in those mechanism
03:59to how symptoms manifest in
04:02certain mental disorders.
04:03And
04:05they are very precise,
04:06so they can make very
04:08specific predictions
04:09about behavioral and clinical data.
04:11And this is very important
04:13because clinical data is very
04:14precious.
04:15Clinical data is hard to
04:16acquire. There are a lot
04:17of ethics surrounding it, so
04:19it takes a lot of
04:20time to even get to
04:21the point where you can
04:21collect it
04:23and often involves, like, a
04:25lot of effort. So you
04:26don't want to
04:28mess around a lot with
04:29the data.
04:30You want to make want
04:32to be able to be
04:33very precise and very robust.
04:35So models help with those
04:37type of inferences.
04:38And, eventually, hopefully, this kind
04:40of, like, feeds into how
04:42we design interventions because models
04:44help us to further our
04:46understanding about these cognitive processes
04:48and how they are translated
04:50to symptoms.
04:51Now this is all good.
04:52It sounds great as a
04:53sales pitch for combinational psychiatry.
04:55Unfortunately,
04:56it's not that simple.
04:58Computational modeling relies on huge
05:00amounts of code written by
05:01nonexperts.
05:03Usually, the PhD students who
05:04come to come into
05:06the field because they want
05:07to study mental disorder, not
05:09because they want to study
05:11low level programming or figure
05:13out how to parallelize
05:14on CPU course without breaking
05:16the computer. Right?
05:18And that leads us to
05:20the problem of, like,
05:21everybody writing their own code.
05:23Everybody reinventing the wheel a
05:25lot of times across different
05:26labs. So it's resolved in
05:28this kind of fragmentation
05:29when there are a lot
05:30of changes made in the
05:32code that the so that
05:34it works for your specific
05:35use case, so for your
05:36specific
05:38for your specific experiment.
05:39And this results in a
05:40lot of inconsistencies
05:42in how model is applied
05:44and inconsistencies
05:45in how methods or parameter
05:47estimations are carried out. Now
05:49this is a problem because
05:51it means there are undocumented
05:53assumptions in the code often
05:55buried under thousands of lines
05:56of code that no human
05:58wants to read. Now you
05:59can ask chat GPD to
06:00read it, but that still
06:02involves an extra effort in
06:03the process.
06:04And these
06:06and these codes, of course,
06:08have very weak documentations.
06:10These are not simply undocumented.
06:12They are often just like
06:13code comments
06:14that you have to find
06:16and you have to look
06:17for.
06:17And
06:19this
06:20this kind of creates a
06:21huge barrier for adoption and
06:23creation creates a huge barrier
06:25for repreusability
06:26as well. It's it's this,
06:28it worked on my laptop
06:30moments. I know we all
06:31encountered it that some code
06:33ran for,
06:34my,
06:36PhD peer laptop, but it
06:38did not run on my
06:38laptop. It's a it's a
06:40very common experience.
06:41And this reproducibility is a
06:43huge issue because we are
06:44dealing with clinical data. We
06:46are dealing with interventions that
06:47and we hope to apply
06:49what we find in our
06:50research.
06:51But
06:52this is the reproducibility
06:54is a requirement for it.
06:57So in on top of
06:59these two things,
07:02they kinda interact to create
07:03these, like, very high technical
07:05barriers so that when people
07:06come in,
07:07they have to learn a
07:08lot of jargon. They have
07:09a lot of, like, inaccessible
07:11technical terms that they have
07:12to figure out, especially if
07:14you code. It's,
07:15it can be a quite
07:16daunting process.
07:18And this results in very
07:19slow onboarding. It takes time
07:21to learn. Often when we
07:23hire intern here as well,
07:24we want to make sure
07:26that they have programming experience
07:28because it's much easier to
07:30teach psychology to people than
07:31coding.
07:33It's, unfortunate, but it is
07:34true. And
07:36that,
07:37results in a very limited
07:38reuse of the code and
07:40the implementations that people share.
07:43So what we think is
07:44a solution for at least
07:45some of it or at
07:46least to alleviate this problem
07:47somewhat
07:48is that,
07:50we need shared, transparent, and
07:52tested workflows,
07:53something that is accessible, something
07:55that is tested,
07:56and can be reliable.
07:58Enter CPM. Now CPM is
08:00a Python library.
08:02It is an open source
08:03in general purpose like a
08:04toolbox for computational modeling, and
08:07it does exactly what, what
08:09I just said said we
08:10want, implements standardized workflows.
08:13Now it covers all aspects
08:15like simulation, fitting, model comparison,
08:17and evaluation.
08:18And it does so by
08:20having,
08:21this level of modularity
08:24that tasks, models, inference, metrics
08:26are all kinda
08:27implemented in a way that
08:28they can be be swapped
08:30so that people can play
08:31around.
08:32Mistakes are not costly.
08:34You can just, like, swap
08:36components. You swap the model.
08:38You swap the optimizer.
08:39And it's very easy to
08:41just, like, play around and
08:42experiment,
08:43which is really a really
08:44good educational tool as well,
08:45but also helps you to
08:47figure out what is going
08:48on inside your research.
08:51And this modularity is the
08:52one that really helps
08:54with accommodating novices and also
08:56power users. Now if you're
08:58a novice, you probably want
08:59to get going with things.
09:01You want to have some
09:02success early on in the
09:04modeling, in your modeling journey
09:05because that helps you to
09:07develop competence. So this modularity
09:09allows people to
09:11enter at the stage that
09:12is appropriate for their experience
09:14so they can get on
09:15with very established, standardized workflows.
09:18Or
09:19if they gain competence or
09:21you are power they are
09:22power users, they can mix
09:24and match and also hack
09:25the system.
09:26So this modularity helps us
09:28quite a bit in accommodating
09:30both novices
09:31and more,
09:33like, nerds.
09:35The the main thing that
09:37I wanna get across here
09:38is that our goal here
09:39with the standardization, with the
09:41modularity, is that
09:43we want to make sure
09:44that people spend more time
09:46on science and less on
09:48glucose.
09:49Now the situation is that
09:52a lot of the time
09:53spent around with cleaning data
09:55and developing, like, code that
09:57loops through trials, that organizes
09:59your results, that organizes things
10:01in pendant data frames or,
10:02like, any kind of, like,
10:03r data frames.
10:05And that's and that's a
10:06lot of time spent on
10:07not the actual problem, but
10:08just the tidying and the
10:09dusting of your results or
10:11simulation outputs.
10:13That's something that we want
10:14to, like, take off people's
10:17shoulder shoulders so they can
10:18actually spend time on the
10:19scientific problem as opposed to
10:21the technical one.
10:23And CPM is fairly new.
10:25Yeah. Here I have a
10:26bit of a timeline that
10:27first official release was in
10:29twenty twenty five July
10:31just before the CPC happened
10:33in Tubingen,
10:34the computational
10:35conference.
10:36And in September,
10:39a preprint follow followed, which
10:41the QR code, like, takes
10:43you to the OSF website
10:44where you can find the
10:45preprint.
10:46And
10:48by this time, the toolbox
10:49was
10:50available for free as in
10:52freedom.
10:53We shared it on the,
10:55on the Python library archive
10:56for PyPI
10:58so people can install it
10:59as easily as just typing
11:00some command in into the
11:02terminal or in the Python,
11:04Python terminal. You can just
11:06copy paste whatever's on the
11:07website. There are, like, a
11:08lot of instructions there.
11:11But the September was the
11:12latest release when we added
11:14a few prospect theory based
11:16models to the mix. So
11:17now we kinda, like, beyond
11:19the simple reinforcement learning problems.
11:22And
11:23the latest release has been
11:24downloaded
11:25as of January by about,
11:27like, one thousand people. Now
11:30whether they used it or
11:31not, at least they checked
11:32it out.
11:33And there is some traffic.
11:35And since then, there were,
11:35like, contributions outside of the
11:37lab where people fix bugs,
11:39people made feature implementations
11:41so that they, the toolbox
11:43runs better on their computer.
11:45So there are some traffic,
11:47generated just by, like, putting
11:49things
11:50out there for free available
11:51for people.
11:54Now do not worry about
11:55what's on the slide, to
11:56be honest. This is just
11:57demonstrates
11:58that
11:59the toolbox is not just
12:01that. It's not just that
12:02we shared it out there
12:03and it's available and you
12:04can write on your computer.
12:06The toolbox come with an
12:07ecosystem. There's a whole infrastructure
12:09in place for documentations,
12:11for tutorials, and this is
12:12two example of the documentation
12:14where we have extensive, like,
12:16information about where
12:18certain things come from, what
12:19are the theoretical backgrounds of
12:21the methods we implement. And
12:22especially for the models, we
12:24have equations, references,
12:26and
12:27explanations of the default settings
12:28or what are the edge
12:29cases that we cover. This
12:31is important because most people
12:33when they end up,
12:35writing their paper, they
12:37often forget what they have
12:38done with the modeling or
12:39what the equations were. And
12:41these documentations essentially
12:43help you to get access
12:45to these type of informations
12:46and also for us to
12:48be very transparent
12:49about what we are doing.
12:51And if you're a bit
12:53more, more of a nerd,
12:54if you have more, experience
12:56with these type of, like,
12:57models that are more sophisticated,
13:00this is very useful for
13:01you because you know what
13:03you're do
13:04doing. You know what type
13:06of model you are accessing
13:07when you interact with the
13:08toolbox.
13:10Beyond the documentations, we have,
13:11like, extensive tutorials.
13:13So I think the Wilson
13:14and Collins paper that was,
13:16like, really big. If you
13:16don't know it, don't worry
13:17about it,
13:19Had a bunch of recommendations
13:21about what is a good
13:22model evaluation technique and what
13:24you should do with your
13:25models before you conclude or
13:27put faith in your results.
13:29Now those things we try
13:31to cover in the tutorials
13:33with the toolbox. The toolbox
13:34has features to cover to,
13:36that is,
13:38in line with the standard
13:39recommendations
13:40made by the field.
13:43And this is kind of
13:44a, like, a balance between
13:45being a bit didactic, educational,
13:48and also letting you just
13:50to copy paste stuff into
13:51your own, like, workspace.
13:53We are leaning towards more
13:55of the educational side because
13:56we are aiming at nonexperts.
13:58And nonexperts,
14:00we should be able to
14:02have this information available even
14:04though that
14:05they will likely, like, copy
14:06paste things into their own
14:07workspace.
14:08And we have these tutorials
14:10written, checked in
14:11Jupyter notebooks. If you're not
14:13familiar with the term, it's
14:14essentially just a way
14:16so that you can run
14:18it on on the website
14:20and or to generate reports
14:21about these techniques.
14:24Alright. Now that I have
14:25a bit of an interaction
14:26of what the toolbox is,
14:27before I go into the
14:29actual workflows and the tutorial,
14:31there are some things that
14:32I
14:33want to say about boundaries
14:35that there are things we
14:37are not covering so we
14:38can stay relatively stay same
14:40and uninterrupted.
14:42So I'm not gonna talk
14:42about the math behind the
14:43models. You can all obviously
14:45ask me about these things.
14:46I'm not gonna talk about
14:47different optimizer we have. We
14:49can I can also tell
14:50you or you can just
14:51go to the website and
14:52check it? I'm not gonna
14:53talk too much about the
14:54architecture
14:55or the engineering of the
14:56toolbox. And I I'm in
14:58no way or shape is
15:00qualified to make clinical claims.
15:01I'm, I'm, developing toolbox here.
15:04So this is not a
15:05diagnostic tool. We it's it's
15:07it's a tool to help
15:08you get your modeling done.
15:10So I'm what I'm gonna
15:11focus on is instead how
15:13you interact with the toolbox,
15:15what it enables you to
15:16do, and
15:18talking about at a more
15:19high level about how to
15:20get your modeling done with
15:22the toolbox.
15:24So
15:26to start
15:27off with that,
15:28we
15:29have here very minimal
15:32very and very minimal and
15:33barebone, like, modeling workflow.
15:35On the left, you have
15:36your data that is ready
15:38to input into the toolbox.
15:40You have the model
15:42that you want to use.
15:44You have a likelihood, which
15:46is essentially just a way
15:47to compare the
15:49how different your model behavior
15:51is from the data.
15:53And then you have the,
15:54have the optimizer, which essentially
15:56estimates your parameters and find
15:58the, those parameters
16:01that
16:02have you,
16:03that help you minimize the
16:05difference between your data and
16:06the model you have.
16:09Now this is very minimal,
16:10and this is, of course,
16:11in more sophisticated approaches like
16:13parameter recovery or
16:15model recovery. This is embedded
16:18into the bigger workflow, but
16:19this is just to demonstrate
16:20what the toolbox covers here.
16:23Now when we started out,
16:25we
16:26focused on building and constructing
16:29models
16:30and making sure that you
16:31have the right tools to
16:32compare model outputs to your
16:34data
16:35and to make sure that
16:36you can actually estimate parameters
16:38on your data.
16:40This was, like, the very,
16:41very beginning, and this is
16:43where the toolbox is more
16:44most developed.
16:46But as time went by,
16:48we added a bunch of
16:49other features. So in order
16:51to cover cover, like, the
16:52model evaluation technique techniques that
16:54are kinda like the field
16:56is converging towards, we implemented
16:58features that cover generating artificial
17:01data to evaluate your models.
17:03We develop feature to have,
17:05like, hierarchical parameter estimation techniques.
17:08Again, if you're a bit
17:09unsure what hierarchical parameter estimation
17:11means, don't worry about it.
17:12We're gonna talk a bit
17:13about it later. It's just
17:15a very fancy term to
17:16say that you can estimate
17:19group
17:20level, like, latent variables in
17:23a very sophisticated and powerful
17:24ways, but,
17:26more on that later. And
17:28at this point in time,
17:30we are
17:31working a lot on making
17:32it easier for you to
17:34transition from your experimental data
17:36to a data format that
17:38is that's that's the tool
17:40that the toolbox prefers.
17:42And this, comes,
17:45because we also want to
17:47make sure that the toolbox
17:48can interact with,
17:50recruitment platforms. For example,
17:52we have the Brain Explorer
17:54app in the lab, which
17:55is essentially a mobile app
17:56where we collect a lot
17:57of data. Now those data
17:59come out in a certain
18:01format, not particularly like suited
18:03for modeling straight on. We
18:05want to bridge that gap.
18:06So our goal is that
18:07you get your experimental data,
18:09you input it in a
18:10function,
18:12define some variables, and then
18:13it outputs it in a
18:14format that is accessible by
18:16the built
18:18models. So yeah?
18:21Quick question.
18:24Does this,
18:27transition from data to model
18:28incorporates
18:29all these usual data cleaning
18:32steps that can be,
18:34very complicated?
18:36Or is it expected that
18:37people upload data that
18:39is already cleaned and reprocessed
18:42and you know? So that
18:43that part where I find
18:44there is a lot of
18:44difficulties in the field.
18:46Yeah. There are two things.
18:48So you, in a sense,
18:49you don't really upload data
18:50anywhere. It's, it's all happening
18:52on the computer, and I
18:53think I just, like, wanna
18:54make it clear so there
18:55are no misunderstanding.
18:57Because if you upload data
18:58anywhere, there are all sorts
18:59of privacy concerns that we
19:01don't want to deal with.
19:02So that's that. The second
19:04bit is that we don't
19:06filter data on that side.
19:09One reason is that there
19:11are so many
19:12factors that contribute to you
19:14excluding a participant, for example,
19:16or excluding a trial that
19:18it can't be determined prior
19:20to you looking at your
19:21data.
19:22So So you don't include
19:23some sort of flags,
19:25saying that be careful.
19:27You you're dealing with this
19:28and that.
19:29That would be very useful,
19:30I think. We do have
19:31that in, like, built in
19:33stuff. So especially in the
19:34parameter estimation part,
19:37we do have this information
19:38available that there is if
19:39there is some problem,
19:40we're gonna give you an
19:41error message. And if if
19:43something happened, you're gonna have
19:45it in the output
19:46documented for you.
19:48We do have, like Yeah.
19:49Because, of course, of course,
19:50the concern is that,
19:52is,
19:53you make this pipeline more
19:55user friendly,
19:57users
19:58might invest less
20:00time into thinking through the
20:01data, and then all kinds
20:03of
20:04results that come from just
20:06by distributional
20:07outliers might very quickly show
20:08up. Right?
20:10Yeah. That was one of
20:10my main concerns as well
20:12is that in order
20:13some people say that in
20:14order to really understand the
20:16model, you have to program
20:17it. But that, again, that
20:18is, like, kinda like a
20:20creates another problem. And making
20:21things more accessible,
20:23does reduce the time you
20:26spend on thinking things,
20:28thinking about things. We do
20:29have a bunch of error
20:30messages, and we
20:32continually increase the number of,
20:33like, FYI information that we,
20:35like, tell you. You know?
20:37In the metacognitive
20:38pipeline, we have a, a
20:40model, the meta d model.
20:42Some people might have heard
20:43about it. We do have
20:47ways to estimate, like, metacognitive
20:49performance in perceptual decision making
20:51tasks.
20:52And during those, we still
20:54require people to input, like,
20:56raw data
20:57and all the data aggregation
20:59or the calculating of mean
21:00or sorting data into things.
21:04We do check for all
21:06sorts of things, like whether
21:07there are empty bins, whether
21:09there are, like, bimodal distributions
21:11in the data and those
21:12type of stuff that we
21:13tell,
21:15people when they are running
21:16the,
21:17the optimization, when they're running
21:18these models.
21:22This is not complete, of
21:24course, because, well, there are
21:25a lot of error messages
21:26that you could potentially get.
21:28You yeah. You cannot, you
21:30know, you cannot make it
21:31algorithmic. Right? It's, a lot
21:33of things are very unpredictable.
21:34And Yes. Okay. And it's
21:36I don't know about
21:38you. I often find it
21:40the case that I don't
21:40know there's a problem until
21:42I come across one and
21:43start to do something that
21:44the models were not designed
21:46for.
21:46But then again, the
21:48the goal of the toolbox,
21:50especially to figure that one
21:51out. So this modular approach
21:52that we have allows you
21:54to stop at any point
21:55in your pipeline
21:57to make sure to investigate
21:58what is going on. So
21:59if you have a model,
22:00you can try to understand
22:01it before you just put
22:02it through, and we have
22:04features
22:05that lets you
22:07organize the model model output
22:08so that you can actually,
22:10plot and see what is
22:11going on, you know, behind
22:13the scenes under the hood.
22:16Oh, yeah. Alright. I'm gonna
22:17move on. Is there any
22:19other questions about these, error
22:20messages?
22:24I'm gonna just, like, get
22:26on then.
22:27So
22:28now I'm gonna, like,
22:30start with the walkthrough.
22:32So it it's gonna be
22:33a lot of hypothetical scenarios.
22:35So
22:36imagine that you're,
22:38designing an experiment with the
22:39two armed bended tasks. I'm
22:41going to explain how the
22:42task works in a second.
22:44You want to investigate, like,
22:45differences in learning between groups.
22:47You want to compare
22:49groups like neurotypical and neurotypical
22:51populations,
22:52for example, people with attention
22:54deficit
22:54hypertrophy disorder. And,
22:58experiment, for example, involves some
23:00manipulation, like distractors in the
23:02while you learn.
23:04And your goal is to
23:06estimate some latent variables like,
23:09learning rates that are captured
23:11by the parameters of the
23:12model.
23:14So what you start with
23:16is the task.
23:18In these two armed bandit
23:20tasks, you usually have two
23:21items appearing on screen, and
23:23you have to select one
23:24of them.
23:25You select one of them.
23:26You receive a feedback in
23:27the form of number of
23:29points that you got.
23:30And here, I selected the
23:32right one.
23:33Then I have plus hundred
23:35points.
23:37Now
23:38I move on to the
23:39next trial.
23:40Another two item appears. I
23:42get feedback. I get a
23:43certain amount of points, and
23:45I have to figure out
23:46which choice give me the
23:47most points.
23:48And as time goes by,
23:50there's an underlying structure in
23:52this task so that some
23:53items result in more points
23:55while other items result in
23:57less points.
23:58And that's the learning problem
24:00essentially. You have to learn
24:01which choice gives you the
24:02most points.
24:05Now
24:06after these experiments, you end
24:07up with your data, and
24:09that's where you
24:10start you can enter into
24:12the toolbox.
24:13Now this is a built
24:14in dataset in the toolbox
24:15that I'm see that I'm
24:16showing you right there, and
24:18don't worry about it too
24:19much the lines of code.
24:24The data is more important
24:25here because that, that tells
24:27us something important.
24:29So these built in datasets
24:31essentially just to help you
24:33start your modeling before the
24:35data actually collected. So for
24:36the case that you have
24:37to wait, that the collection
24:39is going on, so there
24:40is no restriction about when
24:42you can start your modeling
24:44in your project.
24:45So these datasets are just
24:46like examples, and these are
24:47the datasets that the built
24:49in models also take. Something
24:51that we talk about it
24:52later.
24:54Now the important bit here
24:56is that each row in
24:57the data
24:59represents one trial,
25:00and each row has to
25:02contain all the information that
25:04the model needs
25:05to calculate, like, probability of
25:07response or whatever you want
25:08to want it to calculate.
25:10Each row,
25:11contains both,
25:12the experimental
25:14variables and also what the
25:16feedback was or what the
25:17human responded in the on
25:19that given trial.
25:22And and this is how
25:23the data organized. This is
25:25a long format data. So
25:26if you're familiar with the
25:27term, it just means that
25:29it's,
25:31it's long as opposed to
25:33wide.
25:34So the data is organized
25:35as row by row and
25:37not column by column.
25:40Now if you have this
25:41data, this is how it
25:42will look, and this is
25:43what the new features will
25:45enable you to convert your
25:46data to.
25:49You have to start about
25:51what kind of models you
25:52want to use. You have
25:53to either build a model
25:54or you have or you
25:55can use one of the
25:56built in ones. Now if
25:57you choose the easy way,
25:59we have models that we
26:01prebuilt,
26:02models that we tested, that
26:04we made sure they work.
26:06All these models have some
26:07sort of, like, a backing
26:09in terms of, like, peer
26:10reviewed literature.
26:11And
26:12the number of models are
26:14always increasing. We're already working
26:15on to incorporate, like, the
26:17model based model free learning,
26:19free learning models
26:21just,
26:22just as as a sort
26:23of archival process where you
26:25can just, like, pick one
26:26out,
26:27as in, like, an application
26:29that you send your data
26:30to.
26:32And here,
26:34you can just
26:36have to define certain parameters
26:38for the model,
26:40not the free parameters.
26:42What I mean is that
26:43we try to make sure
26:44that models are scalable.
26:45Not all of them are
26:46scalable at the moment, but
26:48we're working hard on being
26:49scalable. So the model that,
26:51you see here, the r
26:53l r v RW,
26:55is
26:57able to take in any
26:58number of stimuli. It's not
26:59just a two arm benefit
27:01problem. It can take in,
27:03like, four, six, seven, eight,
27:04nine, ten
27:05bandwidth at a time. So
27:06this,
27:08the model is scalable
27:10to new experiments, to more
27:11complex experiment, and that is
27:12important because that helps kind
27:14of, like, the comparison process
27:15to other models as well.
27:17And you don't have to
27:18rewrite your code all the
27:19time to apply it to
27:20your new experiment if you
27:22change something,
27:23and that's something that we
27:24also try to facilitate.
27:28If you decide to write
27:30your model for scram from
27:31scratch because you are very
27:32advantageous,
27:33adventurous,
27:35Then the toolbox
27:37helps you by,
27:39having these two steps. So
27:41the modeling is broken down
27:42into two two steps.
27:44One is handled by what
27:45I call the parameter management
27:47system, which is a very
27:47fancy way of saying that
27:49it helps you to store
27:51and to define your parameters,
27:53like learning rates, so how
27:54fast people learn, how much
27:56new information they retain, or,
27:58like, the decision rules so
28:00that how decisive people are,
28:02which is the temperature down
28:04there and offers learning rate.
28:06So
28:07when you're doing this, you
28:08have to define a bunch
28:09of things that you might
28:11not you might not know.
28:14One thing that we really
28:16enforce is the lower and
28:17upper bounds of parameters.
28:19It's often the case that
28:21people do not
28:22explicitly state these things in
28:24their paper or even in
28:25their codes. It requires a
28:26lot of digging to figure
28:27it out. And that is
28:29a problem because it has
28:30been shown that in a
28:31lot of cases,
28:33these bounds matter because these
28:34bounds determine,
28:36when the model behavior is,
28:37like, consistent or, like, sensical.
28:40And with some parameter combination,
28:42the model becomes, like, nonsense
28:44nonsensical and, you know, gets
28:45out of hand. And we
28:47have to know,
28:48whether the model succeeds
28:50within these, like, bounds where
28:52it makes a reasonable prediction,
28:53where the intermediate variables it
28:55calculates are reasonable.
28:57And this is, essentially just
28:59a practice that we're trying
29:00to enforce here.
29:01Similar with the priors. Now
29:03if you don't know what
29:04priors are, they're essentially just
29:06another constraints on the parameters.
29:09And those are something that
29:10we also expect people to,
29:12to select to explicitly say
29:15what the priors and the
29:15parameters are. Again, to enforce
29:17some standard practices that, we
29:19would like people to have
29:21in the field.
29:23These are all,
29:24like, accompanied by support from
29:26the documentation. So if you
29:27don't know where to begin,
29:28you can just, like, go
29:29to the website and you
29:31can,
29:32you can find the
29:34right,
29:35the right answer for you.
29:38Now this parameter management system
29:41takes care of other things
29:42than the parameters as well.
29:44So in the learning task
29:45that we just had before,
29:47it is safe to assume
29:48that people don't have prior
29:50beliefs how those cartoon images
29:53result in rewards.
29:55That means that each subjective
29:57value
29:58associated
29:58with those items
30:00will have a starting value
30:02of zero, which means that
30:04we don't know what those,
30:07what the rewardingness of those
30:09items are. And that's something
30:11that the toolbox also, like,
30:13saves by
30:14defining it in the parameter
30:15management system.
30:17And all these things, the
30:18extra things that that you
30:20define here will be,
30:21kept track of. So if
30:23you want something to update
30:24incrementally,
30:26like on every trial, for
30:27example, you can define it
30:28in this parameter management system,
30:30and then the toolbox will
30:32update it for you. It
30:33will organize it for you,
30:35and it will output it
30:36for you with a reasonable
30:38names and in reasonable date
30:39format. So this is, again,
30:41in order to reduce the
30:43number of
30:44all lines of codes that
30:45you have to write to
30:46keep track of things, to
30:48keep your data tidy is
30:49reduced because this is something
30:50that
30:51we are taking care of,
30:52the technical bits.
30:55Now after if you have
30:56your parameters already done, you
30:57can move on to the
30:59model execution framework, which is
31:01essentially just like a way
31:02to build your models.
31:04In this step of model
31:05building, the only thing that
31:07you have to do is
31:07to write a function
31:09that tells the model what
31:11to do on every trial.
31:13So the only thing that
31:14you have to be concerned
31:15about is what happens on
31:16a single trial.
31:17So if it's a learning
31:18model, then, of course,
31:20it it makes a response.
31:21It updates
31:23its belief about how rewarding
31:25each item is, and that's
31:26the only thing that you
31:27have to be concerned about.
31:29You don't have to worry
31:30about how you, how you
31:32save the results after each
31:33trial, how you organize it,
31:35how you apply it to
31:36your data row by row.
31:38That is taken care of
31:39you, by the toolbox. Again,
31:41the technical bits are on
31:43us, and you can focus
31:44on what the computations are,
31:46what you want the model
31:47to do.
31:49Now one solution here is
31:51to write things from scratch.
31:52So in this function between,
31:55the input and between what
31:56the function outputs, you have
31:58you can write things from
31:59scratch, and you can look
32:00at equation. You can
32:02write it out, like, to
32:03match exactly what happens on
32:05the on the paper that
32:07you just like using.
32:09But we do have have
32:11for you. So we have
32:14these things that I call,
32:16more computational LEGO blocks
32:18that we have a bunch
32:19of, like, decision rules and
32:21learning rules that we already
32:22implemented so you can just
32:24stick it into your model
32:25and get running.
32:26And these are designed
32:28so that they
32:30handle edge cases and they
32:31handle certain, like, out of
32:33bounds parameters as well,
32:35and tell you about these
32:37things.
32:38Now the goal of this
32:39is that there are a
32:41bunch of equations out there
32:42that you have to stitch
32:43together in order to make
32:44your model. Now those
32:46equations are often reused. So
32:49softmax is always the same.
32:51It's not gonna change when
32:52you're using it in a
32:53new paper or in a
32:54new project. So we implemented
32:56it so that you can
32:57just, like, stick it in.
32:58It
32:59is, again, scalable to any
33:01number of, like, choice options.
33:02So you can use it
33:04in all,
33:05in all different experiments that
33:07you're running. And that is
33:08helpful because then you do
33:10not have to reinvent the
33:11wheel again. You don't have
33:12to, like, rewrite your code.
33:14You can use the same
33:15thing that worked before in
33:17your own, like,
33:18project.
33:21Now
33:22this
33:23model building
33:24step essentially ends with you
33:26defining all the things that
33:27you want to save in
33:28the model output.
33:30Now here,
33:32the output contains a bunch
33:33of things that I'm interested
33:34in, and those are all
33:36saved and organized for you.
33:38You don't have to bind
33:39them together into one big
33:41data set. This is something
33:42that we take care of.
33:44The only one thing that
33:46you must define
33:49is the dependent variable,
33:51what variable you are interested
33:52in. So if you're running
33:53a learning task,
33:54you're interested in people's choices.
33:57So the model has to
33:58make predictions about what those
34:00choices are, and you have
34:01to tell us what those,
34:05what those choices correspond
34:06to in the model. Here,
34:08these are, like, the choice
34:09probabilities, the output by this
34:11decision rule that I mentioned,
34:12the softmax.
34:14And
34:15that is shown on the
34:17last line on that output.
34:20But, essentially, these these are
34:21the steps when you want
34:23to create a model from
34:24scratch. And after this, you
34:26essentially have a model.
34:27You input it into
34:30a a function we call
34:31the wrapper that kinda, like,
34:32pulls your parameters and the
34:33model function together and applies
34:35it to data.
34:36And that
34:37wrapper is essentially,
34:39the one that organizes everything
34:41for you.
34:42And after you applied it
34:43to data, you get an
34:44output, something similar to what
34:46you see on the bottom
34:46where you have a bunch
34:48of intermediate variable.
34:50Maybe the most salient ones
34:51are the errors
34:53that you see on the
34:54more right side
34:55is that the prediction errors
34:57on any given trial. These
34:58are, again, organized for you,
35:00saved. You don't have to
35:02worry about, like, combining them
35:03together.
35:05All the things that you
35:05define in the model output
35:07are, like can be exported
35:08as a CSV file so
35:10that you can save it,
35:11you can plot it, you
35:11can investigate it, and do
35:14fancy graphs like this.
35:15Now
35:17because things are organized for
35:18you, you can spend time
35:20on understanding
35:21how the model works. For
35:22example, we have the RLRV,
35:25model here.
35:27And these are just the
35:28what you see on the
35:29y axis,
35:30the belief about how rewarding
35:32certain items are. They are
35:33q values if you're familiar
35:34with the term.
35:36Here, I just picked three
35:37random, like, learning parameters. Like,
35:39one is, like, slower slower
35:42learning that is alpha zero
35:43point twenty five, and it
35:44reaches, like, quite high because
35:46in these models,
35:48the maximum learning rates can
35:50take is one.
35:51That one is zero point
35:52eight. And you can you
35:53can try to understand what
35:54is happening here. You can
35:55see that more stable, like,
35:57learning takes place when the
35:58learning rate is lower and
36:00when the learning rate is
36:01higher.
36:02The, the model is very
36:03sensitive to feedback so that
36:05beliefs are kinda, like, jiggity
36:07and drop very strongly when
36:09there is, like, a a
36:10change in what you think
36:12will happen and what you
36:13received.
36:14And these are and and
36:16this is facilitated by this,
36:18like, data organization that we're
36:19doing in the background.
36:22Of course, at some point,
36:24when you will have this
36:25model, you don't just wanna
36:26play around with it and
36:27plot, like, fancy graphs like
36:29this, which is part of
36:30the tutorial.
36:32We want to
36:33optimize a little bit and
36:34estimate some parameters. We are
36:36interested in, you know,
36:38how these,
36:39latent variables,
36:41are captured by these parameters
36:43and how they relate to
36:44what we're interested in, like
36:45the learning differences.
36:47Now the toolbox has this
36:50optimization frame framework built in,
36:52and one disclaimer here is
36:53that we do not want
36:54to reinvent the wheel.
36:56So we don't want
36:58to reproduce this code fragmentation
37:00that we talked about. So
37:02instead of, like, writing things
37:03from scratch,
37:05we are pulling in is,
37:06like, methods and optimization procedures
37:09from established libraries like the
37:11sci fi, one of the,
37:12like, biggest optimization library out
37:14there
37:14so that
37:16we do not, like, introduce,
37:18like, unwanted changes and inconsistencies
37:20into the code that into
37:22methods that are already working.
37:25So instead, we're making sure
37:26that they are optimized.
37:29Those methods from scipy, for
37:30example, optimized
37:32for computational psychiatry.
37:34For example, things like we
37:35want to estimate, like, learning
37:37rates for each subject as
37:38opposed to on the group
37:39level
37:40so that we make sure
37:41that the opt that the
37:43these
37:44these methods, these optimization functions
37:46imported from other libraries,
37:49like, do it. So they
37:50they are doing fast. They
37:51are organizing things for you
37:53in that fashion with a
37:54single goal of estimating subject
37:56level parameters.
37:57And all the performance improvements
37:59that you might want are
38:01also implemented here with that,
38:03like, caveat in mind. So
38:05on if you're looking through
38:06the code, there's a parallel
38:08variable there that I just
38:09set to true,
38:10which means that the
38:13the parameter estimation is distributed
38:15across like CPU core so
38:17that instead of, like, one
38:19parameter estimation going on on
38:20your computer, there are, like,
38:22estimate parameter estimations going on
38:24simultaneously
38:25as your computer can handle.
38:27And that is something that
38:28we are also taking care
38:29of because we don't want
38:30people to
38:31to get bogged down how
38:33on how to parallelize things
38:35in Python because it's not
38:36straightforward. And I'm, like, programming
38:38for a living. It's not
38:39straightforward for me. So I
38:41don't expect it to be
38:42straightforward for people who just
38:43like starting out or people
38:44who are non experts.
38:46And these and we and
38:48that is something that we're
38:48also making sure that it
38:50works across different operating systems
38:52than in different environments, and
38:53we have tutorials on how
38:54to get it to work
38:55in different environments.
38:56But, again, those are, like,
38:58the technical details.
38:59If your simulation is running
39:01too, too slow, if your
39:03optimization is running too slow,
39:05you can just flip a
39:06switch inside the toolbox and
39:07make it run faster. And
39:09that's something that we're also
39:10trying to optimize where I'm
39:11taking take care of.
39:14Essentially, so you have these
39:15methods here just stuck in
39:16your model. You're stuck in
39:17your data. You're, stuck in
39:19your,
39:20log likelihood, which is, again,
39:22the goodness of fit measure
39:23of how close the model
39:24and the human
39:26behavior gets. And then everything
39:27is kind of pulled together
39:28inside.
39:29You don't have to write
39:31very low level code about,
39:32like, calculating,
39:33these goodness of fit metrics.
39:35This is something that we
39:36have implemented, and you can
39:37just, like,
39:38pick the right one suitable
39:40for you. We are taking
39:41care of all the edge
39:42cases, all the out of
39:43bounds parameters, and all the
39:45out of bounds values and
39:46infinities there that you don't
39:48want to be concerned about
39:49while you're trying to
39:51start the
39:52start your project. Right?
39:54So at the end of
39:55the day, you run your
39:56optimization.
39:57And, again,
39:59because the toolbox is organized,
40:00you can, like, easily investigate
40:02how your parameters are distributed,
40:04how good your model done,
40:05and your goodness of fit
40:06metrics. And you can do
40:08these things because we organize
40:10the data for you. So
40:11the tiding and the dusting
40:13is, again, taken care of
40:14by us, and you can
40:15be more concerned about what
40:16kind of optimizer you're using,
40:18what
40:19what do you what are
40:20you actually interested in? Are
40:21you interested in point estimates,
40:22like, or are you interested
40:24in posterior predictive, like, distributions,
40:26all the fancy words that
40:27you might come across? And
40:29you can be more concerned
40:31about the scientific problem as
40:33opposed to the technical one
40:34on how to get it
40:35working.
40:38So
40:39the optimization doesn't stop here.
40:41One of the biggest things
40:42out there is model comparison.
40:43And
40:45right now, we are working
40:47on making it more fluid
40:48and more integrated.
40:49But we already have all
40:51the model comparison metrics that
40:52you might want to use,
40:53like Bayesian information criterion, the
40:55information
40:56criterion,
40:57and we're getting and we're,
40:58of course, increasing that list,
41:00minimum description lengths or base
41:02factors and all the other,
41:03like, standard things that you
41:04might want to use.
41:08We also have the hierarchical
41:10modeling that I mentioned before.
41:11So for example, if you're
41:12interested in, like, group level
41:14estimates, which you might as
41:15well be
41:16because you want to get
41:17a sense of the distribution
41:19of learning rates or the
41:20distribution of certain, like,
41:22latent variables. And
41:25that allow and these hierarchical
41:26estimation methods essentially allow you
41:29to to estimate those things,
41:31to get a sense of
41:32how does the group, how
41:34does the group that I'm
41:35investigating is doing.
41:37And
41:38the
41:39these are all, again,
41:41implanted in toolbox. There are
41:42multiple, like, ways to do
41:44it.
41:45The way to get it
41:46done is to just take
41:47the optimization that you wrote
41:49now, the those,
41:50some of
41:51those codes and just stick
41:53it into this method. So
41:55or you already created the
41:56Python objects. You can stick
41:57it into these, like, hierarchical
41:58estimation methods,
41:59make it run, and then
42:01it just organizes all the
42:02data and things for you.
42:04And
42:05we also have, like, methods
42:07to check how well this,
42:08like, optimization worked. Those are,
42:10part of the documentations, and
42:12we have features to cover
42:13that.
42:15But the point here is
42:16that, again, this plug and
42:17play stuff that if you
42:18got to a certain point
42:20in your modeling work, like,
42:21you already have, like, the
42:22optimization done and you know
42:24how you want to estimate,
42:25like, subject of the parameters,
42:27you don't have to start
42:28all to write some,
42:31some new, like, thousand lines
42:32of code
42:34and try to figure out
42:35how to estimate these, like,
42:36group level priors or group
42:38level parameters.
42:39There are methods there, and
42:40you can just stick it
42:41in,
42:42what you wrote before and
42:43get on with your life.
42:46And you can do fancy
42:47graphs like this that I
42:48don't think I can explain,
42:50but they are very nice.
42:52And,
42:53it's been a while. But,
42:54essentially,
42:56the the tutorials cover how
42:58to make these graphs if
42:59you're interested in.
43:00If if you spend a
43:01bit of time, there are
43:02ways you can, like, investigate
43:04how well your, for example,
43:07and hierarchical modeling
43:09improves your model estimates, your
43:11parameter estimates on a group
43:12level and on an individual
43:13level.
43:14And the tutorials
43:15include ways,
43:17on how to compare it,
43:18how to investigate it. These
43:19fancy graphs are probably, like,
43:21coming from those tutorials.
43:25And, of course,
43:26I don't wanna spend too
43:27much time here because I
43:28am getting to the end.
43:30We have a bunch of,
43:31like,
43:32features that cover things that
43:34you need for, stuff like
43:35parameter recovery and model recovery,
43:37and we have tutorials that
43:39correspond to it. And that
43:40is an
43:41unprecedented
43:42like, that is way too
43:44much recovery there. Those correlations
43:46are way too good. If
43:47you are doing computation modeling,
43:48this is not how you
43:49encounter them at first. So
43:51I really hit the nail
43:52on the head here. It's
43:53like I'm I'm a bit
43:54proud.
43:55But, essentially, the we have
43:57ways to generate data from
43:59modules. We have way tutorials
44:01to help you get these
44:02things done.
44:04So in order to just
44:05wrap up, I'm gonna emphasize
44:06some things here is that
44:08as you can see, there
44:09is a lot of,
44:10entry points for the toolbox.
44:13So if you're a novice,
44:14you can go you can
44:15start very easily. If you're
44:17more experts,
44:18you can,
44:19you you can hack your
44:20way through things.
44:22Again, if you're interested in
44:23modeling,
44:24you can you don't have
44:25to worry about the
44:27optimization frameworks. You can spend
44:29more time on model building.
44:32And this, like, modularity
44:35reduces the number of, lines
44:37of codes that you have
44:38to write and because we
44:39take care, we we we
44:41take care of all the
44:42bits and bobs for you.
44:45It is supported by built
44:46in datasets and tutorials to
44:48lower the barrier to doing
44:49it right.
44:50Now the only one one
44:52main thing that I wanna
44:52mention before I wrap up
44:54before I end the presentation
44:55is
44:56that reproducible science needs reproducible
44:58code. So we have huge
45:00amount of test units that
45:01essentially
45:02are there to make sure
45:03that any changes to the
45:05toolbox or any contribution that
45:06other people make to those
45:07toolbox
45:08will not break your code,
45:10will not break your simulations
45:12and the parameter estimations that
45:14you've done one year ago.
45:15So
45:16there are some things that
45:18I listed there that we
45:19are doing.
45:20The point here is that
45:21this is something that we're
45:22actively developing and making sure
45:24that,
45:25like, kinda like,
45:28that using the toolbox is
45:29safe. It's not something that's
45:31going to break with the
45:32next release.
45:34Something that I had some
45:35good experience with with other
45:36tools.
45:37And this is really good
45:39for us because it tests
45:40the toolbox across different operating
45:42systems and different Python versions.
45:44So if you change your
45:45laptops, things will still run.
45:46And this is a huge
45:47payoff because it makes us
45:49very easy to to allow
45:52contributions from others because we've
45:54checked for these things. We
45:56can we want people to
45:57develop extensions
45:59so that we can make
46:00but we want to make
46:01sure that the toolbox remains
46:03reliable.
46:05And,
46:07yeah, I think I'm gonna
46:08skip that because I'm kinda
46:09close to the end now.
46:11But, yeah, there's a lot
46:12of planned direction. The one
46:14thing that I want to
46:14emphasize is that we are
46:16really working on making this
46:18transition from,
46:20experimental data to,
46:22to the toolbox easier for
46:24you. And it comes with
46:25this intent of making the
46:27ecosystem a bit larger to
46:29include, like, things like the
46:31brain explorer, which is like
46:32data collection, like platforms.
46:34So that is it. The
46:36QR code down there takes
46:37you to the GitHub page
46:38where you have instruction of
46:39how to install stuff and
46:41how to debug things. That's
46:43the place where I also
46:45ask you to report any
46:47kind of issues or bugs.
46:48And I know that people
46:49are kind of shy about
46:50putting it out there because
46:51they are afraid that it
46:52is just their laptop that's
46:53making the problem. We also
46:55wanna know if your laptop
46:56is the one making the
46:57problem because it's informative for
46:59us. So if you if
47:00you're a bit shy, then
47:01you can also send an
47:02email out there.
47:04And
47:05that is it. Thank you
47:06very much for your kind
47:08attention. The my website is
47:09on the left QR code.
47:10The preprint is on the
47:11orange one. And
47:13thank you
47:14all the people from my
47:16lab. I think only a
47:17few of them are here
47:18probably. So, yeah,
47:20that is all. Thank you
47:21very much.
47:22Great. Thank you. Okay.
47:27Amazing. Thank you so much,
47:28Leonard,
47:29for this really, yeah, great
47:32detailed walk through and explanation.
47:35Are there any questions?
47:38I think people should just
47:39unmute themselves. Yeah. Yeah. I
47:40have a I have a
47:41question.
47:43And this was a was
47:44a really great talk, and
47:45I think you've built an
47:46incredibly
47:47powerful
47:48platform.
47:51My question kinda comes to
47:52those
47:53who use
47:54who, like, use choice rules
47:55such as sequential sampling models.
47:57And so how flexible and
47:59how much you've thought about
48:00integrating
48:01those types of models into,
48:03your platform because, you know,
48:05they're notoriously hard to fit
48:06with conventional
48:08optimization tools, and they require
48:10kind of specialized approaches sometimes.
48:12So I just kinda wanted
48:13to get your thoughts on,
48:14like, how how you might
48:15integrate those types of models.
48:17Yeah.
48:18That is really great. Thank
48:19you very much for asking,
48:21asking that question. So two
48:22things there. First one is
48:24that
48:25you define your model how
48:27it works. If you're more
48:28advanced and,
48:29you can probably hack your
48:31way through the toolbox.
48:33But because it's modular, there's
48:34a lot of entry point.
48:35You can build the model
48:36yourself, and you can develop
48:38these sequential,
48:40sequential, like, sampling models and
48:42the model construction phase and
48:44then stop before the optimizer.
48:47And you and we have
48:48a feature
48:49that exports
48:51your the model that you
48:52just wrote
48:54so that it can be
48:55used by third parties, like
48:57other library optimizers because they
48:59might be better fitted to
49:00to your solution. So we're
49:02trying to play nice with
49:04other tools as well.
49:06And,
49:06you know, most of the
49:07models that
49:09are coming our ways, we
49:10we just, like, encounter them
49:11on the go. We we
49:13planned a lot for our
49:14own use cases, but, of
49:15course, they are limited because
49:16we are one lab. Right?
49:18So
49:19there are ways you can,
49:20like, build these sequential models.
49:22And
49:23whether it will work with
49:25the optimizers that we have,
49:27I'm not exactly sure about
49:28that. We have, like,
49:30something that
49:31might fit, like, Bayesian
49:33directed adaptive sampling. This is
49:35a lot of fancy words.
49:37But
49:39with in terms of the
49:40model construction
49:41or building your models up
49:43until the likelihood,
49:44it's it's very easy. So
49:45you can definitely do it
49:47there. In terms of the
49:48optimization, you can try it
49:49with our tools, or you
49:50can export it somewhere else
49:52as well. If if you're
49:54interested in, like, write trying
49:55to write one inside the
49:57toolbox,
49:58feel free to do so.
49:58I'm I'm happy to debug
50:00as well and and have
50:01to Yeah. Get it inside
50:02the toolbox because this is
50:03something that is kinda outside
50:05of what we're usually doing,
50:07but something that is very
50:08interesting in terms of the
50:09research. Right? So would be
50:11nice to cover something like
50:12that. Don't know. Does it,
50:13like, clear up how you
50:14can, like, get these things
50:16done?
50:17Yeah. So if I understand
50:18you correctly, you can you
50:19can build the model with
50:21with your toolbox. But perhaps
50:23if you need a different
50:24optimizer, you can call in
50:26the optimizer from other packages
50:28Yes. Exactly. Correctly.
50:30Yeah. Yeah. I mean, that's
50:31it's tough because,
50:35for, like, a DDM, for
50:36example, that's, like, kind of
50:37the test case I'm thinking
50:39of is the Mhmm. There
50:40are a lot of different
50:40approaches to try and fitting
50:41these models, but they're notoriously
50:43kinda tricky.
50:45And it's not clear to
50:46me that there's, like, a
50:47specific optimizer that's worked. It's
50:48like people have written specialized
50:50code for each package that
50:52allows it to be fit.
50:54Yeah.
50:55That's a good point because
50:56the the DDM has probably
50:58the most coverage in terms
50:59of toolboxes out there. Like,
51:01there
51:01that I can think of,
51:02like, eight just from the
51:03top of my head right
51:04now that does DDM and
51:06does inference.
51:07So this is something that
51:09that we are that is
51:10not in our focus probably
51:11because of that that this
51:12has so much coverage.
51:14I have So you're you're
51:15more looking at, the models
51:16that, like, kind of are
51:18have been neglected by the
51:19field.
51:20Well, and
51:21my intuition
51:23is that we're looking at
51:24anything
51:25that can make trial level
51:26predictions. We're interested in learning,
51:28decision makings, and,
51:31any model that kind of
51:32takes in, information trial by
51:34trial, and that's where most
51:36of our use cases are.
51:38Now I do I did
51:39have some student who wanted
51:41to make,
51:43reinforcement
51:44learning DDM combination there, which
51:46has been done before.
51:48I haven't heard from him
51:49for a while, so I
51:50assume it was beyond what
51:52he was, like, hoping,
51:53what the toolbox at the
51:55time could cover.
51:56But this is something that
51:57we can also, like, explore
51:58because people if people are
52:00interested, it can we can
52:01make it happen. You know?
52:03Okay. Yeah. Thank you.
52:08So I'm sorry. Do do
52:09I understand correctly that, the
52:11the idea of the model
52:12of your package is, somewhat
52:14similar,
52:15to our plus
52:17where you invite,
52:19free open contributors
52:20and offer
52:22support
52:23in terms of troubleshooting and
52:24integrating. Is it is it
52:26the model? Exactly. So the
52:28thing is that, this is
52:29open source,
52:30and this is free as
52:31in freedom in a sense
52:33that users do have a
52:34lot of control over how
52:35the toolbox works and what
52:37is what are the features
52:38that are implemented.
52:39We already had some compute
52:41contributions
52:41from, from external parties, and,
52:44you know, they made changes
52:45to a toolbox. That's why
52:46we have the test units
52:48so that people can freely
52:50make,
52:51suggestions and changes without breaking
52:53everything.
52:54And
52:55and I had a lot
52:56of good experience with, working
52:59with external contributions
53:01in other packages. I had
53:02some r packages, so I'm
53:04definitely trying to, like,
53:06transpose that philosophy
53:08into into this Python library.
53:11In terms of troubleshooting,
53:14yes.
53:15I'm currently, like, involved in
53:17all the troubleshooting attempts,
53:19but we do have platform
53:21on GitHub where people can
53:22actually,
53:24solve it within each other.
53:25We have,
53:27we have this, discussion platform
53:29where people can pose their
53:30problem, where they can look
53:31for solutions as well.
53:33And please do,
53:35go there and share things
53:36because I'm interested in the
53:37people's experience, but also want
53:39to generate some traffic so
53:40that if if you encountered
53:42an error or or encountered
53:44the problem, it's very likely
53:45someone else did as well.
53:51Related to that, so I
53:52know right now you're the
53:53main support when it comes
53:55to the toolbox.
53:56Is there any plan, especially
53:58what's it,
54:00as a hopeful scale
54:03to have a more, yes,
54:04dramatically, like, support system in
54:06place? Or will you see
54:08how it goes and then
54:09react to the demand?
54:12So there are two things
54:13for that. I think you're
54:14asking whether Tobias is going
54:16to hire more people.
54:18He probably will at some
54:19point.
54:20So
54:21that's, so my PI will
54:23probably, like, involve, like, more
54:24people in the project as
54:25well. Currently, it's just me.
54:28There are no immediate plans
54:30of extending that,
54:32that team.
54:35But,
54:36it raises an interesting questions
54:38about how reliable the toolbox
54:40maintenance is or whether if
54:42you're, like, two years from
54:43now, you reach out because
54:44you have an issue. Real,
54:45some don't fix it.
54:47Yeah. Now the,
54:49the oldest code that I
54:50contributed to the toolbox was
54:52in twenty
54:54sixteen.
54:55It was,
54:56a model called Sustain, which
54:57is a categorization model. It's
54:59a network model,
55:01that, implements a prototype
55:04theory of categorization.
55:06It's it's a lot of
55:06fancy words. Again, just remember
55:08the model I sustain.
55:12I got an email about
55:13it a year ago that
55:14they want
55:16to get something done and
55:17want, want to make it
55:19work for their own use
55:20case, and that led to
55:21me discovering a bug in
55:22the code and making a
55:24fix and updating it,
55:27and pushing the change to
55:29the r package. And, you
55:30know,
55:32it's,
55:33I can make the commitment
55:34that I will at least
55:35look at problems,
55:37years from now. Whether there
55:39will be a bigger support
55:41team support team, I can't
55:42answer that.
55:44That makes sense. Thank you.
55:46I love it. I love
55:47it. Really impressive.
55:50Oh, yeah. There's a lot
55:51of lot of sources of
55:52audio.
55:54Alright. Let me mute this.
55:55Can you still hear me?
55:57Yes. I hear you.
55:58Perfectly. Yeah. Very impressive work.
56:01I guess my question is
56:02related
56:03to Alyssa's, but I'm gonna
56:05take it to the next
56:05level, which is,
56:07you know, I have I
56:09have seen,
56:11these two boxes
56:13in different fields, right, being
56:15developed. Yeah. And some had
56:17really massive impact on the
56:18field,
56:20like SPM.
56:22Some are
56:23kind of you know, some
56:25just, like, gradually died off,
56:27but also there are some
56:28that are more used in,
56:29like, a very bespoke way.
56:31Right? It's mostly just shared
56:33with a few maybe collaborators.
56:36I guess
56:38I wonder I don't know
56:39if Tobias is here, but,
56:40you know, like, maybe you
56:41can speak on his behalf.
56:43What are you guys thinking
56:44in terms of,
56:46I guess, the type of,
56:48yeah, the type of toolbox
56:49you wanna you wanna make.
56:50Right?
56:51That are you are you
56:52envisioning something that's more like
56:54SPM? Because that would be
56:56very resource heavy
56:58and involves,
56:59like, basically,
57:00a yeah. Like, more than
57:01just you or or you
57:03two. Right? But, like, a
57:05team of people with actually
57:07sustainable funding also
57:09and, you know, host supporting
57:11also workshops and all these
57:12things. Right? So that that's
57:14one that then the other
57:15one would be maybe more
57:17like,
57:18EdgeGF,
57:19you know, that, Mhmm. And
57:20I think it's mostly used
57:22in a a selected group
57:24of labs,
57:25mostly through, like, individual collaborations.
57:30Like, what are you, yeah,
57:31what are you thinking about
57:32at this moment?
57:34That's a that's a very
57:35philosophical and reflective question. Thank
57:37you very much.
57:40It I mean, I can't
57:41predict the future. What we
57:42want to do here is
57:43to have some flexibility in
57:45the type of models that
57:46you can use without
57:47calling in thousands of libraries.
57:50Of course, the toolbox started
57:51out by focusing on what
57:53our lab needs, what are
57:54the needs here, and how
57:55we can make sure that
57:56people can do their modeling
57:58bit. And then it grew
58:00into a bigger thing. Right?
58:01Because now there are external
58:03collaborators, and there are people
58:04who apply the toolbox outside
58:06of the lab as well.
58:07And,
58:08well,
58:09the
58:10the big thing is that
58:12well, I think that
58:13the interesting thing is that,
58:16lot of times you get
58:18this argument that if it's
58:19open source, then it's maintained
58:21and used as long as
58:22one person cares.
58:23Now I find that a
58:24bit weak argument. So even
58:26if the toolbox doesn't reach,
58:28like, SPM
58:29height
58:30or the, like,
58:32size, it can still be
58:33used and maintained regardless.
58:36I don't
58:37I
58:38I'm more focused on making
58:40sure that, you know, the
58:41toolbox that we have and
58:42the one that we're developing,
58:44focuses on the right thing,
58:45which I think is the
58:47human computer interaction bit is
58:49that how people interact with
58:50these tools as opposed to,
58:51like, trying to make it
58:53into a big toolbox that
58:54will define the the field.
58:57I think the important bit
58:58is, you know, making sure
59:00that what we have works,
59:01and hopefully people will benefit
59:03from it.
59:04But yeah. I
59:05so maybe I just avoided
59:07your question totally. I'm not
59:10sure. But our focus is,
59:13is, of course, on making
59:14sure the toolbox works and
59:16it, and people can use
59:17the toolbox,
59:18and we see what happens
59:20later on. Yeah. No. No.
59:21No. That's a good answer.
59:22I I just wanna I
59:23feel like, you know, I
59:24just wanna sort of challenge
59:26you to start to think
59:27about these things because,
59:29you know, it's clearly a
59:30lot of effort that you
59:31personally have put in into
59:33this product. Right? And you
59:35want the fields the community
59:37to use it. And I,
59:38a hundred percent agree with
59:40all the issues you raised
59:41early in the opening
59:43because those are the things
59:44that, you know, that is
59:46blocking the field, right, in
59:48some in some aspects.
59:50So,
59:51I think we all collectively
59:53feel that if there's a
59:55very robust tool
59:57that people can use with
59:59also, you know, good,
01:00:01ECG use.
01:00:03And, again, I feel like
01:00:05with with some resources and
01:00:06funding, right, and, you know,
01:00:08how collectively look for those
01:00:09things
01:00:10as a community
01:00:12that can support you and
01:00:13the team maybe to work
01:00:15on this, that would be
01:00:16really beneficial for the field.
01:00:18Yeah. It would be cool.
01:00:19I mean, the thing sorry
01:00:20to interrupt.
01:00:21Finish finish finish what you
01:00:22were saying. No. No. No.
01:00:23I was just the other
01:00:24thing is something that Blair
01:00:25raised, which is I think
01:00:27the other struggle we have
01:00:28is, you know, for example,
01:00:29XGF. Right? It's a very
01:00:31particular type of model, right,
01:00:33which I think is limited.
01:00:34It's used in some degree.
01:00:36But, but this one is
01:00:37much more flexible in terms
01:00:39of the model that people
01:00:40can actually develop.
01:00:43So I think, like, thinking
01:00:44about,
01:00:45you know, DDM
01:00:46or even alternative models
01:00:49that
01:00:50would tackle another issue, which
01:00:51is the lack of diversity
01:00:52in our models right now.
01:00:55Yeah. The so my my
01:00:57take on it, if I
01:00:58may, is that mainly that,
01:00:59you know,
01:01:00in psychology
01:01:02or especially, it's like
01:01:04excuse me. Science software is
01:01:05a fairly new phenomena that
01:01:06you pay attention to. It's
01:01:08like, previously, you just wrote
01:01:09your code and hope it
01:01:10worked,
01:01:11and then that's what happened
01:01:13in the seventies, eighties, nineties,
01:01:15and so on. And I
01:01:16still looked up code that
01:01:18was written in the, in
01:01:19the eighties and then, you
01:01:21know, but I didn't manage
01:01:22to get it to run
01:01:23because, of course, it was
01:01:24written in the eighties.
01:01:27But the idea is that
01:01:29science software and our the
01:01:30attention that we pay,
01:01:32pay to it is is
01:01:34a fairly new thing. And
01:01:35the a lot of funding
01:01:36agencies were not aware of
01:01:38it before,
01:01:39and now it's becoming a
01:01:41bit more of a concern
01:01:42with the new technologies that
01:01:43are developed. And maybe it
01:01:45gets more attention, maybe it
01:01:46gets more funding as well.
01:01:47On the other side, I
01:01:49think
01:01:51that,
01:01:52you know,
01:01:55toolboxes
01:01:55are kinda
01:01:57well, no. Actually, I'm gonna
01:01:58stop there. This is that,
01:02:00the the point is the
01:02:01science software is gonna, like,
01:02:02get more attention, and, hopefully,
01:02:04we'll get more support as
01:02:05well because it's an important
01:02:06part of our workflow and
01:02:08all sorts of things depend
01:02:09on it. But, yeah, we'll
01:02:10we'll see what happens.
01:02:12Yeah. We wanna even look
01:02:14into BRAIN initiative
01:02:16brands. They have certain mechanisms
01:02:18that support toolbox development.
01:02:22So Yeah. But yeah. Anyway,
01:02:24I was just thinking because
01:02:25the the beginning of neuroimaging,
01:02:27you know, is very similar,
01:02:28and people wrote their own
01:02:29codes to analyze their own
01:02:30fMRI data.
01:02:32Yes. Yeah.
01:02:34I mean,
01:02:36to be honest, I I
01:02:37do view, like, previous toolboxes
01:02:39be becoming, like, very supported
01:02:41as kind of like an
01:02:42accident because there was no
01:02:43infrastructure for it, as you
01:02:44say. So SPM big SPM,
01:02:47filled a particular need, and
01:02:49it became, like, big because,
01:02:52people started to adopt it
01:02:53very well. And, you know,
01:02:54it's just, that's the that's
01:02:56the thing that you have
01:02:57to focus on. And, yeah,
01:02:58the rest will, like, hopefully
01:03:00come or not. Who knows?
01:03:04Hey. If there is no
01:03:05more questions, then thank you,
01:03:06Leonard, again. This is Thank
01:03:08you very much.