Equity in Biomedical Research with Jennifer E. Miller, PhD

June 01, 2023

Information

April 19, 2023

Equity in Biomedical Research

Jennifer E. Miller, PhD

Associate Professor of Medicine (General Medicine), Yale School of Medicine

ID10004

To CiteDCA Citation Guide

00:00So welcome everybody and to the folks online,
00:05this is the program for Biomedical
00:07Ethics evening Ethics Seminar series.
00:09We're going to give it just one or two
00:11more minutes as folks come into the room,
00:13both this room here at Cohen
00:15Auditorium as well as the virtual room.
00:17So in just a couple of minutes, I'm going to,
00:19I'm going to introduce our our guest tonight
00:21Professor Miller and and we'll get started.
00:24So thank you very much for
00:25joining us in the room and online.
00:27And for those online, what are we
00:29having in the room tonight for dinner?
00:30We've got lobster and Steamship Roast beef.
00:34Look at that. That's nice.
00:36And look at that. That's great.
00:37And pizza from all of four New Haven's
00:404 finest pizzerias all out there.
00:42So keep that in mind.
00:42Next time, join us in Cone.
00:44We'd love to have the in person
00:47community here. What's that? Oh, great.
00:50There you go. Great Lobster.
00:51They're they're eating it right up.
00:52I don't know who's going to clean this up
00:54with all these lobsters on the floor, but.
00:55We'll deal with that.
00:56We'll give it one more minute
00:57and we will get started.
00:58I'll be right back.
01:43So good evening and welcome.
01:45My name is Mark Mercurio.
01:47I'm on the director of the Program
01:48for Biomedical Ethics here at
01:49the Yale School of Medicine.
01:51Welcome to the folks in the
01:52room and the folks online.
01:54It's a pleasure tonight
01:55to introduce our speaker,
01:57who I'll get to in just a moment to
01:59let you know how this is going to work.
02:01And I think many of you
02:02are familiar with this.
02:03And just a minute,
02:04I'll introduce Jen Miller,
02:05our guest for tonight and then we will.
02:09Professor Miller will speak for 45 minutes,
02:12plus or minus.
02:12We'll see how it goes,
02:13a PowerPoint presentation.
02:14After that we'll have a Q&amp;A session for
02:17the room as well As for the folks online.
02:19For the folks online,
02:20you won't be able to do this through chat.
02:22I would ask that you submit your
02:24questions through the Q&amp;A function
02:26and then I will read the questions to
02:29Professor Miller and we will go until.
02:31For a little while,
02:32I'll see how the questions go,
02:33see how the conversation goes.
02:35But if it's still going at 6:30,
02:36I will be stopping it.
02:37So you're wondering,
02:38is this going to go on forever?
02:39The answer is no.
02:40Sometimes it feels like we stop too
02:42soon because we're really into it.
02:44But to respect everybody's time,
02:45we do quit at 6:30.
02:47But right now we're just getting
02:48we're just at the beginning.
02:49And I'm delighted to tell you.
02:50So let me tell you about my
02:52friend Jennifer Miller, PhD.
02:53She's an associate professor
02:54in the old School of Medicine
02:56in the Department of Medicine.
02:57She's also the director of a program
02:59called Good Pharma Scorecard,
03:01as well as an organization
03:03called Bioethics International.
03:05I don't know about you,
03:06but when I was in college and afterwards,
03:07I figured out a pretty early on that
03:09the smartest people on campus were
03:11two different people and I was neither.
03:13There were the physics majors,
03:14I would say 3, the physics majors,
03:16the math majors and the philosophy majors,
03:19and one rarely encounters someone who
03:20actually develops expertise in both,
03:22so.
03:23Professor Miller actually did her
03:25Bachelorette at Fordham in Physics,
03:27then went on to study bioethics at
03:29Duke and at Harvard and eventually
03:32received her PhD at the Regina
03:34Apostleorum Pontifical University in Rome.
03:37She then founded Bioethics
03:39International and became a a well
03:41respected authority on the bioethics
03:43and the Pharmaceutical industry
03:45and our relationship with them.
03:47She also developed expertise and has
03:50spoken on artificial intelligence.
03:51And on bioethical issues with data
03:54sharing and on clinical research,
03:56she joined the REL faculty a few years ago.
03:59She came here from, I believe NYU,
04:00right Jen, and she came here from NYU.
04:02It's a marvelous addition to our
04:04faculty and I'm really pleased
04:05that she agreed to spend some
04:07time with us this evening.
04:08So I give you Doctor Jennifer Miller to
04:10discuss equity and biomedical research.
04:12Please welcome Jennifer Miller.
04:21Thanks Mark for that generous introduction.
04:24So as Doctor Mercario mentioned,
04:26today I'm going to talk about
04:29equity and biomedical research and
04:30focus on two areas in particular,
04:32diversity and fair inclusion in
04:35clinical trial enrollment and
04:36fair access to the benefits of
04:38research on a global level. Thank
04:45you, so for those who are. Meeting CME,
04:48there'll be 3 program objectives.
04:50First, I hope you walk away with the
04:52an ability to describe key ways for
04:55evaluating the adequacy of clinical
04:57trial diversity and representation,
04:59ways to analyze the degree to which women,
05:02older adults and racial and ethnic
05:04minoritized patients are fairly
05:06included in clinical research,
05:08and a better understanding of how
05:09to evaluate fair access to the
05:11benefits of clinical research among
05:12low and middle income countries.
05:18OK, so countless studies have shown a
05:21lack of diversity in clinical research,
05:23including our own. In general,
05:27we tend to test new medicines in vaccines
05:30on patients who are healthier, younger,
05:33and more likely to identify as white and
05:36male than real world US patients with
05:39the studied conditions and diseases.
05:44Other populations are also underrepresented.
05:46Policy efforts to improve clinical
05:49trial trial diversity span decades.
05:52Early efforts include in 19/19/83
05:57published guidelines from the FDA
05:59on the importance of including older
06:01adults age 65 years and older,
06:03which was finalized in 1989 and
06:06all the way through more recently
06:08with President Biden signing the
06:10Food and Drug Omnibus Reform Act,
06:12or FEDORA for short.
06:13Newly requiring research sponsors to
06:15submit diversity action plans for
06:18their pivotal trials and other later
06:20stage trials outlining enrollment
06:22goals for the first time by age,
06:25sex, race, ethnicity,
06:27geographic location and socioeconomic status,
06:30along with rationales for setting each
06:33goal and plans for how the sponsor
06:36aims to meet enrollment targets.
06:42We've had a lot of policy efforts and at
06:44the same time there's been substantial
06:48documentation on patient barriers and
06:51facilitators to trial participation.
06:53I'll just name a few.
06:55One are the use of overly restrictive
06:59inclusion exclusion criteria when
07:02designing trials in protocols.
07:04So for example,
07:05many trials include blanket
07:07exclusions for certain comorbidities
07:09or for concomitant medication use.
07:11So for example,
07:12you could have high blood pressure
07:16or another common condition
07:18and thereby be precluded from
07:20enrolling in a clinical trial.
07:22They're also known rural and urban
07:25gaps in clinical trial site locations.
07:27Most of our clinical trials particularly
07:30in oncology take place on the coasts.
07:33In in large academic medical
07:35centers and in major cities.
07:43And so given the amount of policy
07:46efforts that have targeted improving
07:48diversity in clinical research
07:50and the well documented barriers,
07:53many experts have started wondering you know,
07:54what else can we do to improve
07:56clinical trial diversity?
07:57And there was a paper that came out in New
07:59England Journal of Medicine this month.
08:01That said, you know what's very important
08:03is to to find why we're aiming,
08:05why we're striving for diversity
08:07and clinical research.
08:08And we wrote a similar paper
08:10led by Tom V Varma,
08:12brilliant medical student here
08:13at Yale with Kamara Jones,
08:15Carol Oladele and myself asking
08:18the saying the same question,
08:20stating the same problem when you
08:22read these policy guidance documents.
08:24Most of them fail to explain
08:27why clinical trial diversity is
08:29critical and why racial and ethnic
08:31representation in clinical research
08:33in particular is important given
08:35race is a social construct and is
08:37often grouped with sex and age,
08:38which are biological variables or attributes.
08:43And we are pretty worried that the
08:47existing guidance could unintentionally
08:49endorse a biological basis for
08:51race and ethnicity.
08:52So we worked pretty hard to.
08:54Provide some missing arguments for
08:56why racial and ethnic representation
08:58in clinical research is essential,
09:00although I'm going to say that Aaron
09:02Schwartz and colleagues in New England
09:04Journal of Medicine did it better.
09:06So basically we said the same
09:08thing that they did,
09:09that improving clinical trial diversity
09:11was critical for three reasons.
09:13One was enhancing trust in medical
09:16research and research institutions.
09:20So it's not just.
09:22How a technology is How a technology
09:24is developed affects who adopts it.
09:26And there have been studies that show
09:28that underrepresented patients are
09:30more likely to are less likely to
09:34trust medical evidence when they're
09:36underrepresented in clinical research,
09:37and less likely to believe a
09:39drug will be affected for them.
09:41And their doctors are less likely
09:43to prescribe and use medicines when
09:45their patients are underrepresented
09:47in in research samples.
09:52Further, clinical trial diversity
09:54is critical for promoting fairness,
09:57for providing equal opportunities or fair
10:00opportunities to participate in trials.
10:02And in the New England
10:04Journal Medicine paper,
10:05they note that by increasing infrastructure
10:08and building capacity to participate
10:11in clinical trials among community hospitals,
10:14you're you're also improving
10:17infrastructure for for care.
10:19And also, clinical trial diversity is
10:21critical for generating biomedical knowledge,
10:23for developing equitable access to and
10:26representation of medical evidence.
10:33So while there's been some preliminary
10:36work describing why diversity in
10:38clinical trial enrollment is important,
10:41there hasn't been a lot of work defining
10:44what good representation looks like.
10:47And so back in 2001, in October of 2001,
10:50the editors of the New England
10:52Journal of Medicine actually
10:53called this out and says that said,
10:54that we need a conversation about what
10:57constitutes acceptable and reasonable
10:59representative in clinical research.
11:01And similarly,
11:01the National Academies of Science,
11:03Engineering and Medicine published
11:05its report in May of last year
11:07also saying that we need to have a
11:09conversation on what constitutes
11:11appropriate representativeness.
11:16And so Tanvi Varma, Kerry Gross and
11:18I wrote a paper on this very subject
11:21sort of asking clinical trial diversity
11:22will you know it when you see it.
11:26And so the first thing to that we
11:28worked on was conceptualizing what
11:29does adequate representation mean.
11:33And in the literature there are
11:35two leading ways to conceptualize
11:38adequate representation which
11:39we call the country population
11:41approach versus the condition based.
11:44Approach The country population approach,
11:46as the name suggests,
11:50argues that the trial participant
11:52demographics should mirror a
11:54country's population demographics.
11:56So for the the US this would mean
12:00enrolling 50.5% female trial participants,
12:0313.6 black identifying
12:05participants and the like.
12:07And this would be condition neutral,
12:10so regardless of a trial's indication
12:13or targeted condition or disease.
12:15The condition based approach,
12:17in contrast,
12:19suggests the trial participant
12:20demographics should mirror those
12:22of the patient population with the
12:25studied condition or targeted disease.
12:30And here you can just see two
12:33different research papers.
12:35Each have used the respective approaches.
12:38For the country population approach,
12:39there's a paper here with the
12:41senior author was Janet Woodcock.
12:43And that was pretty recent.
12:44And then the other condition based
12:47approach dates back much earlier to
12:492013 with the paper led by Doctor Rita
12:51**** who was at the FDA at the time.
12:57So while both of these approaches are common,
13:01they applying them yields markedly
13:04different enrollment goals
13:05that we're not talking about.
13:08So in the paper we.
13:11We did two trials till we showed
13:12two trials and apply these two
13:14approaches to show how how you'd get
13:16markedly different enrollment target.
13:18So in case A,
13:20we said there's a Melanoma trial
13:23that enrolled 500 patients and we
13:26applied the country population approach
13:30and for if you use the general
13:35population you would aim to enroll.
13:3914% patients identifying as
13:41black for the Melanoma trial,
13:43but if you use the condition based approach,
13:45you would be aiming to enroll .5%
13:49patients identifying as black.
13:51And if you look at the multiple myeloma case,
13:55if you use a country population approach
13:58for patients identifying as Latino,
14:00you'd aim to enroll 19% patients
14:04identifying as Latino for.
14:06Sorry, the country population approach.
14:07And if you use a conditionbased approach,
14:09you'd aim to enroll 9%
14:13Country targets are 28 times greater
14:16than the conditionbased approach.
14:18And in the multiple my little mother's
14:20a 200% difference in enrollment targets.
14:28And so a group of us set out to try and
14:31flesh out what good representation looks
14:34like and how to conceptualize adequate
14:37diversity targets and enrollment goals.
14:40So we set, so we developed a measure and
14:42then we applied the measure to benchmark
14:44the adequacy of representation for pivotal
14:46trials supporting novel oncology products
14:48approved by the FDA between 2012 and 2017.
14:52And this study again was led by
14:54Tanvi done with Michelle Mello,
14:56Joe Ross who's in the room,
14:58Carrie Gross and myself.
15:01And so we had three main outcomes
15:04for the paper.
15:04The first thing we wanted to
15:06do was assess transparency.
15:08Could we determine from
15:09public sources the sex, age,
15:11and racial and ethnic identity
15:13of trial participants?
15:15Second,
15:15we looked at representation
15:16using the second approach,
15:18the country population approach,
15:19looking to see whether trial
15:21participant demographics mirrored
15:22those of the US patient population
15:25for the studied condition or disease,
15:27which we calculated by using a
15:30participation to prevalence ratio.
15:32And then we did a fair inclusion measure,
15:34which was the average of the
15:36transparency and representation
15:37scores and we reported results on
15:40the trial product and sponsor level.
15:42And so this is the characteristics
15:43of the sample we looked at.
15:45So between 2012 and 2017,
15:48the FDA approved a total of 59 products,
15:5139 drugs and 20 biologics sponsored
15:53by 25 unique pharmaceutical
15:56companies which targeted 16 broad.
16:00Oncology indications based on
16:02a total of 64 pivotal trials,
16:04a median of 1 pivotal trial per product.
16:10And here's what we found on the first column,
16:14you can see what we found on the trial level.
16:18While 100% of pivotal trials were
16:20transparent about the sex of participants,
16:24only 67% transparently reported.
16:26The age and proportion of older adult
16:30participants and only 41% the race and
16:34ethnic identity of trial participants.
16:37In terms of representation,
16:3981% of pivotal trials supporting
16:41the Oncology Products Nurse sample
16:43adequately represented women,
16:45but only about a quarter,
16:4726% adequately represented older
16:49adults patients aged 65 and older,
16:52and only 10% racial and
16:55ethnic minoritized patients.
16:57And then when you look at fair inclusion,
16:58both aside from women,
17:00the numbers go slightly down on the
17:03sponsor level, on the company level,
17:06on the fair inclusion measures,
17:07we found 50,
17:08only 54% of sponsors,
17:10pharmaceutical companies
17:11fairly included women,
17:1320% older adults and 4% racial
17:16and ethnic minoritized patients.
17:26And here you can see that
17:28in terms of representation,
17:30patients identifying as Asian
17:31are much better represented than
17:33patients identifying as Latino or
17:35than patients identifying as Black.
17:40What you also see here is
17:42that the transparency around
17:44patients identifying as Native,
17:46Hawaiian or Alaskan native is really low
17:50and so we couldn't actually benchmark
17:52the representation of these groups.
17:55In clinical trials,
17:56because we didn't know the percentage
17:59of patients identifying in these
18:01groups amongst trial participants
18:03and also we didn't necessarily know
18:05the incidence rate for the condition
18:08for these groups because of the
18:10limitations in the CDC databases.
18:18When you talk with pharmaceutical companies,
18:21often an anecdote that will come up
18:24is that well was an acknowledgement,
18:27well, maybe. We didn't get it
18:29right in the premarket studies,
18:30but if you had just looked at
18:31the post marketing studies,
18:33that's when we focus on clinical trial
18:35diversity and things would look a lot better.
18:37And so again our same group with the
18:41addition of some other researchers
18:43here at Yale took a look at premarket
18:46and post marketing studies and found
18:48that all things considered post
18:50marketing studies were generally
18:51worse in terms of representation
18:56that paper. Was led by Tom
18:58B and Josh Wallach, right.
19:00Joe, do you remember?
19:02Yeah, so here's where a lot of my work
19:08focuses is developing on account,
19:10is developing accounting ability
19:12measures and using them to benchmark
19:14pharmaceutical companies on those
19:16using those measures to incentivize
19:19or catalyze improve behaviors.
19:22And so I run something called
19:23the Good Pharmace scorecard that
19:25Mark mentioned at the outset.
19:26Which is an index that ranks and
19:28rates biotech pharma met device
19:29companies on their bioethics and
19:31social responsibility performance.
19:32It helped.
19:33It aims to help set and communicate
19:35clear bioethics goals and targets,
19:37track progress on those goals,
19:39recognize where there are best practices,
19:41and catalyze better behaviors were needed.
19:45And because there appeared to
19:46be market and guidance failures
19:48to address the clinical trial
19:51diversity problem I built,
19:52we built these measures into the
19:53Good Pharma scorecard.
19:58I'm hoping that the Good Pharma
20:00scorecard will help move the needle
20:02on clinical trial diversity as it has
20:04on other research ethics concerns,
20:06notably on clinical trial
20:08transparency and data sharing.
20:09So trial registration,
20:11results reporting, publication,
20:12and commitments to sharing individual
20:15patient level data from trials.
20:18On those measures,
20:19the good pharma scorecard has
20:21had measurable impact.
20:22Half of low scoring large companies
20:24will improve their procedures
20:25within 30 days of getting a low
20:27Good Pharma scorecard results.
20:29And the industry median scores
20:31have risen year after year on
20:33those measures since we began
20:36benchmarking for the 2012 approvals.
20:39And it's widely cited and used in annual
20:42reports human rights due diligence.
20:46Reports and social media accounts
20:49when companies score well.
20:54So that's why we broke up the
20:58diversity performance scores and
21:00aggregated onto the company level and
21:03introduced a rating system on this.
21:05So here you can see some companies
21:08scored in the top 25% and got
21:10a gold rating somewhere above
21:12the median and received a silver
21:13rating and the rest are unrated.
21:20And now we have a grant from the FDA
21:23Oncology Center for Excellence through
21:25the Cersei program led by Joe Ross.
21:29And here we're looking to identify positive
21:33deviant trials and sponsors leaders,
21:37trials that have gotten it right
21:39that have adequately represented
21:41specific demographic groups to set
21:43up a qualitative study to go in and
21:45interview them to see how they did it.
21:47What were the factors antecedent
21:49behavior strategies that they think
21:51enabled them to achieve top performance
21:54and perform better than peers.
21:59So as part of that process we extended
22:02our sample from just looking at 2012 and
22:0520/17/2012 through 2017 FDA oncology
22:07product approvals to a full 10 year sample,
22:10the 2012 to 2021 approvals and
22:14this is preliminary results.
22:16I was really curious to see if things had
22:19gotten better because another anecdote was,
22:21well, those are old trials.
22:22Those are you know,
22:24approvals that happened back in 2017.
22:26If only you had looked at
22:27the more recent approvals,
22:29things would look a lot better.
22:31And So what do you guys think?
22:33Do you think they look better anyone?
22:39So this sample looks at 111 products
22:42sponsored by 70 unique companies.
22:46Based on 121 pivotal trials
22:48that enrolled over 40,
22:49about 40,000 participants and
22:51each novel oncology product was
22:53approved based on one pivotal trial,
22:55median of 1 pivotal trial.
23:00Hear it from this slide.
23:01Again, this is not published yet.
23:04The what you see is that patients
23:06identifying as Asian are consistently
23:08overrepresented and remember that
23:10representation was calculated by using
23:12that participation to prevalence score.
23:14Where you compare trial participant
23:17demographics to the patient
23:19population demographics in the US.
23:22So taking out patients identifying
23:24as Asian for a second,
23:26here you can see that female women are
23:30generally well represented in research,
23:32but older adults remain
23:34under underrepresented.
23:35Patients identifying as black and
23:38Latino also remain underrepresented
23:40with no statistical statistically
23:43significant changes.
23:44Over the 10 year period
23:54that was a very US centered presentation
23:59and I'm we're not getting it right
24:04here and how are we doing elsewhere,
24:08It's something I was been sort of asking.
24:10So the first step in answering that question
24:14was understanding a bit more of where
24:17our clinical trials are taking place.
24:20And so we did a study looking at
24:25where our novel drugs and biologics
24:28approved by the FDA in 2012 and 2014
24:30were tested on the country level.
24:36And what we found is that these
24:38novel products were tested in a
24:40median of 26 different countries.
24:47And these trials enrolled
24:49about 300 participants each,
24:51a meeting of 300 participants.
24:53Roughly 20 of these
24:56countries were high income,
24:58a median of six were upper,
24:59middle and one low,
25:01middle and 0 low income.
25:06And so now another question we need to ask
25:08ourselves as ethicist is this the right
25:12way to situate multiregional clinical trials?
25:18And how do we, how do we start
25:20thinking about that question
25:22After we did our study,
25:25Jonathan Kimmelman's group led by
25:28Awan did a similar study in file
25:31and similar findings that most of
25:32our clinical trials are taking
25:34place in high income countries.
25:38And so do we need to increase geographic
25:41representation and research Joe Millum and I.
25:45Explored this question for BMJ Global
25:49Health and asking is this uneven
25:52distribution of trial sites by geography
25:55and income level and ethical concern.
25:58And we suggested that it was for two reasons.
26:04One, has the pandemic illustrated very well?
26:08The patients who can benefit from
26:10many of these new interventions are
26:12not limited to wealthier regions.
26:141/3 of the drugs that we reviewed
26:17treated infectious disease diseases like
26:20tuberculosis which disproportionately
26:21affects low middle income countries and
26:24the other 3/4 of drugs were for non
26:27communicable diseases which are also highly
26:29relevant to low middle income countries.
26:31Given that 3/4 of deaths now occur
26:35in them and at the same time there
26:37are concerns that trial data may
26:40not extrapolate across geographies.
26:42And product effectiveness can
26:43vary substantially by region and
26:45we just named one example,
26:47the PEN Avalent rotavirus vaccine,
26:49which had markedly different efficacy
26:51rates in low middle income countries
26:54compared to high income with preventing
26:58severe rotavirus gastroenteritis and 64% of
27:01vaccinated children in Subsaharan Africa,
27:0451% in Asia in compared to in comparison
27:07to 98% in high income countries.
27:11And similar efficacy variations have been
27:13found for other vaccines ranging from polio,
27:16cholera, yellow and yellow fever,
27:18as well as drugs including antimicrobials.
27:21Often the explanations for the
27:23variance are unknown.
27:25They might occur because of
27:27social determinants, for example,
27:29dietary nutritional differences,
27:30differences in healthcare,
27:31delivery and the like.
27:43Research ethics often relies on the
27:45social value principle or the social
27:48value requirements that states clinical
27:51research is ethical only if it generates
27:55generates generalizable knowledge
27:56that is expected to promote health.
27:59Traditionally, this requirement has
28:03been interpreted quite permissively,
28:06provided a study.
28:07Was expected to generate data that can
28:10benefit someone or some populations health.
28:13It's been understood to have social
28:16value and more recently we've been
28:18starting to ask who should benefit,
28:20for whom should the value accrue
28:26and by what This is Doug McKay and
28:28Kate Saylor have raised this issue
28:30in a particular salient way and
28:32noted that this is just unfair,
28:34that we haven't been asking The Who question.
28:37Sure. No, I don't mind.
28:42So do you mean is it,
28:46is it at the code to do it or is
28:47that the code the funnet research,
28:49you know, so something benefits
28:51just children or just just we say
28:55that it has to benefit everyone?
28:58In terms of the ethics of doing
29:00the research or finding it,
29:01I just want the following.
29:03I ask do me a favor,
29:06just repeat the question.
29:08So, so
29:10I think the question was who is the
29:14target audience for the question and the
29:17short answer is we didn't answer that.
29:20We asked the apriori question which was in
29:24it was more oriented from the sponsor level.
29:26How should you think about what
29:29are the ethical considerations?
29:31When situating your clinical
29:33trials on the country level,
29:34it was more that and we come to this
29:39conclusion which is we suggest that
29:42you should think about the distribution
29:44of the disease burden across the
29:46globe and ideally your your trial site
29:48locations should correlate with the
29:50disease distribution is what we suggest.
29:56So very preliminary cut
29:57and analysis and then.
29:59Hoping just to raise awareness
30:00about this issue and challenge
30:02others to think about it as well.
30:08So that was the first question, right?
30:09Where are we conducting our trials?
30:11How should we be thinking about
30:13situating our clinical trial site
30:15locations on the country level?
30:17But then at the same time,
30:19I was sort of wondering, well,
30:20what happens to these countries that
30:23participate in clinical research?
30:24Do they get access to the
30:27products that they helped test?
30:29So the next piece of that study after
30:33we found out where all the trials
30:35were located was to go to the the
30:37equivalent of their FDA sites and
30:38see if the product that had been
30:41tested in the country received,
30:43if it received regulatory approval
30:44in that country.
30:47And what we found that of the
30:5070 countries contributing trial
30:51participants for FDA approvals,
30:537% received market access to
30:55the drugs they helped test.
30:57Within one year of FDA approval
30:59and 31% within five years.
31:04And we looked for a subsample
31:06at 7 years and didn't find
31:09any significant improvements.
31:14When we broke up the sample by high income,
31:17lower middle income and upper
31:18middle income countries,
31:19you find that high income countries
31:22were more likely than lower
31:24middle income countries and upper
31:25middle income countries.
31:27To get product access
31:35and then when you bring it,
31:36break it up by geographic location.
31:38Unsurprisingly, you find that Eastern
31:42European countries, Western Europe,
31:46Canada got 100% or close to it access
31:51to the products they helped test
31:53by five years post FDA approval,
31:56in contrast to other countries
31:58like those in Africa that had zero.
32:01Percent access and then the Middle
32:05East falling in the middle and
32:07Central and South America also
32:10towards the middle of the pack.
32:16And other studies, this one not done by
32:19our group went to see that even if if
32:22a product was commercially available,
32:23was it affordable, which is right.
32:25So you could submit a product
32:27for regulatory approval,
32:28get approval to market the product.
32:30But the next question is,
32:31is it accessible And a piece of
32:35accessibility is affordability.
32:36And these this study shows that
32:39all the products but one product
32:42that they analyzed cost more than
32:43the monthly minimum wage and all
32:45the countries where they were
32:47tested and 12 cost five times more
32:49than the monthly minimum wage.
32:50But they only focused on
32:52Latin American countries.
32:53So now we're taking our sample and looking,
32:56trying to look at affordability for all
32:58of the countries that hosted trials.
33:03So they concluded that most
33:06pharmaceutical products tested in
33:07Latin America are unavailable and
33:10unaffordable to most of the populations.
33:13And then we did a study
33:15led by Reshma Ramachandra,
33:17who's an assistant professor in
33:20internal medicine here at Yale,
33:22looking at the COVID vaccines
33:24that were recommended for
33:26emergency use authorization by
33:28the World Health Organization.
33:30And we were curious, you know,
33:32where they were tested,
33:34where they authorized for emergency
33:36use in the countries hosting trials
33:39in support of their FDA approval.
33:41And then were there inequities in
33:44delivery or procurement of supplies.
33:47And while we found that most of the,
33:51if not all of the vaccines were
33:53authorized for emergency use,
33:55generally speaking in the
33:56countries where they were tested.
33:59We found inequities in
34:01procurement of supplies
34:08and so a question for us is,
34:10is this ethically problematic,
34:14the gaps between where we test drugs and
34:16where they become available for patients,
34:24and So what do we know?
34:25From some bedrock principles and ethics.
34:27So bedrock principle of research
34:28ethics is that the benefits and
34:30burdens of research should be shared
34:32equitably by the people affected by it.
34:34This is in the CIOMS guidelines,
34:36and that a corollary of that principle
34:38is that to avoid exploitation,
34:40research should not ordinarily be
34:42conducted in a national population
34:44that does not stand to benefit from
34:46the knowledge or the interventions
34:47to be gained from the study.
34:49The interesting thing about these
34:51principles is they sound really good,
34:52but they don't specify the type of
34:54benefit that needs to be provided,
34:56how much benefit,
34:57or exactly who should receive the benefit.
35:02And in theory, you could argue that
35:04there's two camps in this space.
35:05There is the responsiveness requirement
35:08camp in among ethicists in the
35:10Fair Benefits Framework group.
35:12On this issue, the responsiveness
35:15requirement is imposes content restrictions.
35:18On that benefit that can provide
35:20be provided and argues that the
35:23type of benefit matters and that it
35:25should probably include the product
35:28that the country helped test.
35:30In contrast to the fair benefits framework,
35:32which I think you can argue in
35:35some ways is content neutral,
35:36it doesn't specify what the
35:39benefit has to be,
35:41but rather specifies the process by which.
35:44You must follow to identify the benefit
35:46and that it should be a collaborative
35:48partnership with the country and a
35:51transparent collaborative partnership in
35:53identifying and agreeing upon benefits.
35:56The responsiveness requirement
35:57framework usually responds to this
35:59and says that's nonsense on stilts.
36:02How could you possibly?
36:04Think that a low income country has
36:07any kind of negotiating power with a
36:09multinational major pharmaceutical company.
36:11Given that pharma companies can just shop
36:15around for a different trial site location.
36:18The fair benefits framework also
36:20implies that quantity matters.
36:22And so they might argue that, well,
36:25if a country only contributes,
36:27you know, 10 participants,
36:29which is entirely possible and likely that.
36:32That country may not be owed as much as a
36:36country that supplies more participants,
36:38say 100,
36:39right?
36:40And so the amount of participants
36:43for them might correlate with the
36:44amount of benefit that's owed,
36:48regardless of which camp you've fallen.
36:50None of this is likely happening, right?
36:52There's likely not collaborative partnership.
36:54There's not likely transparent collaborative
36:57partnerships around determining benefits.
37:00So it's really.
37:01So that's you know, something that
37:03I'd like to start investigating is
37:05what do these contracts look like?
37:07Are there countries that are
37:08doing better than others, right?
37:09Are certain countries able to
37:11achieve and procure consistent
37:13access to products that they help
37:15develop than others and if so, how?
37:21So wrapping up,
37:23I focused on two sides of a coin.
37:27In one case we were selling products.
37:30Two populations without testing
37:32adequately or at all in those
37:35populations and then the other case
37:37we were testing and not selling.
37:45So I have raised more questions than
37:47I've answered because we're at that
37:49stage and some of these issues.
37:51So I'm just merely going to end with,
37:52we really need a lot more work
37:54amongst us ethicists to conceptualize
37:56what constitutes fair access to
37:58the benefits of clinical research
38:00and then how to operationalize
38:03that conceptualization. Thanks.
38:09Are you ready for it?
38:11All right, Thank you so much,
38:14Doctor Miller, lots of questions.
38:16I invite you now.
38:18Should we stop the share?
38:24And that's, that's all That looks good.
38:25And that all that looks even better.
38:27Great. We're all there.
38:28Except now if we could turn the screen off so
38:30that we don't have Jen behind Jen behind Jen.
38:32That was like the Quaker votes.
38:34They're all there. Thank you, Sir.
38:36All right, thank you.
38:37That was a great talk.
38:38This is really interesting stuff.
38:39I'm like taking notes here,
38:41an old man, you know,
38:428 hours into the work day and
38:4310 hours into the work day.
38:44And you got me taking notes.
38:45So I will invite you all,
38:47please online to contribute your
38:49questions to the Q&amp;A function.
38:51And, and,
38:51and I'm going to take the prerogative
38:53of asking the first one and then invite
38:55you guys also to kind of jump in here.
38:57So here's I was thinking as you went to this
38:59and your last slide really touched on it,
39:01Jen, I was thinking, all right,
39:03looking at this from the point
39:04of view of I'm a manufacturer,
39:05I've got this new drug for a
39:07certain disease and I'm thinking
39:08this is going to be really good.
39:10And it strikes me that,
39:11well,
39:11this one thing is clear as this
39:13drug is going to be expensive.
39:15So now I think perhaps I'm damned
39:17if I do and damned if I don't.
39:20Because here's the deal.
39:20If I test this in a country where in fact
39:22they're not going to be able to afford it,
39:24or many won't be able to afford it,
39:26most won't be able to afford it,
39:27that kind of smacks of exploitation, right?
39:30That would be your testing,
39:31but not selling framework.
39:32So if I test this in a in a in
39:36a much lower income country,
39:37that seems wrong.
39:40And if I don't test it in a
39:41lower income country, well,
39:42now it's not good because I didn't do.
39:43If the disease burden is
39:45significant in that country,
39:46I'm supposed to be looking at
39:47the global disease burden.
39:48So it seems I can't really win.
39:50And the response from those who
39:53know this stuff well would be what?
39:56How do I get around this?
39:57It's going to,
39:57you know,
39:58do I should do I test it in a low
39:59income country when I know they're not
40:01going to be able to afford it very well?
40:03Or do I just test it here and
40:05I know I'll be able to sell it,
40:06but then someone's going to criticize
40:08me for not testing it more globally?
40:21Yeah. So ideally we would want every
40:24patient needs a product to be able to
40:27afford and access the products, just
40:30move it up a little higher.
40:31And in some ways that question is,
40:35was an inspiration for looking for
40:38where new products were tested.
40:41Under a hunch that if we tested a product
40:44locally that it might be more likely that
40:46we would submit the product for sale,
40:47make it commercially available
40:49and then you know,
40:50we could work on affordability down the road.
40:53But I got stuck on the first piece
40:55because it turns out we're not testing
40:58and then we're not submitting for
41:00regulatory approval and then affordability
41:02is really done far down the road.
41:06So it's really hard to talk about
41:08affordability if you're not even submitting
41:10products for regulatory approval.
41:11Somewhere.
41:12So in the ideal world,
41:13you would do all of those and we're just
41:16really far away from that right now.
41:18And the affordability question is
41:20very pertinent and salient one,
41:22especially as we start developing the gene
41:24therapies which are incredibly expensive
41:26in the US and difficult to develop.
41:32Thank you.
41:35The the next question will go to
41:36Joe Finn's and then I'm going to.
41:37I shouldn't mention names on this,
41:39but I haven't figured out how to do this
41:40without this whole thing popping up.
41:41If you can get rid of the side screens too,
41:43that would be great.
41:44In case somebody wants to
41:45submit a question anonymously.
41:47But now Joey's been out as year
41:48but I'll read his question anyway.
41:50Thank you for your talk.
41:511 area that I missed as an ethical
41:54justification for equity and inclusion
41:55is that we can learn a lot more
41:58scientifically from a diverse sample.
42:00We will see variance,
42:01more we will see variance.
42:03More information on basic
42:04mechanisms or adverse events that
42:06may impact certain populations.
42:09Why hasn't the clear scientific
42:11utility slash instrumentality
42:13been more prominent in the
42:15arguments in favor of equity.
42:17I think it's
42:18always been there I I rarely
42:21see it missing, but I right?
42:24Isn't it part of the
42:25generalizability arguments that
42:26you need to make sure that our.
42:28The clinically distinct groups are
42:30represented in the medical evidence,
42:31so I I haven't really seen it missing.
42:36But
42:38on the other side, I've seen it
42:40in the ethical justification,
42:41but I haven't seen as many studies
42:47showing how pervasive different reactions
42:51or different efficacy profiles are
42:54for different demographic groups.
42:58Other questions, Bonnie, wait,
43:02wait one second. So that the
43:03folks online can hear you too.
43:05So put that microphone on close. OK.
43:09Again, thank you. You're talking
43:11about a really important
43:12issue and I'm wondering about
43:16the point you just made, for instance,
43:18that you want to have various
43:20sorts of representative groups
43:21represented in your data sample
43:23because then you would know a lot
43:25more about how this particular.
43:27Medication or therapy
43:29might affect those groups,
43:31but I'm also thinking about groups
43:35that may be unusual in that they're
43:38rather insular in their behavior.
43:40Like I'm thinking about religious groups
43:42or their insular in their genetics.
43:44Same thing with religious groups,
43:46various immigrant groups,
43:50groups where they tend to focus
43:52in one particular location.
43:54So you're going to have a whole bunch of
43:57different genetic and environmental and
43:58behavioral factors that are particular
44:00to those groups that may not be captured
44:04if you have these wide ranges of age,
44:08race, etcetera, gender.
44:10And I'm wondering how you deal with
44:13those kinds of diversity issues because
44:15there may be important differences.
44:17Yeah. So the guidance documents are
44:19starting to acknowledge that, right.
44:21And so when you look at the at
44:23Fedora includes geographic location,
44:25socioeconomic status and some of the
44:28different variables you mentioned,
44:30you know how many variables we need to
44:33add is is and and I have a very crude
44:36conceptualization of of diversity,
44:38right, just focusing on those big
44:41categories because we haven't
44:42even gotten those right yet.
44:45And so and it's.
44:47It it's really hard to benchmark how
44:49we're doing on the other representations
44:51groups based on public data.
44:56Yeah. And so, you know patients
44:59with disabilities, pregnant women,
45:00women who are lactating and
45:02not an adequately controlled
45:03and not taking contraception,
45:05all those groups have been
45:08known to be underrepresented in
45:10clinical research that I didn't
45:12talk about all the all the groups.
45:17You get the mic
45:20you get the hand mic so you don't have to
45:24I just had coffee so I don't want to
45:26subjected to my
45:29I think that was
45:30a great presentation really compelling
45:33really an important issue And what
45:36I what I wanted to ask is sort of
45:38you know I think there there are a
45:40lot of not I think I know
45:42the data demonstrate there
45:43there are a lot of these like.
45:44Really big system level problems
45:47and and I think a lot of
45:49times we as as individuals
45:51feel a little bit almost like this
45:54paralysis like the problem's so big
45:56like what can we do about it. And
45:58and I and I was wondering if you could
46:00speak a little bit about that like
46:01I I know that there is data showing
46:03that for example trials,
46:05clinical trials led by women
46:07tend to have more gender as well
46:09as racial and ethnic diversity.
46:11And that you know some proposed
46:13solutions are trying to recruit more
46:16women and individuals underrepresented
46:18in medicine and and science or or
46:20minoritize populations into these
46:22positions of leadership to help with that.
46:24So again, that's not so much individual
46:26but at least institutional rather
46:28than like so broadly systemic.
46:29But can you speak a little
46:31bit realizing that you know,
46:32no one person can fix this but
46:34what are some things that that
46:36maybe we can do as as individuals?
46:40Or as institutions, you know,
46:41within our own institution to
46:43maybe advance this cause forward.
46:46Yeah, so there's a lot of documentation
46:48of theories on barriers and facilitators,
46:51and some of them are more than theories.
46:53But the short answer is evidence
46:55based action is still needed,
46:58which is part of the reason that we want to
47:01do the positive deviant study where we find
47:04out the trials that got it right, right,
47:06the ones that were able to adequately.
47:09Represent patients identifying as Latino
47:11or black or older adults 65 and older,
47:1475 and older to go into study, you know,
47:18how did they do it and then be able to
47:21develop generalizable best practices
47:22that can be implemented by everybody.
47:24We don't actually have that evidence
47:27based guidance.
47:28So I can tell you the barriers and
47:31facilitators that you could address that
47:34are already documented in the literature
47:36but aren't necessarily evidence based yet.
47:38So when if you're designing trials,
47:40you're going to be looking at your protocol
47:42inclusion exclusion criteria, right?
47:43Did you cut and paste certain exclusions?
47:46Because you've always done it and
47:47that's the way things have been done.
47:49If you're on the IRB,
47:50you're going to be looking for those
47:53unnecessary exclusion criteria,
47:54overly restrictive.
47:56Your trial site locations,
47:57you can invest in infrastructure to
47:59make sure that community hospitals
48:01are able to participate in trials.
48:03And it's not just the large academic
48:05medical centers that are hosting trials.
48:08Workforce diversity as you mentioned,
48:12working on ensuring inter that we're
48:15not discriminating consciously or
48:17unconsciously you know against certain.
48:20Groups that were offering the the
48:22opportunity to participate in trial
48:25consistently and fairly to all
48:27demographics who qualify for trials
48:29that were addressing language barriers,
48:31right to trial enrollment that we have
48:36translation translators available.
48:39Other barriers are child care
48:43and elder care sometimes.
48:45And transportation, right.
48:46If you want to participate in a trial,
48:48you have to be able to get to a trial,
48:50you have to have care for
48:53any dependents that you have.
48:56Distrust has to be addressed.
48:59There's distrust that's
49:02heightened in certain groups,
49:04justifiably so,
49:05in in research and in in medical
49:09institutions given prior injustices.
49:17Literacy, right. And an awareness of trial
49:21opportunities because studies show that
49:26there's conflicting evidence,
49:27but a lot of studies show that an equal
49:31interest in participating in trials
49:34but an unequal opportunity to enroll.
49:37So there's there a few, thanks.
49:40Let's hear from Steve, please.
49:41And then Jack and then we have all right.
49:45Steve and then Jack and
49:46then lady on the left. I'm
49:50getting older and things happened long,
49:52longer and longer ago.
49:53But my memory is that the the,
49:55the idea of Fair benefits first
49:58started cropping up because people
50:00started realizing that most trials
50:02fail and you've got populations who
50:04are going to be subject to research
50:06risks and they're never going to get
50:09the drugs because the drugs never
50:11going to prove to be efficacious.
50:14So you want to give them some fair
50:16benefit instead and that might be
50:18building of infrastructure and that
50:20might be training of nursing staff
50:22or it might be any one of the things
50:24that you just sort of went through
50:28and that could be happening.
50:31I'm probably pretty sure that it's not,
50:33but that could be happening.
50:34If you look at your data saying,
50:35oh, they test this in these poor
50:37countries and those countries
50:39never get access to the drug,
50:41well okay, but maybe they're getting.
50:44Nursing training instead,
50:47would that satisfy you if
50:49that were in fact happening?
50:51As I say, I suspect it's not
50:53actually happening that much,
50:54but if people were getting some
50:56kind of non drug fair benefit as
50:59a result of having participated
51:01in trial, is does that,
51:03Yeah, it's entirely possible
51:05that there have been schools and
51:07playgrounds built everywhere, right?
51:10Ventilators don't need it, right?
51:12The Sarfax in case.
51:14But if if you're in the
51:17fair benefits framework,
51:18you also would like a transparent
51:21collaborative partnership, right?
51:22In in determining and identifying
51:24benefits that are shared and
51:26the part and like you said,
51:27it's just not transparent.
51:29So we don't know if there are schools
51:31and playgrounds all over the place.
51:35Personally, I'm I'm more on the.
51:40Responsiveness principle,
51:41that framework that I think you you
51:45that's it's the benefit should include
51:46the product that you helped test, right,
51:48Because you clearly have a patient
51:51population there who needs the product.
51:55But you would give a followup question, no,
51:58you can't do that in the trial.
52:00Thanks. Oh yeah, it's then. Yeah.
52:03Well, that's why I said and do and.
52:09OK. All right, Jen, thank you very much.
52:15Now my assumption based on very limited
52:19information was years ago that drug
52:22companies did studies in in low income,
52:26middle income countries because
52:28it was cheaper that way.
52:30And so that's understandable.
52:33Now just to look at it from an economic
52:36perspective if we're talking about.
52:38Only distributing within
52:40this country to rural areas,
52:43to smaller community hospitals,
52:45that sort of thing.
52:46How do the costs work out?
52:49Does does it cost any more for
52:52the companies to do that away
52:54from academic medical centers?
52:56Does that add anything to?
52:58Is that if it costs more,
53:00is that it?
53:01That would be a disincentive it seems
53:03like or for all I know it's cheaper,
53:06but I'm just asking.
53:08So, yeah,
53:11well, let's talk about the ethics first and
53:12then we'll talk about the empirical data.
53:14So I think from the you might have been
53:18able to tell what I'm going to say, right.
53:20I I think cost is not the right framework.
53:22I think that we should be thinking
53:24about this as it's the right thing to
53:26do because we need the medical evidence
53:28to be developed generalizable medical
53:30evidence for clinically distinct groups.
53:32We need trust, right and.
53:40We need uptake of products.
53:41There was another one.
53:42I'm blanking on the second one.
53:45So I think cost is not is
53:50not the priority, right.
53:52I think those other values and
53:54goods are going to trump cost.
53:56But on the cost question,
53:57I haven't seen an empirical
53:59study addressing that.
54:00It's an anecdote that flies around a lot.
54:03It will cost too much.
54:04It'll cost more to run a more diverse trial,
54:06more geographically diverse,
54:08more demographically or diverse.
54:10And Joe and I and Kerry were
54:12just exchanging emails saying,
54:14you know,
54:14we really should do that study
54:16because we have a lot of that data
54:18collected where we have the trial
54:20scored on diversity and we have the
54:22trial start dates and end dates.
54:23And is there a way to look at
54:25whether the more diverse trials and
54:27whatever variable you're looking at,
54:29geography, race, ethnicity?
54:30Age,
54:31whether they were slower or
54:34more costly to run in some way,
54:36but I I haven't seen that data.
54:38Has anyone else seen that those data?
54:41Yeah.
54:41But I think you it's an important study
54:44to do not because cost it's a justifier.
54:47But to get rid of that that myth,
54:50it has to be addressed because
54:52that's going to be the objection
54:53of those people who wanted to plan
54:55to run the studies and so you
54:57have to be able to to address it.
55:00And and deal with it
55:04so. So while Jack is handing the
55:06microphone off I'll remind everybody
55:08that we can get CME credit by texting
55:11the text code for tonight's session
55:14is 36149 and to accomplish that you
55:18it's written in the chat portion I
55:19hear believe you can see the phone
55:21number that you need to call on the
55:22code and you can do that it's two O
55:2734429435. And then you, when texts
55:30to that 36149 to get CME credit,
55:35Chuck, I go back to you.
55:36What happens if the empirical study
55:38says that it's more costly to do
55:40to conduct diverse trials? Then
55:42we say dad costs more.
55:45But it's worth it's important
55:48to do. You just want to know you need a mic,
55:53right? We have it. No, it's you.
55:55This is all you want the way.
55:59It's a curiosity rather than
56:00a justification. Yeah. It
56:04is planning your moral strategy that
56:09you have to know what your opponents. If
56:14moral persuasion fails,
56:16it will save you money.
56:19Yeah, or it won't cost more.
56:21Yes, Yes. No, I I agree.
56:25I agree. I agree.
56:30I have to apologize at first since my
56:33English is Limited Head Out Miller.
56:36My Major is Bell Essex,
56:38especially AI Essex and Clinical Essex.
56:41So my question is as there was
56:45research about the comparison of
56:48enrollment goals using two approaches
56:51to achieving Adequate Adequate.
56:53Representation in research,
56:54I would love to know do you think that
56:58it will be meaningful or helpful to
57:02do a research about the comparison of
57:06using AI tools and the traditional
57:08ways in the recruitment or the
57:11retention process in clinical trials,
57:14Since I think maybe AI tools
57:16could help us to solve a lot of
57:19problems in the clinical trials.
57:20So I would love to know
57:22your opinion about that.
57:23Well, there's certainly a lot of efforts
57:27to apply a I to tackle this problem.
57:29I think it's a little early to
57:32see how helpful they will be.
57:33So some just some descriptive
57:35information I've heard of right using
57:38various algorithms and and natural
57:42language processing programs to
57:45identify patients that might qualify.
57:47For trials and notifying A clinician
57:50that their patient qualifies and
57:53that they giving them an opportunity
57:55to enroll their participants.
57:57More so,
57:58I've heard about decentralized trials
58:00and using digital tools right to to
58:03allow participants to enroll in trials
58:06rather than setting up right major
58:09clinical trial sites like we currently do.
58:12But that too has hurdles.
58:13One of them is an ethics related one,
58:15in that with decentralized trials
58:17currently you have to use an IRB at each
58:22each participation.
58:26I don't know what you're calling it a center,
58:29whereas right when in the clinical
58:30trial you can use a centralized IRB.
58:31So in some ways these things will look there,
58:35they're going to help,
58:36but there's still some bureaucratic mess.
58:38Do you have ideas of how
58:39you think AI would help?
58:43I know that there is tools called Mando AI
58:47that help to help the recruitment process
58:50in the clinical trials and it is used,
58:53It was used in some in some centers, yeah.
58:58Right. So it could in theory offer
59:00more opportunities to individuals,
59:02right, by identifying them.
59:03But it it it that wouldn't
59:05necessarily fix an underlying cause,
59:07which it would probably be
59:08applying the inclusion exclusion
59:10criteria in the trial protocol,
59:11which in itself could limit who qualifies,
59:14right. So there's blanket exclusions
59:16for certain comorbidities, polypharmacy,
59:18then older adults might be more likely
59:21to be unable to qualify, right?
59:23Or other different demographic groups.
59:25So applying a I to.
59:28Problematic inclusion exclusion
59:29criteria could double down right on or
59:32triple down a I down on the problem.
59:36And yeah,
59:38I want to switch gears a little bit,
59:39Jen, because I don't know if you folks
59:41are really aware of the the the work,
59:43the earlier work that Jen did when we
59:45first met in Bioethics International and
59:48this notion of the of the scorecard.
59:51And and I was so pleased when I when
59:52you saw that that as a result of
59:53a low score half of the companies,
59:55you know we can see the glass is half
59:57full here that people do respond to this.
59:59But this was as far as I know that
01:00:00you were the first one,
01:00:02this is even before you got your PhD,
01:00:03you were the first one who was doing this
01:00:05work and it's it's really quite interesting.
01:00:07So could you talk for a minute or two
01:00:10about what the scorecard entails and how
01:00:13how you evaluate A pharmaceutical company?
01:00:17What are the,
01:00:18what are the criteria that you're
01:00:19looking for and how they get scored,
01:00:20if you will.
01:00:21Yeah,
01:00:24thanks. So this, the scorecard started
01:00:29out very humbly as a tool to bridge
01:00:32asymmetries of information about
01:00:34the performance of pharma companies,
01:00:37a lot of the media.
01:00:39And the court cases build a pretty
01:00:41damning picture of pharma companies
01:00:43and the settlement agreements,
01:00:46corporate integrity agreements.
01:00:48But when you would speak
01:00:49with the pharma companies,
01:00:50they would say, well,
01:00:51those are outlier
01:00:55rogue companies in an
01:00:58otherwise good industry.
01:01:01Or if you spoke to the
01:01:02company that had the scandal,
01:01:04that was a rogue employee.
01:01:07Or an outlier department in
01:01:09an otherwise sound company.
01:01:11And so it was really hard for those
01:01:13of us who weren't in the industry
01:01:15to understand what was going on.
01:01:17And another talking point was those are
01:01:19old issues that have been resolved.
01:01:22And so the good from a scorecard in
01:01:24some ways started as a prevalent study
01:01:26starting with clinical trial transparency.
01:01:30Which is aware that companies weren't
01:01:32being honest and truthful about the
01:01:34safety and efficacy information about
01:01:35new medicines and vaccines that they
01:01:37were selectively selectively reporting
01:01:39trial outcomes or selectively reporting
01:01:41trial trial results and trial outcomes.
01:01:45And so I just set out with Joe.
01:01:48Way back when,
01:01:49like a decade ago,
01:01:50more than a decade ago,
01:01:52to figure out what does honesty and
01:01:54truth telling look like in the context
01:01:57of clinical trial results, right.
01:01:59How do you operationalize
01:02:00these these principles?
01:02:01We we talked about in values that we
01:02:03talked about in bioethics and how you
01:02:06develop accountability measures around there.
01:02:08And so the first thing is what's the goal,
01:02:10right?
01:02:10Honesty and truth telling.
01:02:11What does that look like in the
01:02:13context of medical evidence,
01:02:14registering trials,
01:02:16results reporting?
01:02:17Publishing results and then that changed
01:02:19to act to include sharing of individual
01:02:22patient level data and clinical trials.
01:02:24So you get these accountability measures and
01:02:26we use them to benchmark pharma companies.
01:02:29And what we found was that most companies
01:02:33did not meet the measures that we developed.
01:02:36And so we got all these companies
01:02:40together back in 2009 and then
01:02:44again in 2000 and I don't know.
01:02:46Early 2000, maybe 12,
01:02:47and said what happened?
01:02:49You guys said this was an outlier problem,
01:02:51a rogue company, an old issue.
01:02:55Why aren't you scoring better?
01:02:57And there was this backandforth dialogue,
01:02:59right?
01:02:59Oh well, you and you know,
01:03:01it was sort of scratching their heads.
01:03:02And then the meeting ended and
01:03:03we held another meeting and
01:03:05they came back and they said,
01:03:06well, you looked,
01:03:07you measured the wrong thing.
01:03:10And we were looking at all trials
01:03:12where pharmaceutical companies
01:03:13disclosing the results of all trials
01:03:15supporting FDA approval of products.
01:03:17And we said, oh, well,
01:03:18what trials should we have looked at, right.
01:03:19They said just the trials and
01:03:22patients for the approved indication.
01:03:25And before that they said, well,
01:03:26actually legally we're not required to
01:03:28disclose all the all the trial results.
01:03:30We just followed the law, right.
01:03:31And so this is an opportunity
01:03:33for a little education.
01:03:34Oh, so for ethics for you means minimal
01:03:36compliance with the law, right.
01:03:38What is ethics? Yeah.
01:03:41And so we realized that the good
01:03:43pharma scorecard could also create
01:03:45a knowledge exchange platform,
01:03:46right, where we could have this
01:03:48bidirectional education and
01:03:49dialogue on what good looks like.
01:03:51Is it just minimal compliance with the law,
01:03:53but is the law even being met?
01:03:55And so the next paper that we did,
01:03:58we added an analysis of legal compliance.
01:04:00We'd actually already done it in
01:04:01advance because I kind of figured
01:04:02that would be their pushback, right?
01:04:03And when we put up the slide,
01:04:06we showed that you know,
01:04:06less than half of companies were meeting
01:04:09minimal legal requirements for transparency.
01:04:10And so you know, you would just
01:04:12year after year sort of chip away.
01:04:14That's too expensive, right,
01:04:16to conduct a more diverse trial.
01:04:18Our competitors will get more investments,
01:04:21you know.
01:04:21So now we look at whether more ethical
01:04:25companies can financially outperform
01:04:26their peers and it turns out they do,
01:04:29but that hasn't been published yet.
01:04:31There's alpha.
01:04:33Yeah.
01:04:34So the good pharma scorecard started
01:04:35as the way to bridge asymmetries
01:04:37of information about the ethical
01:04:39performance of pharma companies.
01:04:40But then we turned,
01:04:42we turned out that there were pervasive.
01:04:44Genuine,
01:04:45widespread and current ethics
01:04:46problems within the sector.
01:04:48So we turned our question to
01:04:49how do you reform them?
01:04:50And there are many reform
01:04:52strategies out there, right?
01:04:53Passing laws.
01:04:53But as I just mentioned,
01:04:54they weren't sufficiently moving the needle.
01:04:58There's civil society activism,
01:04:59which there had already been
01:05:00in the space of clinical trial
01:05:02transparency with Ben Gold Acres work,
01:05:03for example,
01:05:04in London with the All Trials campaign.
01:05:08And then there's a lot of different tools,
01:05:10but they weren't working just like in
01:05:11clinical trial diversity for 10 years,
01:05:13right? No, no statistical,
01:05:14at least significant changes
01:05:16in in in representation.
01:05:18And so that leads you to
01:05:19ask what else can you do?
01:05:21And almost every industry has used
01:05:23an accreditation certification
01:05:24rating or labeling program as a way
01:05:26of communicating what good looks
01:05:28like benchmarking and signaling.
01:05:29Performance on measures including
01:05:31hospitals which was pioneered in
01:05:33some ways by Harlan Krumholz here,
01:05:35right with a hospital quality measurements
01:05:36which is part of the reason I was
01:05:38excited to come to Yale many years ago,
01:05:40several years ago and Joe and Joe's
01:05:43work and we also have an environmental
01:05:45performance index where we rank countries
01:05:47on their environmental performance.
01:05:49What is what does good look like and
01:05:51how are different countries performing.
01:05:53So that so then we thought
01:05:55we'll we'll develop.
01:05:56An accreditation or a certification
01:05:58or rating or ranking,
01:05:59it turned into a ranking that's
01:06:01now a rating and a label.
01:06:04You get a badge because pharma
01:06:06companies created their own badge
01:06:07and tweeted when they scored.
01:06:08Well, so we were,
01:06:09we thought we better create our
01:06:11badge for them. That's standardized.
01:06:12And now there's a badge you can display
01:06:14and it goes into annual reports,
01:06:15as I mentioned.
01:06:17And it's become pretty widespread
01:06:19across the sector and it looks bad
01:06:20if it makes it into your annual
01:06:21report one year and then it's
01:06:23not in it the next year, right.
01:06:24So and then we rely on the help of everyone.
01:06:26So if there no eyeballs on the scorecard,
01:06:28it doesn't have as much impact.
01:06:29So it really have to work with the
01:06:31media to get attention on the scorecard.
01:06:33It's been a journey and now we're
01:06:34trying to work with investors that's
01:06:36why we're looking to see if if
01:06:38ethical performance is correlated
01:06:40with financial performance on
01:06:41the firm level
01:06:43or negatively correlated.
01:06:44Is that what you're saying? Yeah.
01:06:46Well, that would hopefully not.
01:06:47Yeah. So the, the spoiler alert,
01:06:50it's preliminary is that many of
01:06:53the measures are correlated with
01:06:55positive financial performance,
01:06:55which is what we were hoping to find.
01:06:58That's exciting. That's wonderful.
01:06:59I congratulate you on that.
01:07:01I mean I say that's exciting stuff
01:07:02is to be doing something to be to be
01:07:07cutting a new trail that
01:07:08others haven't. Yeah,
01:07:09everyone's, everyone's cutting.
01:07:11It's been good to talk with them.
01:07:13It's great. We have a,
01:07:14Jackie, have a question.
01:07:17We don't mind.
01:07:23Thank you, Chris.
01:07:26Although George Bush looks,
01:07:28George Young Young George Bush looks,
01:07:32perhaps looks better in retrospect compared
01:07:35to what's happened subsequently before.
01:07:39Eight years ago, 10 years ago,
01:07:42I really thought that.
01:07:44He was pretty much a disaster
01:07:47except for PEPFAR and which is
01:07:49really as near as I can tell,
01:07:52a pretty amazing accomplishment.
01:07:54So maybe Jack, if you would,
01:07:56you're referring to the work and about AIDS
01:07:58research and from the president in Africa,
01:08:00etcetera, yes,
01:08:01if you could give us because not
01:08:04everybody may have be familiar with it,
01:08:06not everybody was, you know.
01:08:08Paying close attention when the
01:08:09young George Bush was doing stuff.
01:08:10You could give us a four sentence
01:08:12summary of Pepsi or A2 sentence
01:08:13summary of that program.
01:08:14Maybe one sentence,
01:08:15one sentence would be fine.
01:08:17Yeah, it was presidents.
01:08:19Well, I don't remember that.
01:08:21I can't possibly repeat the the type,
01:08:23the full title.
01:08:25At any rate it was money for treatment
01:08:29of HIV in Africa and it was a a gift
01:08:34from the United States and George Bush.
01:08:37Authorized it and made sure that it went
01:08:39through as near as I near as I can tell.
01:08:41So and it's estimated right that
01:08:43that saved 20 million lives a lot.
01:08:47So,
01:08:49so my question is did any of that
01:08:53money go into testing within Africa?
01:08:56That's one question. And then the
01:09:00second question is if we are to ever.
01:09:08Donate. If we are ever to become
01:09:11generous enough again to have a
01:09:13PEPFAR like initiative for other
01:09:16illnesses in low income countries,
01:09:19are you would it would it make sense
01:09:23to you to incorporate the research
01:09:25arm of that into that funding which
01:09:30so I don't know how the PEPFAR
01:09:33spending. Was allocated.
01:09:35But if you do look at trial locations,
01:09:40the trials for HIV are geographically
01:09:44on the country level, the most diverse.
01:09:49So it's it's possible,
01:09:56yeah. So your question raises a
01:09:59really interesting one about whose
01:10:02responsibility it is to fund.
01:10:04Global clinical trials, right.
01:10:05And to ensure that clinical trials
01:10:08are taking place in countries
01:10:09with high disease burden,
01:10:14the FDA is the US, the SPOT trial
01:10:17sponsors and it's an unanswered question.
01:10:20I would say I'd like to see the
01:10:22pharma company just pay for it, right.
01:10:26I think primarily it's
01:10:28their first responsibility.
01:10:29They're the ones profiting
01:10:30off of marketing a product.
01:10:32I'd like to see if that happen first.
01:10:39What what do you think about that?
01:10:41Because if you, if you have a
01:10:43government come in and pay you that,
01:10:45you're just kind of speaking to whose
01:10:47responsibility it is. Exactly. Yeah.
01:10:50No, I I I think whatever we could.
01:10:55Contributions from the Pharmaceutical
01:10:56industry would be great.
01:10:57What how do we incentivize that?
01:10:59How do we build that in?
01:11:00Well, that's what I'm trying to do
01:11:01with the good pharma scorecard, right.
01:11:03So one of the pieces is looking at that's why
01:11:07I teed up the conceptual piece which is well,
01:11:10first the the empirical where
01:11:12are we testing products.
01:11:13The second piece was conceptually where
01:11:15should we be testing products, right.
01:11:17And I hint that I think.
01:11:19That site selection to track the
01:11:21burden of disease on the country
01:11:22level and then the next piece is to
01:11:25go in and see I'm going to find no,
01:11:27but do site selections correlate with
01:11:29disease burden, it's going to be no.
01:11:30And then the and then the next piece
01:11:32is to build it into the good pharma
01:11:34scorecard right to rank companies on
01:11:36whether their site selections are
01:11:38correlating with the burden disease and
01:11:41then to look at countries to see if some
01:11:44countries are better at getting sites.
01:11:46Than others with with high burns
01:11:49of disease and why right barriers
01:11:51and facilitators for hosting trials
01:11:54or barriers and facilitating yeah
01:11:55to selecting certain sites
01:12:03but it but remember just because
01:12:04you have a trial site doesn't mean
01:12:05that the product gets submitted for
01:12:07marketing then it doesn't mean that it's
01:12:09affordable that there's enough supply.
01:12:17I I just wonder about tactics for addressing
01:12:22the lack of representation in trials,
01:12:24because I kind of wonder whose fault
01:12:27it is or who's best situated to fix it
01:12:33is. For example, if if some
01:12:36pharma company has a Pi at Yale.
01:12:38It might just be that the Pi at Yale
01:12:40has a really hard time recruiting
01:12:43a representative number of black
01:12:45patients from the New Haven community.
01:12:47So then whose fault is it that the recruiting
01:12:49is not sufficiently representative?
01:12:52Well, maybe it's the company's
01:12:53fault because they should find
01:12:55PI's in places where that are,
01:12:58where minority communities
01:12:59are more dense on the ground.
01:13:01Maybe that means finding API
01:13:02in in rural areas of the South
01:13:06that are predominantly black.
01:13:08I also was just curious whether you
01:13:10knew whether some of this lack of
01:13:12representation is due to pharma companies
01:13:15relying on disease groups for recruiting,
01:13:18because I sort of strongly
01:13:21suspect disease groups of not
01:13:23being particularly representative
01:13:25of the people with the disease
01:13:28burden because they're largely
01:13:31fundraising vehicles for pharma.
01:13:34So they're probably disproportionately
01:13:36wealthy and therefore,
01:13:37I would guess disproportionately
01:13:39white and so on.
01:13:41Right. And in some cases getting
01:13:43royalty royalties from products.
01:13:45If you think about Cystic
01:13:47Fibrosis Foundation,
01:13:48that's a really interesting model.
01:13:51Yeah, I guess, Steve, I wouldn't look at.
01:13:53So when you say whose fault is it,
01:13:55is that, is that your.
01:13:58Yeah. Yeah, that's right.
01:13:59What's the root of the the problem?
01:14:01So we can strike at it, right.
01:14:04The, the, the roots are so
01:14:07pervasive and so systemic,
01:14:09but it's hard to find a dominant route.
01:14:12And so I think going back to Sarah,
01:14:15Doctor Hall's question is that we need to go,
01:14:17you know,
01:14:17we all need to be doing something right.
01:14:20And so part of it is, as you mentioned,
01:14:22selecting diverse sites on
01:14:25the geographic level,
01:14:27sites where there are diverse
01:14:29patient populations,
01:14:30Yale happens to be one of them,
01:14:31right, which is.
01:14:34Helpful for us making sure
01:14:37that our workforce is diverse,
01:14:38right,
01:14:38so that we're recruiting and
01:14:40retaining A diverse workforce.
01:14:41But that starts you know that's
01:14:43also systemic challenge that
01:14:45starts really on and early on
01:14:47in life and generations passed.
01:14:49So it's the roots are so deep and
01:14:51so multipronged on this challenge.
01:14:53I can't really tell you which
01:14:55route to strike most.
01:14:56You know we have to strike all of them.
01:15:00Or what are all of them?
01:15:02But but we are trying to
01:15:04answer that question
01:15:07with the positive deviant study, right?
01:15:08Seeing that the trials that did get it right,
01:15:10you know for the sponsors who
01:15:11did get some something right,
01:15:13right one measure right,
01:15:13how did they do it?
01:15:14So at least we can start developing
01:15:17generalizable knowledge for best
01:15:18practices that have worked in the past.
01:15:22Which which route would you strike? Site
01:15:26selection seems to be really important.
01:15:29That's a popular one.
01:15:31Yeah, maybe the implementation
01:15:33of decentralized trials and.
01:15:37But that also introduces more inequities,
01:15:39right, The digital divide.
01:15:40But only some people have access
01:15:42to Internet and it's complicated.
01:15:47They just keep coming.
01:15:49No silver bullets. Bring it on. Jack.
01:15:53I'm. I'm fascinated.
01:15:54Well, I'm delighted to hear that.
01:15:57That good performance on your,
01:15:59on your measure correlates with success,
01:16:04if am I interpreting what you said correctly.
01:16:08And so I want to know what how much
01:16:12of that do you think is cause and
01:16:14effect is and you know we when we
01:16:17hear about hospitals that perform
01:16:18well and they do score well on
01:16:21their performance evaluations,
01:16:23they tend to be hospitals that are.
01:16:26Doing well, but they're also hospitals
01:16:30that have that are adequately staffed
01:16:33and they have good cash flows and
01:16:35they are capable of addressing the
01:16:37performance measures and making sure that
01:16:41everything's getting recorded correctly.
01:16:43Is it possible that the that the
01:16:46pharmaceutical companies that are
01:16:48doing well or that are that are seem
01:16:50to be morally superior are actually
01:16:52just able to to address your.
01:16:55Your scorecard better and it I I
01:16:59suppose in a way we don't care if you're
01:17:02leading to moral improvement as long as
01:17:05you're leading to better performance.
01:17:07And so we'll let people just
01:17:10fake it until they make it or.
01:17:15Well, it's a little early to talk
01:17:17about the results of the Alpha study,
01:17:19but I we did control,
01:17:21so did various snapshots.
01:17:23Again it's it's very preliminary,
01:17:25but I held constant for large companies.
01:17:28So just looking at the largest
01:17:30companies by market cap and you
01:17:31still see an outperformance.
01:17:32So in so there you would have
01:17:35controlled for in theory
01:17:38some level of resource
01:17:40resource access to resources.
01:17:42You still see a correlation,
01:17:49yeah, but. But I don't mean
01:17:53to incentivize that companies
01:17:55don't have to do the right
01:17:56thing when it doesn't pay right.
01:17:57We want them to do the right
01:17:58thing no matter what.
01:17:59But it helps and that it's
01:18:01another lever to pull if it's also
01:18:03not going to be more expensive
01:18:05and possibly even profitable.
01:18:08Right? Gentlemen back there, please.
01:18:10Oh wait before you speak,
01:18:12excuse me just one second because
01:18:13it occurs to me there's a disclosure
01:18:15that I should have given here.
01:18:17I am talking about the wonderful work
01:18:18your organization does the Bioethics
01:18:20International scorecard and I actually on
01:18:22the Advisory Board of this organization.
01:18:23So I should disclose that however the the,
01:18:26the payment checks are are
01:18:28still in the mail apparently.
01:18:30So this is a a a volunteer service but
01:18:33just as a disclosure because I didn't say
01:18:35that at the beginning and I should have,
01:18:36I apologize for that Sir.
01:18:38Please go ahead.
01:18:39Not a problem.
01:18:40Thank you for the interesting talk.
01:18:42I think the scorecard is super cool
01:18:45because it's sometimes tough to like
01:18:47translate research into actually
01:18:49changing how organizations and
01:18:51corporations are actually working.
01:18:53And I think it's cool that you've
01:18:55like gotten in and you can sort of add
01:18:58layers to the to what a good score is.
01:19:01But I guess the question is like,
01:19:03what does it take to reach
01:19:06consensus in the bioethics?
01:19:08Community or like,
01:19:09what does it take for you to say
01:19:11this is the next thing that needs
01:19:12to be added to the scorecard?
01:19:14Because it seems like there's a lot
01:19:16of frameworks for evaluating some of
01:19:18these things that aren't entirely like
01:19:19this is the right way versus this.
01:19:21So I'm just curious what you think
01:19:23are the next steps for you to
01:19:24be able to say like,
01:19:25and now here's the next big priority.
01:19:28Yeah,
01:19:29so priority setting, right?
01:19:30Because we'd like to address
01:19:33everything now, but we can't.
01:19:36So there are a couple of factors.
01:19:38So what are the factors that
01:19:40sort of drive decision making?
01:19:42One is practicality,
01:19:43it doesn't mean those are the right drivers.
01:19:45What what can we measure,
01:19:47where can we get data or where do
01:19:50we need to work in the interim to
01:19:52make sure that the data that we
01:19:53can get the data in the future.
01:19:55So for example if you look at the
01:19:57clinical trial diversity measures,
01:19:59they only looked at oncology.
01:20:01Because the CDC publishes publishes
01:20:04the CR database with the cancer
01:20:07incidence data by some demographics.
01:20:09But outside of oncology,
01:20:10it's really hard to get incidence
01:20:13data for conditions by demographics.
01:20:15And so you're right to point out
01:20:18how small steps we have to take and
01:20:20how do we prioritize those steps.
01:20:22So that's why we prioritize oncology
01:20:26part of it was a practical data.
01:20:29Access consideration.
01:20:30It happens to also be major
01:20:33disease burden for for the US.
01:20:37Other considerations are public health goals,
01:20:41ethical imperatives?
01:20:42What data do we already have
01:20:45that we can leverage quickly?
01:20:47What's ripe for change?
01:20:49What's salient?
01:20:50What are people paying attention to?
01:20:51But we have behind all this are is it
01:20:55with a 20 year old dissertation that maps.
01:20:59You know,
01:20:59except for maybe cutting edge things,
01:21:00but there hasn't really been much
01:21:02cutting edge problems that you know,
01:21:04300 pages of things that would be good
01:21:06to to address right to advance patient,
01:21:10public global health and justice
01:21:13for for people around the world.
01:21:15And we're just chipping away at it.
01:21:17So the ordering is,
01:21:20is mostly practical salience,
01:21:23health needs and justice considerations.
01:21:27And resources.
01:21:32So my memory also goes back a long way
01:21:35and I'm remembering when there were a lot
01:21:39of research was not necessarily coming
01:21:41out of funding either by pharma or by
01:21:44government that there was a sense that
01:21:47you needed homogeneity in your subjects
01:21:50because the more variation you had,
01:21:53the harder it was going to be to
01:21:54draw any conclusions. And of course.
01:21:56One easy way to get more homogeneous
01:21:59populations is some of the really
01:22:02egregious examples we have in bioethics
01:22:04from syphilis studies or mental health
01:22:06patients and so on in the conversation,
01:22:10of course, has shifted over those years to
01:22:13say some of this is just not allowable.
01:22:16But there's still a concern, I think,
01:22:18with the sense that you may be
01:22:20doing racial targeting.
01:22:22So I'm wondering about.
01:22:24How some of the ideas have changed
01:22:27and what may help some change more
01:22:30and what directions you would like
01:22:33to see things changing in that may
01:22:36get incorporated into some of the
01:22:39scorecarding or the advocacy work or ways
01:22:42in which we should be doing our studies.
01:22:46Yeah, I think the big changes
01:22:48on that social value principle,
01:22:49right,
01:22:50where we were very permissive in the
01:22:53interpretation where we didn't ask.
01:22:54That justice question of who should
01:22:57be benefiting right we we defined,
01:23:00we justified clinical research if
01:23:02it had a potential to generate
01:23:04generalizable knowledge that could
01:23:06help someone or some populations
01:23:08health and we didn't think about as
01:23:10much whose health and the fairness
01:23:13considerations in there And because of
01:23:17various recent tragedies we've
01:23:21been starting to rightfully.
01:23:25Ask those justice questions.
01:23:29And those justice questions are
01:23:33trumping the old ways of thinking,
01:23:35which, from what I heard,
01:23:36the way you contextualize it and correct
01:23:38me if I didn't interpret this correctly,
01:23:40was that science and this sort of
01:23:44pristine lab experiment was more
01:23:46important than these justice questions.
01:23:49And that balance of science and
01:23:51justice has has changed, is changing.
01:23:54At least it's changing now.
01:23:57And it turns out that that science
01:24:00question may no longer be valid
01:24:02because that that science didn't
01:24:04may not be generalizable to many,
01:24:09if any, you know many people.
01:24:11And so the even the scientific
01:24:14validity of that, that, that.
01:24:17Overly controlled setting is coming
01:24:20into play right And the pushes for real
01:24:23world data and and other ways of of
01:24:26developing knowledge are really strong.
01:24:29We're really far away from
01:24:30using real world data.
01:24:31It's been fun to sort of model what
01:24:33you can and cannot do with it.
01:24:35But I think yeah this sort of
01:24:38reordering and revaluing of of goals
01:24:40is is rightfully taking place more
01:24:42widely than it has in the past.
01:24:49The researchers in
01:24:53the population at large.
01:24:54I'm curious about where you're seeing
01:24:57that change happening and ways in which
01:25:01that can be addressed to help achieve the
01:25:04goals that that you're advocating for.
01:25:06Well, where is it taking place
01:25:07as an empirical question?
01:25:09And I don't have, I like to answer
01:25:11empirical questions with empirical data,
01:25:13which I don't have it at my fingertips.
01:25:16But certainly I can just comment right on,
01:25:18on anecdotally,
01:25:19you see it on the policy level, right.
01:25:21You've seen it over 40 years as sort
01:25:24of growing wealth of policy efforts
01:25:28to target injustices in these areas.
01:25:32You see it in the literature that's
01:25:34getting published more and more studies
01:25:36and focusing on the problems, right.
01:25:38A lot of the studies focus on the
01:25:40problems and now ethicists are at least
01:25:41some of us are starting to look at
01:25:42what does we know there's a problem,
01:25:44what does good look like?
01:25:45Right.
01:25:46And how do we track and measure
01:25:48progress on goals?
01:25:49So I think it's happening on many levels.
01:25:50I think the more interesting
01:25:52question might be where is it not
01:25:54happening that it needs to happen.
01:25:56So I'd have to think about
01:25:59that and get back to you.
01:26:01Thank you. I think that that's our time.
01:26:05This was a fascinating evening.
01:26:06Thank you so much, Doctor Jennifer Miller.