Qualitative Comparative Analysis: Applying to Research on Public Health Interventions

February 14, 2024

Speaker: Lingrui Liu, ScD, MS

Wednesday, October 6, 2021

Information

ID: 11309
To Cite: DCA Citation Guide

Download Transcript

00:01<v Presenter>I'm an Associate Professor of Epidemiology</v>
00:04at the Yale School of Public Health,
00:05as well as the school's Associate Dean
00:07for Diversity, Equity and Inclusion.
00:10It is my pleasure to step in
00:12for Professor Donna Spiegelman today
00:14to introduce our speaker, Dr. Lingrui Liu.
00:18Before I do that though,
00:20I'd like to acknowledge that today's seminar is sponsored
00:22by both the Yale Center for Methods and Implementation
00:25and Prevention Science, or CMIPS
00:28and the Yale Center for Implementation Science.
00:31Based at the Yale School of Health,
00:33CMIPS develops and disseminates
00:35innovative methodological approaches
00:38to address implementation gaps
00:40and improve public health worldwide,
00:43strategically selecting the issues
00:45that carry the greatest burden and hold the greatest promise
00:48for amelioration right now.
00:50If you would like to be informed
00:52about future CMIPS seminars,
00:54please let William Tutle know in the chat
00:57and he will add you to the CMIPS listserv.
01:00Based at the Yale School of Medicine,
01:02YCIS accelerates the late stage translation
01:06of evidence-based treatments, practices and policies
01:09to improve the health
01:10of the residents of New Haven and beyond.
01:13Its Yale Scholars and Implementation Science,
01:15or YSIS program is the training core of the center.
01:19YCIS is funded by a five-year
01:21National Heart, Lung and Blood Institute K-12 award
01:24and is designed to train junior faculty
01:27and postdoctoral fellows
01:28in late stage dissemination and implementation science.
01:33Turning to our speaker today,
01:35Dr. Lingrui Liu is an associate research scientist
01:38in the Department of Health Policy and Management at YSPH
01:42as well as a K-12 EL Scholar Implementation Science
01:45at the Yale Center for Implementation Science
01:48and a fellow at CMIPS.
01:51Her research focuses on healthcare management
01:54and organizational studies, healthcare systems,
01:57quality improvement, patient safety,
02:00decision science and implementation science.
02:03Dr. Liu received has received national recognition
02:07for her work, including awards
02:08from the Academy of Management and Academy Health.
02:12Dr. Liu received her doctorate from Harvard University
02:15in Health Systems, Economics and Decision Science.
02:19The title of her talk today is,
02:20"Qualitative Comparative Analysis:
02:22Applying to Research on Public Health Interventions."
02:26Welcome Ling, over to you.
02:28<v Dr. Liu>Thank you Matt and thank you William,</v>
02:30and also Donna for having me today,
02:34have this opportunity to present
02:38the qualitative comparative analysis,
02:42relative new methodology to public health.
02:46And I consider this opportunity
02:48as a way to discuss with our scholars
02:57and also broader the community,
03:00to explore the utility of this method
03:02in public health, in intervention, evaluative science.
03:10And I am not expecting to use this opportunity
03:14as an education of this method,
03:19like within this 30 or 40 minutes.
03:26And so today, I am going to introduce
03:30this new method to you and also discuss about the literature
03:36and the discussion of the utility of QCA methodology
03:40in implementation sense.
03:41And lastly, I will use empirical study from my work,
03:48set as an example to show a little bit
03:51about my experienced business method.
03:55So what is QCA?
03:59So in one short sentence, it uses Boolean logic
04:03to identify all possible combinations
04:06of variables, conditions.
04:09It's a QCA terminology that influences the outcome,
04:13and also as scholars and we are very curious
04:17about the receptiveness of this matter
04:20and what are the publication opportunities
04:23of using this method.
04:25And the good news,
04:26that it welcomes for growing opportunities on conferences
04:31and also the publications in journals
04:35that have been majorly published in business,
04:39management, political science,
04:42and also is emerging in public health
04:45and health services research journals.
04:47But we also acknowledge the challenges exist,
04:52that's mostly in how to visualize
04:56the findings from the QCA and how to interpret
04:59and how to communicate this non-QCA experts
05:05in our scholarship.
05:11And here, I cite a figure from the latest math book
05:18from Melloo and he is based in Europe
05:24and so his book has summarized
05:29the recent journal articles published on QCA
05:33and you can see that public health is emerging,
05:37but it is still relatively new.
05:42And here, I would like to summarize three main features
05:48of the QC methodology
05:50and I first consider it is a mixed method
05:54that bridges the qualitative and quantitative analysis.
05:58So it's a non-additive and nonlinear method
06:02that is to identify all the combinations
06:04of necessary and sufficient conditions,
06:08the factors for an outcome.
06:11And so for using QCA, it requires the researchers
06:17to have the familiarity of these cases
06:20to have the in-depth knowledge of your cases,
06:25but also it's enables the researchers
06:28to examine the cross-case patterns.
06:31So it acknowledges the diversity and also heterogeneity
06:37of the research study,
06:39with regard that it allows the researchers to identify
06:44what are the different solutions
06:47of different combinations of the conditions
06:51for the occurrence of the outcome of your interest.
06:55And second feature, I think is important to understand QCA
07:00is to assess the sufficient and necessary conditions
07:05for the success or failure of the outcome.
07:08So it's very different
07:09from our conventional statistic influence techniques.
07:15And by looking at the sufficient and necessary conditions
07:21for the success or failure of the outcomes,
07:23it provides researchers or the practitioners the approach
07:31to identify more than one solution
07:33or we call recipe to an outcome.
07:38And also the presence or the absence of these factors
07:42in relation to the other conditions might be key.
07:46And also in QCA study
07:49and the specific factors explain the success
07:53does not imply that their absence would lead to the failure.
07:57And the third feature, I think is important,
08:02is that QCA is ideal, given the nature of this
08:06and the underlying logic of this matter.
08:09It's very ideal
08:11for the small to intermediate sample size research design.
08:16And because it's within this range, like 10 to 15 cases,
08:21there are often too many cases
08:24for researchers to keep all the knowledge
08:26about all the cases,
08:28but too few cases
08:30for most conventional statistical techniques.
08:34And here I share a few important methodology references,
08:40which may be helpful
08:41if some of you are interested to read and explore more.
08:47And so come to our interest,
08:51what is the utility of this method
08:54in implementation science?
08:56And I found two literature
08:58which I found they are very useful for me to understand
09:03what has been done in this area.
09:06And so one is the systematic review by Hanckel
09:10and which is just published is out at BMC Public Health
09:14and the other report paper
09:19and it has been using, apply the QCA
09:23to identify the features of the strategies
09:28related to mental health care
09:30for children and adolescents by the RTI
09:34and it has been published as agency for healthcare
09:39and quality research publication series.
09:43So I think those two are very useful
09:46if you want to read more.
09:50So here, just to summarize,
09:53and in existing literature
09:57so the selection criteria have been used
10:01as published in English and up to December 2019
10:05and there are a total of 27 papers used QCA
10:11in evaluating the public health interventions.
10:15And here are a few domains I list out,
10:20like nutrition, obesity, health equality,
10:24community engagement and also chronic condition management.
10:29And so here, I want to show you
10:32the sample research questions
10:34or rationale for using the QCA
10:37that you may be interested to consider
10:41whether QCA's potential approach for you to use
10:45in your research or analyzing your data.
10:47So the simple research questions,
10:50you can see that QCA also answers questions,
10:54"What combinations of the components
10:57might serve as recipes for success."
10:59Of the outcome or if you are interested to identify
11:04the critical features or characteristics
11:07of the implementation program
11:10that leads to the successful implementation outcomes
11:15or if you are interested to identify
11:19what are the necessary or sufficient conditions or factors
11:24that are key to the implementation
11:28of this public health intervention.
11:31And also I want to say the QCA is mostly used
11:39in the description or explanation studies.
11:43So in the description studies,
11:46it's very straightforward QCA,
11:49you can use QCA to summarize the patterns
11:52across your cases
11:56or used in explanation studies that help you test
12:02your existing hypothesis and choose to test out
12:07whether your empirical cases can be reflected
12:10by some or any combinations of the factors
12:16from this existing theory.
12:19And here is the QCA research cycle.
12:23So it starts to pose the research problem
12:29and ask the research question
12:31and to figure out the scope of your research question
12:35and then use the existing theories
12:40or like the empirical evidence,
12:44to consider to select which cases into your study
12:51and then you will need to select the conditions,
12:54like conditions are the factors,
12:56you can under consider those are the factors
13:00and then the key steps are to calibrate the data
13:06and to conduct the analysis
13:14to identify the necessary conditions
13:17and also the sufficient conditions.
13:20And the full process is an iterative process
13:25that you will need to come forth and back
13:29and to adjust with the scope of your research question
13:36and also the selection of cases and the thresholds
13:42and the decision rules for calibrating your data sets.
13:49And so the strengths and weakness
13:51of the QCA implementation science has been discussed
13:57in a way that it provides a systematic approach
14:01for understanding the mechanisms that work
14:03in implementation across the context.
14:07And the weakness however have been reported
14:09related to the data availability limitation,
14:12especially on ineffective interventions
14:17And the software packages are evolving,
14:23still in development,
14:24but a few packages are major and ready for use
14:30and you can go to this website for a full list,
14:35but for the major softwares have been developed
14:39as the FSQCA, QCA software
14:43and also there are a few packages developed
14:46on our environment that are for use.
14:51And here I want to just show a few examples
14:55of visualizing the QCA findings.
14:58And so, in the existing literature
15:02I found that the Venn diagram and also the chart table
15:06have been mostly used and again it's evolving
15:12and so scholars and researchers are still exploring
15:18what are the most efficient ways to communicate
15:24with our audience about the findings from the QCA study.
15:30So here I want to use one example study from my work
15:38that I collaborated with my colleagues
15:43and we use data from primary care practices
15:46to explore the system features of primary care practice
15:50that promotes better provider experience.
15:53And this work had been variously published
15:57at Academy of Management
15:59and also Healthcare Management Review.
16:05So in this study
16:08we focus on the providers in primary care practices
16:12and we know that the primary care providers also experience
16:16low rates of clinical work satisfaction
16:19and high rates of burnout
16:21and the poor satisfaction may adversely affect the quality
16:26of care they deliver to their patients
16:29and adversely related to patient outcomes
16:33and patient experience.
16:35And what have been not exams or studies much
16:41is what are the system level features
16:45affecting the provider's satisfaction
16:47in their clinical work practice?
16:51So in this study we asked a research question,
16:55"Which system features and in what combinations
16:58of this features can help to improve primary care provider's
17:03clinical work satisfaction?"
17:06And this study was conducted in collaboration
17:11with 19 Harvard affiliate primary care practices
17:15and we surveyed a total of 19 managers
17:19and a total of 854 primary care providers
17:25completed the survey.
17:27And for the managers of the survey and interview,
17:31one manager of each of these 19 primary care practice.
17:38Our hypothesis is to look
17:43at our automated outcome of increase, the system outcome,
17:47providers clinical work satisfaction,
17:51and this outcome is positively related,
17:56as the system features of primary care practices
18:00that include the team dynamics
18:03and provider perceptions of safety culture
18:06and also the care coordination among the providers
18:10to their patient care.
18:13And further, the hypothesis is that the enabling functions
18:20of these primary care practices,
18:22including operational functions,
18:25that goes into eight domains,
18:27and also the health information technology HIT functions
18:33are positively related with each of our system features,
18:39the overall team dynamics,
18:42the provider perceptions of safety culture
18:45and also the care coordination among providers.
18:50And within the operational care process functionality
18:54of the practices, we categorize into eight domains.
18:59So including appointment and referral system
19:03for high risk patients and also for routine patients,
19:07abnormal test result management,
19:10cancer screening for high risk patients
19:12and also for routine patients,
19:14patient center care, patient safety
19:17and care transitions across the primary care practice
19:22and emergency departments or the hospitals
19:25or the other specialist departments
19:29and the data, so we use the self-assessment service
19:34for primary care providers variables,
19:40including the clinical work satisfaction and team dynamics,
19:45provider perceptions of safety culture and care coordination
19:49among the providers towards patient care.
19:52And for enabling functions,
19:55they surveyed and interviewed the managers of the practice
20:00on a total of eight domains
20:02of the operational care process functions
20:05and also 42 items of HIT functions.
20:10And so, the method is we use the QCA
20:14and again it's based on Boolean logic.
20:17Here are just a few examples of the logic
20:22and logic or, or negation knots
20:25that is used in Boolean logic.
20:28And so, we use QCA to compare the cases
20:32to identify the combinations
20:34of necessary and sufficient conditions,
20:37the variables that trigger the outcome.
20:41And in QCA the key step is to construct the Truth Table.
20:47And here is an example from one hypothesis,
20:50of testing one hypothesis in our study.
20:55And so you can see that this is a table
20:58that includes eight rows
21:00and so the table is a two case power table
21:05and we have three explanatory variables here
21:08and one outcome.
21:09So we have a total of eight rows
21:12and each row we can consider as a recipe
21:15that's a combination of the logically possible conditions.
21:21And the one indicates the presence
21:24of this factor in this recipe
21:28and the zero indicates the absence of this factor
21:32in this recipe.
21:35Then the other main methodology
21:39is to use the Quine-McCluskey algorithm
21:44to reduce the roles of the Truth Tables to bring equations
21:50and to minimize the combinations
21:52which yields the prime recipes.
21:56And QCA uses two goodness-of-fit statistics
22:00and what is consistency?
22:03So it's range from zero to one
22:07and it indicates the strength of association
22:10between the conditions and outcome
22:13and the coverage also ranges from zero to one
22:17and it indicates the proportion of the cases
22:22that are covered in a specific recipe.
22:24So here in our study,
22:26it indicates the proportion of the practice sites
22:30that have a specific bundle.
22:33Here is a example of the consistency and coverage scores
22:38from testing this hypothesis in our study.
22:43And here I want to talk again
22:49about a few main features of QCA method.
22:54So it identifies the combinations of the conditions,
22:57capable of yielding the same outcome
23:01and therefore it can be multiple pathways
23:05and the outcome and non-outcome
23:07may require different explanations.
23:09So it is important that you define the outcome
23:13in your testing,
23:15for example in the example which I just presented,
23:19is the outcome we define as grade level satisfaction
23:25among the primary care providers.
23:30So the non-outcome is the lower level satisfaction
23:35among the primary care providers
23:37and so it requests the researchers to test the separate sets
23:43of the combinations of the factors,
23:45leads to the outcome or the non-outcome.
23:49So it contrasts with the next effects thinking
23:54that usually have been applied
23:56in conventional statistical techniques.
24:00And in this study using the QCA,
24:03the rationale for choosing QCA is that it enable us
24:07to identify the system level characteristics
24:10that influence the provider experience.
24:16And the findings, so the first task,
24:21the contribution of each system features
24:25to our system outcomes.
24:28The team dynamics provide a perception
24:31of safety culture and peer coordination among providers.
24:34Each of those three system features
24:38with the outcome of increase,
24:40the raised clinical work satisfaction
24:43and we can see
24:44that it's yields very high consistency and modus coverage.
24:50And then we test the bundles of these system features,
24:56like team dynamics and provider perceptions
24:59of safety culture together,
25:02you can see it yield very high consistency
25:05and also high coverage
25:08and also the provider perceptions of safety culture
25:12and peer coordination among providers,
25:15you can see the consistency remains very high
25:18but the coverage drops
25:21and also we test all of them together
25:24like the set relations of the three system features
25:28based the outcome.
25:30And you can also see that the consistency remains high,
25:34but the coverage is relatively low, it's only 0.32.
25:41And so, we are expanding the analysis
25:47to include our enabling functions
25:50of the primary care practices
25:53and in our argument, in our final results
25:57and we identify there are three key components
26:02of the operational care process functions.
26:07One is the number three, abnormal test result management
26:11and the number five,
26:13cancel screening for high risk patients.
26:16And number eight is the care transitions
26:19across the primary care practice
26:22and emergency departments or hospitals
26:26and plus this HIT functions,
26:30that they together are the core factors
26:36consisting of the solutions which would yield to the outcome
26:41of our interest, like the strong team dynamics
26:47and also greater level provider perceptions
26:52of safety culture.
26:58So to interpret what we have found in the QCA analysis.
27:08So, "Favorable team dynamics
27:10combined with a strong safety culture contribute most
27:14to the greater clinical work satisfaction."
27:18And, "Provider-perceived safety culture
27:21acts as a core sufficient condition
27:23that presents in both recipes,
27:26yielding for PCP's great clinical work satisfaction."
27:30And, "For the most empirical appliances,"
27:34that means from our empirical cases,
27:38"A strong safety culture is not sufficient on its own
27:44and practice also needs to to create and also implement
27:52the highly functioning teams."
27:54Like to encourage them to foster the strong team dynamics,
28:00visiting the primary care practices.
28:04And also our findings indicate the, "HIT functionality alone
28:10is not sufficient to achieve the desired outcomes."
28:15This is occurring a lot
28:16with what we know from the literature,
28:20because a lot of literature found
28:23that HIT generates a few benefits,
28:27but also to overly emphasize on the utility of HIT
28:34may bring a few adverse effects,
28:37like to increase the volume of the workloads to providers
28:44and contribute to their burnout issues.
28:48So from our analysis, QCA analysis,
28:52it identifies that the HIT functionality
28:56is a core component, but alone, it's not sufficient
29:00to help our providers
29:02to improve their clinical satisfaction.
29:05And also look
29:06at the operational care process functionalities
29:09and we found that the common features of the three factors
29:16which we identify,
29:18they represent it's importance to enable the functions,
29:24identify urgent or complex acute illness
29:29and also request the collaborations
29:32across institutional settings.
29:35And this served as the key factors
29:38within the operational care process functionalities
29:42that can enable our providers
29:45to achieve better team dynamics
29:47and also their perception of safety culture.
29:54And now I want to discuss
30:00a little bit more about the practice implications
30:03of using the QCA in our study.
30:06I see the most attractive part of using QCA
30:12is that it helps to generate the message
30:15that can be very useful and practical for our managers
30:22or the practitioners, in-house systems, for them to use
30:27because usually they are more interested
30:31what are the solution pathways
30:34that we can implement in our systems,
30:37rather than increase on the net effects
30:41of each individual factors, effect on the outcome.
30:46So using QCA, it presents the multiple solution pathways
30:55for our practitioners,
30:59what are the most key factors you can focus
31:03and, or prioritize, in order to achieve the outcome
31:10of your implementation outcome or outcome interest.
31:15So in our study,
31:17because the real obstacles in primary care practices
31:22is usually not possible to enable
31:25or to invest the resource to improve all the conditions
31:30or all the factors in our primary care delivery system.
31:35And so to identify the bundles
31:39can be the prioritized targets for our managers
31:43to emphasize if they want to improve
31:48their provider's work satisfaction.
31:51And our study also highlights the human-centric nature
31:55of the physician clinical work satisfaction,
31:58like how HIT is important as a core components,
32:03but it has to work, function,
32:07with the other key factors together
32:11and it informs the need for non-regulatory strategies.
32:17And we also acknowledge there are a few limitations.
32:21So by using QCA, we're not able to generate
32:25the causal claims.
32:26And also in QCA, one approach,
32:30like to calibrate the data
32:31actually is to refer to the existing empirical evidence,
32:38but because the QCA is relatively new for public health,
32:42evaluative science or health services research,
32:47so external standards or empirical evidence that published
32:53for data calibration are not yet available or established.
33:02And also QCA method may be prone to the type one errors.
33:07And I want to stop at here
33:10and also welcome for a few questions
33:13and if you have data sets
33:16that you think may consider QCA as an approach
33:22to analyze your data to answer the research question,
33:28what QCA can help to work best.
33:34And so welcome to Rachel and I'm very passionate
33:41on exploring computer exploration of this method
33:45in implementation science studies.
33:48Thank you very much.
34:01Any questions?
34:05<v Donna>Yeah, hi Lingrui, this is Donna.</v>
34:08<v Dr. Liu>Hi Donna.</v>
34:08<v Donna>Hi, thank you for the excellent and clear talk.</v>
34:11I really appreciate it.
34:13I have a couple of questions.
34:14So this one is sort of a point of information.
34:17I came in a little late unfortunately
34:19and I didn't catch those two measures.
34:22What is consistency?
34:23And there was something else,
34:25another measure that was appearing
34:27on a number of your slides?
34:29<v Dr. Liu>Yeah, so one is consistency</v>
34:32and one is coverage.
34:33So consistency indicates the strength,
34:36you can consider as the p value,
34:42it's very like the p value in regressions
34:47and it indicates the strengths of the association-
34:52<v Donna>I'm sorry, the strength of the association</v>
34:55of what with what?
34:56<v Dr. Liu>Of explanatory.</v>
34:58I'm trying to not use the terminology from our regression
35:06or statistics to explain the QCA,
35:09because they are definitely two different methods.
35:14But because of my experience of presenting this method,
35:19I know that usually it's helpful
35:21to borrow some terminology
35:24from the conventional statistic analysis
35:28to interpret the terminology in QCA.
35:33And so back to your question,
35:39so consistency indicates the strength of the relationship
35:44between the explanatory variables and the outcome.
35:49So actually in QCA,
35:50because it's based in Boolean logic,
35:53it's actually not, variables is the conditions.
35:57So it's identifying
35:59what are the necessary and sufficient conditions
36:03that would lead to the occurrence of the outcome.
36:08And the other measure is the coverage,
36:15it also ranges from zero to one.
36:18It tells the proportion of, for example, in my studies,
36:23the proportion of the practice sites
36:26which have the specific bundle,
36:30so it tells the empirical appliance.
36:34Because in QCA, you first identify given your conditions
36:39and also the case data, you could identify
36:44what are all the logically possible solutions,
36:48but logically possible solutions are not all applied
36:53in your empirical cases.
36:56So coverage tells within your empirical cases,
37:01what the proportion of your cases have a specific bundle
37:08or a specific solution, I hope it helps!
37:14<v Donna>Sort of, well I had a couple of other questions,</v>
37:17but let's see if other people have questions first.
37:32<v Luke>Thanks for the great talk.</v>
37:34I have a question about the early part
37:36of the thematic analysis part.
37:38Is that different in qualitative comparative analysis
37:40than it would be,
37:41maybe with other more traditional techniques
37:43or does it sort of operate similarly?
37:45Obviously you're applying these codes
37:49to what you're learning from the sites.
37:55<v Dr. Liu>Sorry, I missed the first part of your question.</v>
38:01<v Luke>I'm asking a little bit</v>
38:02about how you code these themes
38:05from the data that you're getting from the participants,
38:08is that different than you do
38:09in traditional qualitative, say, thematic analysis
38:12or does it operate according to similar rules?
38:17<v Dr. Liu>This is a great question.</v>
38:20Yeah, actually this touches up on the key part
38:24of the QCA analysis.
38:30Given my experience working with this method,
38:33I consider, it's kind of a build up
38:38on the conventional qualitative or quantitative analysis,
38:44that you have a few cases
38:48and then you will apply the conventional qualitative coding
38:54to your data, for example you have interview data
38:59and also then the additional steps,
39:03you need to calibrate your data to re-skill your data
39:11into membership range,
39:15just the two in very simple words is from zero to one,
39:19like rescale your data into zero to one.
39:24And there are two approaches in QCA,
39:27one is crisp QCA and one's fuzzy sets QCA.
39:33So basically you set up, you need to discuss this,
39:38the empirical experts
39:40and also who has the knowledge about the cases
39:45to decide what are the thresholds to be used
39:50to rescale your data sets,
39:53like the codes used to calibrate your data sets
39:59into the range of zero to one.
40:03So for example, you have to define
40:08what are the three thresholds you need to use,
40:14to calibrate the outcome of grades level satisfaction
40:21among primary care providers.
40:24So these surveys,
40:27the original survey uses zero to five scores,
40:31but in QCA there are a few decision rules
40:39like consider the statistical characteristic of your data
40:44and also refer to the existing empirical evidence
40:51and also your knowledge,
40:54the researcher's knowledge about your data, your cases.
40:59And together you decide what are the decision rules
41:03to set up the thresholds to rescale the data sets,
41:10rescale the data on the outcome variable
41:14into the zero to one.
41:16So for each of your variable
41:18you have the same principles of decision rules,
41:28which would lead to yields into different thresholds,
41:34for each of variables to be rescaled into that range,
41:39like zero to one range.
41:45Is that helpful?
41:47<v Mona>I had a question,</v>
41:48somewhat related to what Luke asked
41:50and maybe my question will also kind of get into more depth
41:53around this issue.
41:53So it sounded like what you were just describing
41:55was the process of defining your outcome
41:58of sort of clinician satisfaction
42:01and defining how you're gonna take something
42:04that's more continuous score or continuous measure
42:07into a binary outcome
42:09where you could do something like QCA,
42:11on the other side of the predictors or the factors
42:15that you are associating with satisfaction,
42:18it sounds like in this study you were using a survey,
42:21potentially with some validated measures
42:23of certain factors.
42:25Related to Luke's question,
42:26I feel like I've seen some presentations of QCA
42:29where they've used qualitative interviews or focus groups
42:33and they've sort of coded,
42:36using standard qualitative methods,
42:38coded the outcomes or the factors
42:43and then kind of used that group consensus
42:46to sort of translate that coding into quantitative,
42:49I wondered if you could talk to us more about that,
42:51'cause I feel like QCA
42:53to me has a lot of potential in mixed methods approaches
42:56as a way to formalize that hypothesis generation process
43:01that you so nicely kind of displayed here.
43:04So I'd love to have you talk a little bit more
43:06about the different applications of QCA in surveys
43:09versus more pure qualitative interviews
43:12and open-ended responses.
43:14<v Dr. Liu>Yeah, and this is a great question.</v>
43:17So from my understanding of this matter,
43:21I consider this is a mixed matter
43:24and yeah it is true that it can be case oriented QCA
43:32or variable oriented QCA,
43:34but I understand in a way, that's the nature of the case
43:39or variable is the same thing.
43:45I mean, ideally you can design the data collection after,
43:51as I show like in the reference,
43:57like the research starts phase,
43:59you have the research problem,
44:00you have approximate research question you want to explore
44:06and you refer to the existing theory
44:09to guide you post the hypothesis
44:13and to guide your data collection,
44:17to help with your data design.
44:20And you apply the QCA to analyze these data sets.
44:26But in practice, like for example in our study,
44:30and we first have the service completed in the whole program
44:39and then we figure out our research question,
44:44it's more interesting at the system level,
44:47at the primary care practice level.
44:50And we were very curious
44:53to explore how QCA can help us to answer this question.
44:59So we have our data already completed
45:05and the compliments with the interviews,
45:07like the small size interview,
45:11the qualitative interview for the manager
45:15of each primary care practices,
45:18but ideally, if you can apply the QCA,
45:26consider to use this method
45:28at the stage of asking your research question
45:31and write your proposal and to combine the implements,
45:42the way how to design your data collection,
45:48either the service or qualitative interviews
45:52in your data collection,
45:55in the way that the QCA would need,
45:58like the qualitative data or quantitative data.
46:04So that's one limitation of our study
46:09that we were not able to confirm that we want to use the QCA
46:17and then we use the QCA framework
46:21to guide our data collection.
46:23And so now I have a project
46:27that I work with my here and Donna
46:32and also our colleagues in China
46:34that we now already proposed the research questions
46:38as the hospital level or like the organizational level
46:43and we will use the QCA frameworks
46:47to help guide our data collection.
46:50So that will solve a few issues that may come out
46:56as the limitations of the study.
46:59And also another thought related to your question
47:03about using qualitative and quantitative data
47:07in the QCA analysis,
47:12so in my study, the majority of my data is the survey data.
47:18But from my study of this method,
47:21is that for working with QCA in the qualitative data
47:28is that after using the traditional qualitative coding
47:34you will needs to take additional steps,
47:39to calibrate the data into scale of zero to one
47:44to indicate the extent to which of your variable,
47:51from the lower membership to higher membership,
47:56you can understand the way the lower performance
47:58of this variable to the higher performance
48:01of this variable in this study.
48:07Yeah, did I answer your question?
48:11<v Mona>I think so.</v>
48:12Somehow getting group consensus, in terms of quantifying,
48:19almost like labeling of a variable.
48:21So for example, I'm really thinking about
48:24how much this might be of relevance to Leslie,
48:27the work you've done in positive deviant studies,
48:29sort of identifying sites or organizations
48:33that are really excelling,
48:34or individuals that are really excelling
48:36and the sites that maybe are not.
48:38And then using QCA to sort of label
48:41some of the factors that might be potential drivers.
48:44And I think Ling what I'm hearing you say
48:46is that you would have to assign some numerical scale
48:50to those factors, in terms of their presence or absence
48:55and then maybe ranges along that scale.
48:59<v Dr. Liu>Yeah, yeah.</v>
49:01<v Donna>Yeah, I am still mystified by this</v>
49:03and I've heard Ling talk about it,
49:05maybe a half a dozen times.
49:06I really wanna learn and understand, I really do
49:10and, you know, each time I get a little closer,
49:12but in this instance,
49:13the calibration to me feels really daunting.
49:17Like as the analog and a qualitative data set,
49:20if we imagine, those of us,
49:21there's a bunch on the panel here who do qualitative work.
49:24You know, if you're in a large group of coders,
49:28getting consensus on a construct can be really hard.
49:31Just even what is this thing?
49:34And so then to have to parse that even further
49:38to say like, yes, no, it exists.
49:42If you're imagining, I don't know, some intangible quality,
49:46if in our work, if we're looking at organizations
49:48and the way they behave and dimension of culture.
49:52So I think how this might work in these small case studies,
49:56these positive deviant studies
49:58where you may have 10 or 12 organizations
50:00or units of analysis,
50:02would be having to move
50:04from not only consensus around coding,
50:07so how do you interpret
50:08a particular piece of qualitative data,
50:11but then this fuzzy piece, like where's the boundary?
50:16Is it a yes or no?
50:17Is it a leadership engagement?
50:21Is it there, yes or no?
50:23So that to me feels daunting.
50:25But if one could accomplish that
50:27in the coding of the narrative textual data,
50:31it seems like there's huge potential to look differently
50:33at combinations of patterns,
50:35which to me, continues to be the takeaway here,
50:38trying to distill
50:40through many, many, many combinations of variables,
50:42if we look at organizational culture measures,
50:45a hundred variables,
50:47how do you find the right combination of the six
50:51that are gonna get you the farthest,
50:52if you're somebody who needs to intervene,
50:55if you're trying to intervene organizationally.
50:57But yeah, I'm on a learning curve, that's for sure.
51:01But I could see Mona, how it might help
51:02in those kinds of designs.
51:05But there's a lot of work
51:06on the qualitative interpretation side, I think.
51:10<v Dr. Liu>Thank you for your insights and comments Mona.</v>
51:15Yeah, at the beginning I said
51:17this is not a educational workshop about the method
51:21and I consider this a forum where we can discuss
51:26and to explore further
51:30as I present, display, methods relatively new
51:35to probably health and evaluative science.
51:40And so I consider this an opportunity,
51:42I can introduce the method to our broader audience
51:46and if you have more data set,
51:48or you have a similar research question
51:51that you think are similar to this
51:54and you may consider to use the QCA
51:58to help with your analysis.
51:59And as you see the years, like they're pretty new,
52:02like recent, three or four years
52:07and also have been most conducted
52:09in high income country settings.
52:12So I see there is a huge room
52:16for implementation science scholars
52:19and practitioners to explore.
52:25And I'm also on the learning curve!
52:27(Dr. Liu Laughing)
52:28And back to one point, lastly mentioned
52:32about how to select the case
52:35and also select the variables,
52:37from my experience and the study of the method
52:40is like, now I think it's important
52:44you have the hypothesis from existing theory,
52:51to guide you.
52:52So it's not possible to include all the factors
52:56of relevance in your analysis,
53:05like one key step is you use the calibrated data
53:10to construct the Truth Table, the two two case power table.
53:15So for example, you have three conditions
53:19or three explanatory variables,
53:21you will have eight rows.
53:23Think about this way,
53:24you have eight explanatory variables,
53:27you will have over 200 rows,
53:31it's eight multiplied by eight.
53:38So you have to limit, what are the key variables
53:45extracting from existing empirical knowledge
53:50or the theory to guide you,
53:54select the case and also variables.
53:56And I think it's important to also keep in mind the outcome.
54:08I think yeah, definition of your variables
54:11and also calibrating your data are very important,
54:17because I consider it's a way to help you to summarize
54:23or describe the patterns of the relations
54:28between your condition variables
54:31and your outcome variable, yeah.
54:34(Dr. Liu Laughing)
54:35<v Donna>So Ling let me ask</v>
54:36what might be the last question
54:38or it shouldn't even be asked,
54:40'cause people probably have to wrap up and go.
54:42But to me, like the two to the K table
54:46that's I don't wanna say just,
54:48but that seems to be very similar if not identical
54:51to what happens in a full factorial design.
54:55And then once you have the factorial design
54:58and you have all the factors,
55:00there are different approaches to kind of using regression
55:03and variable selection, even machine learning
55:06to try to pick what main effects there might be
55:10and combinations and whole packages and so forth.
55:14That might be the most effective with respect to an outcome.
55:18So I'm not sure how this is different from that,
55:21once you get to the quantitative side,
55:23there probably isn't time to say
55:25and maybe we just need to look more carefully
55:27and maybe it is very similar
55:29and people get to similar spots
55:34from different starting points.
55:37But I'm wondering if this analysis is done
55:40after an intervention is conducted
55:43or is this observational research
55:45to try to develop the intervention
55:48and figure out which two to the K combinations
55:51are the best ones to test now in a randomized trial?
55:55'Cause all of this is also very close
55:57to the MOST design of Linda Collins.
56:01But I don't know, maybe it's just some comments
56:04or food for thought
56:05and we probably should let people go
56:07and maybe Ling, we can talk about it some other time?
56:10<v Dr. Liu>Yeah, and I just agree with your comment.</v>
56:14I'd say now the QCA scholarship,
56:17they are discussing about applying this method
56:20to longitudinal, like large sample sets
56:23and also combine with the techniques
56:30from the conventional statistics,
56:32like the machine learning, like that's what you mentioned.
56:35But that is like too new.
56:38I think that probably, at this point,
56:44probably we want to start from the beginning
56:47to understand this method
56:48and also encourage some exploration
56:51of the utility of this matter
56:54in our implementation science projects.
56:58But yeah, a lot of insights
57:02and comments are very helpful today
57:05from our audience, yeah, thank you.
57:11I'm reading the message on charts.
57:14(Dr. Liu Laughing)
57:16<v Donna>Well, I don't know if mayor, maybe he has left,</v>
57:20but I can as sort of a-co convener thank Ling
57:23for this very interesting presentation
57:26and thank everybody for participating.
57:28And yeah, we'll look forward to further discussions
57:31about this and looking at the relationship
57:33between these different approaches in implementation science
57:37to building complex multilevel interventions
57:40that are effective and cost effective.
57:43So thanks everyone and bye-bye.
57:45<v Dr. Liu>Thank you, bye!</v>