PELC: "Questionnaires in Medical Research and Practice: Selecting Valid Instruments" with Marney White
January 03, 2024Information
PELC 12.18.23: "Questionnaires in Medical Research and Practice: Selecting Valid Instruments" with Marney White
ID11145
To CiteDCA Citation Guide
- 00:00So you don't have to do that.
- 00:02I it's I'm fine introducing myself.
- 00:04It's like if you'd like to,
- 00:06but otherwise I certainly will do a
- 00:09little bit of a more formal intro.
- 00:14I've got a lot to get through in a brief
- 00:16time. People can go to my web page and
- 00:17probably get anything they need to know.
- 00:34I like your questions
- 00:45well as everybody's.
- 00:46Anybody who's just joining us,
- 00:49please feel free to start the
- 00:52survey that Marni with the QR code.
- 00:55I'm just going to welcome everybody.
- 00:57It's our last educational
- 00:59learning community for the year.
- 01:02We'll be starting fresh
- 01:04and strong in January.
- 01:06It is really my pleasure.
- 01:08And Marni, I'll only take 30 seconds
- 01:10since I know you have a lot to go
- 01:12through and I could not possibly get
- 01:14through your whole CV in 30 seconds.
- 01:17So I'm actually really thrilled
- 01:19to have Marni with us.
- 01:22When I asked the let's see,
- 01:25I guess it was the MHSMED,
- 01:27some of our faculty about who they
- 01:31would recommend as somebody to
- 01:33talk about surveys and the process
- 01:36of obtaining validity evidence
- 01:37and who might be a good teacher.
- 01:40And resoundingly, Marty Marty's name came up.
- 01:44And so,
- 01:45just to give you a little bit of background,
- 01:48Marty got her PhD in psychology at LSU,
- 01:51and then her Ms.
- 01:52and chronic disease epidemiology at
- 01:54the Yale School of Public Health.
- 01:56And then she was in the department
- 01:59of Psychiatry as a post doc
- 02:01on eating disorders research,
- 02:03where she did quite a bit of work.
- 02:05And then she gradually,
- 02:07well, not gradually,
- 02:09but became professor of psychiatry as
- 02:14well as social behavioural sciences
- 02:17at the Yale School of Public Health.
- 02:19She's director of online education
- 02:22and social and behavioral sciences,
- 02:24core faculty of the National
- 02:27Clinical Clinician Scholars Program,
- 02:29Track Director of Critical Topics
- 02:31and Public Health,
- 02:32which is the online executive MPH program.
- 02:35She's been teacher of the year at the
- 02:37Yale School of Public Health twice,
- 02:39multiple grants and, let's see,
- 02:41over 170 publications.
- 02:42But as I said,
- 02:44the most salient reason for the
- 02:46invitation was I asked who was
- 02:48the best teacher you've ever
- 02:49had add in this area.
- 02:51And as I said, Barney, your name came up.
- 02:53So thanks so much for joining us today.
- 02:56Thank you so much for having me.
- 02:57It was a really nice introduction.
- 02:59Thank you anyway. Thank you.
- 03:02I appreciate that. Very nice.
- 03:03So I do teach a course or recently retired
- 03:06a course actually at the Yale School
- 03:08of Public Health called Questionnaire
- 03:10Development and Psychometrics.
- 03:12And I'll be giving you all today a
- 03:15very crash course overview in that
- 03:17that I hope will be very relevant
- 03:19to your own research endeavours.
- 03:22I find that, yeah.
- 03:26So I I am by original training
- 03:27a clinical psychologist,
- 03:29secondary training and epidemiologist
- 03:30and the Yale School of Public Health.
- 03:32Several years ago,
- 03:33when I was still a junior faculty
- 03:35member in the Department of Psychiatry,
- 03:38I had been cross trained in public health
- 03:41and epidemiology as part of my key award.
- 03:43And after I finished that,
- 03:46YSPH asked me if I would develop
- 03:48a course on constructing valid
- 03:49surveys because it was a gap in
- 03:52the curriculum in that department.
- 03:54I did do so and then gradually
- 03:56started to teach more and then about
- 03:58five years ago shifted my primary
- 03:59appointment over to the School of
- 04:01Public Health where I now I'm on
- 04:03the educator track and I'm really
- 04:05enjoying the opportunity to teach
- 04:07scholars at YSPH and Yale College
- 04:10and the School of Medicine keeps
- 04:12me very entertained and and and
- 04:15engaged in a bunch of different
- 04:17topics and research endeavours
- 04:19with with academics from all over.
- 04:22It's pretty cool.
- 04:22But none of it was by design.
- 04:25It was just sort of the way this
- 04:27kind of path happened.
- 04:28But going back to the roots in psychology,
- 04:30what a lot of people don't know
- 04:32about psychologists is that before we
- 04:34got into this business of treating
- 04:36people and becoming clinicians,
- 04:39we were really about evaluation
- 04:41and assessment.
- 04:42And so a great deal of the
- 04:43training in psychology,
- 04:44even going back to my first
- 04:46master's degree in psychology,
- 04:48was around assessment and learning
- 04:50how to ask questions and try to
- 04:52and identify sources of of bias.
- 04:54And I think I've taken somewhere on
- 04:57the order of 10 or so graduate courses
- 05:01in assessment or psychometrics.
- 05:03That's really what we as the field do,
- 05:06do as our primary foundational
- 05:10knowledge base.
- 05:11So to be able to extend that into
- 05:14medical research and public health
- 05:16research is a pretty neat opportunity.
- 05:19But I I so here's what we're going
- 05:22to try to do today is to teach you
- 05:25what it means to evaluate self
- 05:28report questionnaires,
- 05:28primarily what their their main purposes are,
- 05:31identify them.
- 05:32Identify their strengths and weaknesses.
- 05:35Know what is meant by psychometric
- 05:38criteria to so that it's a very kind
- 05:41of cut and dry process and select
- 05:44the best measures for your research.
- 05:46Because we are pressed on time,
- 05:48I am not going to get much
- 05:50into questionnaire development,
- 05:51but rather best practices for
- 05:54using existing surveys.
- 05:55And the reason for this is because
- 05:58that is an entire other research area.
- 06:02It costs money,
- 06:03It is time consuming.
- 06:05You'll need a grant just
- 06:06or you know you'll need
- 06:08funding for sure,
- 06:09but you'll probably need about
- 06:10two years to start from scratch.
- 06:13So if there's something out there
- 06:15that you can use that is relevant,
- 06:18that is what I would highly recommend.
- 06:20Unless you're wanting psychometrics
- 06:21to be your research area,
- 06:23as it has been for me throughout my career,
- 06:26kind of by accident,
- 06:28and but it's it's what it is.
- 06:31So I want to teach you how to find
- 06:33these well established questionnaires
- 06:35that I mentioned here and then
- 06:38and the worst case scenario.
- 06:40If there's something out there
- 06:41that looks close to what you need,
- 06:43but it hasn't been done in your particular
- 06:46patient population or research focus,
- 06:47you might how to go about adapting
- 06:50that survey for use in your own work.
- 06:53So when we're talking about
- 06:55selecting A measurement instrument,
- 06:56and for the most part I'm going to be
- 06:58talking about self report questionnaires,
- 07:00the kinds of things that you might
- 07:01hand over to patients or colleagues
- 07:03or community members and say,
- 07:04what is your take on this?
- 07:06You know you're wanting a group of
- 07:08individuals to complete this measure.
- 07:10It could be a screening measure,
- 07:11it could be an assessment or
- 07:12a knowledge based survey.
- 07:13Many times we're actually talking
- 07:16about something that is unobservable.
- 07:18You know,
- 07:19we don't have a lab value
- 07:21to correspond with this.
- 07:23There might be some cases where
- 07:24that that that might happen.
- 07:26There might be cases where we'd
- 07:28have a lab value for a blood test,
- 07:30but it's very, very expensive and
- 07:31our screening tool which is self report,
- 07:34might be adequate.
- 07:35You know we might have a a high correlation
- 07:38at .8 or .9 between our self report
- 07:41measure whether it be pain impairment,
- 07:44subjective interpretation of symptoms,
- 07:47whatever.
- 07:47And it's going to be cheaper to ask
- 07:49people that than it will be to actually
- 07:52do a full physical examination.
- 07:54So we find ourselves kind of
- 07:55weaving the OR finding relevance of
- 07:57self report measures not only in
- 07:59research but in clinical practice.
- 08:01First thing we need to do when we're trying
- 08:02to figure this out is what is our objective,
- 08:04what is our research question?
- 08:05Trying to define that in what we call,
- 08:09you know, the, the,
- 08:11the strongest operational definition.
- 08:13And we're also going to talk about
- 08:14these things called constructs.
- 08:16Welcome back to psychology,
- 08:18these fuzzy things that we don't
- 08:21have a very clear cut observation,
- 08:23You know, it's not a cut and dry.
- 08:26Yes.
- 08:26No.
- 08:26But we aim to define it that way so
- 08:29that we can get consensus as much
- 08:32as possible when we come figure
- 08:34out what it is we're measuring.
- 08:36We're now trying to figure out
- 08:37if there's something out there
- 08:38that measures something close,
- 08:39what's been published.
- 08:40And I'll show you some tools
- 08:42for how to figure that out.
- 08:43Then if they're out there.
- 08:45Are the second metric properties
- 08:46of these established measures
- 08:47or these previously developed
- 08:49measures ideally published?
- 08:50Are the second metric
- 08:53properties good enough and
- 08:57are they established in your particular
- 08:59population, whether it be a patient
- 09:00group or it could be, you know,
- 09:02a group of physicians really depends on
- 09:04where your research is going to take you.
- 09:06So I talk about construct.
- 09:09We just mean a hypothetical,
- 09:11A hypothetical variable that
- 09:13is not directly observable.
- 09:15We're talking about pain.
- 09:17We're talking about anxiety.
- 09:22What are some things outside of
- 09:23psychology that you all might
- 09:25be interested in as physicians,
- 09:33you guys can yell out or put in the chat.
- 09:36Can this be like a bias?
- 09:38Yeah, it could. Absolutely.
- 09:40Do you want to say anything more about that?
- 09:42Like what you mean about bias? Sure.
- 09:46Like if you have a bias against or if
- 09:49you have a bias against like disability
- 09:52or or or I should say ability,
- 09:56you may provide, you may look
- 09:59at people and provide different
- 10:01counseling based on that. Implicit.
- 10:07Yeah. Construct, right? Absolutely. That's
- 10:10exactly it. Perfect.
- 10:11OK, y'all, Y'all are on the same page
- 10:12with me in terms of what I'm talking
- 10:14about with Construct. And by the way,
- 10:16there's Louisiana creeping in.
- 10:17Y'all happens from time to time.
- 10:20OK, so as I mentioned,
- 10:23I've I had to take so many courses in this.
- 10:26And the kind of standard thing when
- 10:28you're teaching about assessment or
- 10:30questionnaire development or psychometrics
- 10:32is to give people the task to get into
- 10:35groups and to define a construct and
- 10:37come up with a means to measure it.
- 10:39It's just an experiential learning exercise,
- 10:42and it's the exercise that I do in the
- 10:44course that Doctor Goldman has taken with me.
- 10:46Have students come up with something that
- 10:49we feel like we know what it is but probably
- 10:52don't all agree on the same definition?
- 10:55How are we going to define this and
- 10:59how importantly will we measure it?
- 11:01I've always been fascinated
- 11:02by senses of humor.
- 11:03I you know,
- 11:04I I was paired with a couple of of
- 11:06students and we did not share research
- 11:09interests or career interests And so I
- 11:11had to come up with something that we
- 11:13would all agree would be a worthwhile
- 11:15endeavour at least entertaining enough to
- 11:17to get through the semester long project.
- 11:20And I thought, you know,
- 11:20how do we define sense of humor?
- 11:22Let's see a chat thing.
- 11:23Here we go. Oh, there we go.
- 11:27Thanks.
- 11:27And, you know, humor,
- 11:30as it turns out,
- 11:31is quite psychologically relevant,
- 11:33Also medically relevant.
- 11:34I've I've learned in subsequent years
- 11:37and thought how how let's come up with a
- 11:41way to measure people's senses of humor.
- 11:44Simply OK.
- 11:45If we look at it at at what we
- 11:48call face value past people,
- 11:50do you think something's funny?
- 11:51Was basically the task we had
- 11:53to administer something quickly
- 11:55to a large group of people.
- 11:56This is before the days of online
- 11:58surveys or if they did exist,
- 12:00I didn't know how to use them yet.
- 12:02It was also the days and it's pre
- 12:05social media and but people did
- 12:07have e-mail and I don't know if
- 12:09anyone remembers when e-mail first
- 12:11became popular in the early 1990s
- 12:14but people would send ridiculous
- 12:16chain emails all the time of
- 12:18like lists of jokes and things.
- 12:20So I decided I would go ahead and get
- 12:22in on that mix and so I sent emails
- 12:25to everyone I knew and simply said
- 12:26give me a on a on a scale of 1 to 9.
- 12:30How funny do you think this joke is?
- 12:33And this is from my class project,
- 12:34please help me out.
- 12:36It was brief 1 liners.
- 12:37I could score them on what's called
- 12:40a Likert scale from 1 to 9 and a
- 12:43priori determined based on face
- 12:45value only and the convergence of
- 12:48researchers who determined whether
- 12:50or not each one liner fit into
- 12:54particular type of humor category.
- 12:56And the types of of humor categories
- 12:59we we we operationally defined.
- 13:01Again,
- 13:01it's like 3 sub constructs under
- 13:04the large construct, right.
- 13:05So we've got the large construct
- 13:07being being humor type of humor,
- 13:09appreciation of humor.
- 13:10But we saw that as being, you know,
- 13:12some people like slapstick,
- 13:13some people only like highbrow.
- 13:14You know what are these humor types.
- 13:17And we sort of did this in a way by,
- 13:19you know, finding a whole bunch
- 13:21of jokes and then classifying them
- 13:23according to kind of the types of
- 13:25comedians or what we thought would
- 13:27be corresponding with each type.
- 13:29So we've got witticisms,
- 13:30the Jerry Seinfeld type of humor,
- 13:33you know, these sort of clever
- 13:36little observations about the
- 13:38actually exposed into the microscope.
- 13:40This is kind of a little funny,
- 13:41isn't it?
- 13:42Then the dry Stephen Wright sort
- 13:46of humor may be a little bit
- 13:49dark and then the dark and and
- 13:53potentially in poor taste.
- 13:55And then the I I think of it as
- 13:57like the Lewis Black type of true
- 14:00but more mocking other people's
- 14:02shortcomings or seemingly subpar
- 14:05intelligence or kind of like everyday
- 14:08gaffes or something like that.
- 14:11Lewis Black.
- 14:12Yeah, OK,
- 14:13What we found highly,
- 14:15highly reliable subscales and and
- 14:18reliability refers to the extent
- 14:20to which items cling together that
- 14:22theoretically should cling together.
- 14:24In other words,
- 14:25if somebody rated one witticism
- 14:27very high as being humorous,
- 14:29they would be more likely to evaluate
- 14:32another witticism as being very humorous,
- 14:34And conversely, if not humorous,
- 14:36also not humorous.
- 14:37So it really is just it's almost like
- 14:40a large correlation coefficient where
- 14:42you're actually controlling for the
- 14:45number of items comprising scale so.
- 14:48But reliability is one of the first
- 14:51tenets that we look for in determining
- 14:54psychometric appropriateness.
- 14:55So you need reliability,
- 14:57and I'm going to get into
- 14:58reliability and validating what
- 15:00these mean a little bit more.
- 15:01But that was pretty cool that we found that,
- 15:04you know,
- 15:05that these did seem to the the items
- 15:08that we determined at the face value
- 15:10should relate to each other did in
- 15:13fact highly intercorrelate with one another,
- 15:16did not do what's called a factor analysis.
- 15:18That was just way too sophisticated for
- 15:19where I was at that level of training.
- 15:21And I didn't really have
- 15:22the table size for it.
- 15:23I'm sorry.
- 15:23I'm I'm saying I I I.
- 15:25Because honestly,
- 15:25I did all the work on this thing
- 15:27and then did get a great grade.
- 15:31Actually,
- 15:31that professor still remembers me and
- 15:32wrote to me some decades later because
- 15:34he very much thought that we should
- 15:36have tried to submit it for publication.
- 15:38But I was aiming to get into a doctoral
- 15:40program to specialize in eating disorders.
- 15:42And it just seemed I was worried
- 15:44at the time that it would be,
- 15:45you know,
- 15:46not that it wouldn't necessarily
- 15:48register me as the serious scholar
- 15:50that I was trying to become.
- 15:52Ironically, 30 years later,
- 15:54I'm still not so much.
- 15:55But you know, whatever,
- 15:57OK, why are we doing this?
- 15:59We,
- 16:01the,
- 16:01the are are going through
- 16:03this entire area of
- 16:04education, the process of defining
- 16:07constructs, coming upon a consensus
- 16:09of what of how we are all,
- 16:11as the researchers and our colleagues,
- 16:15going to agree on this
- 16:17operational definition.
- 16:19Developing questions and these
- 16:21questionnaires is actually pretty hard.
- 16:25Doctor Goldman can speak to this.
- 16:27It's a pretty complicated process.
- 16:29The numbers end up surprising you.
- 16:31The method of data collection
- 16:33ends up surprising you.
- 16:34And then unfortunately it just it
- 16:38takes a lot of time and energy.
- 16:40Subtle little details about the
- 16:43administration of your questions
- 16:45can have a significant impact
- 16:48on on the results of of what
- 16:50you're actually going to see.
- 16:51And I'm going to show you more
- 16:54on that on these subtleties,
- 16:57what I did in this questionnaire
- 16:59that you all just completed, right?
- 17:02Everybody had the QR code and you
- 17:03answered a couple of questions.
- 17:05What you didn't know is that I
- 17:07had embedded a randomizer at the
- 17:09beginning of the questionnaire so that
- 17:11people received slightly different
- 17:13questions in your quick little 6
- 17:16item survey or whatever it was.
- 17:18They were subtly different
- 17:20by just a few little pieces.
- 17:23So everyone should have seen this meme.
- 17:25Did everybody see this meme right and
- 17:28you were simply asked to evaluate?
- 17:30How funny do you think this is?
- 17:33Does anybody happen to know which
- 17:35one they received? Which version?
- 17:37Do you see the difference in the versions?
- 17:40OK,
- 17:45can anyone guess what might happen
- 17:49here if we were to, let's say,
- 17:51let's say let's score this at 123 and four,
- 17:55with four being, you know,
- 17:56people thought it was funnier.
- 17:58This let's not score at all
- 18:00because it's not applicable and
- 18:02we've still got one 2-3 and four.
- 18:06Does anybody have any hypotheses?
- 18:09So let's say for because these people
- 18:11didn't get the not applicable option,
- 18:13are people going to skip that question
- 18:15or what do you think they're going to do?
- 18:25Well, I could tell you what I did,
- 18:27which is I got the first one or the
- 18:30one that's on the left and I didn't
- 18:32get it and so I rated it as not funny.
- 18:35So my hypothesis would be that, you know,
- 18:39that the on the second question,
- 18:41you know, you would probably have
- 18:43whoever said not funny would be
- 18:45further speciated into those two.
- 18:49Yes, I agree. So the the thought is
- 18:52OK didn't get it. I'll explain it.
- 18:54It's a song called Africa by
- 18:56the band Toto And. Yeah, OK.
- 18:59So that was like a hit and early.
- 19:02Oh, now I get it. Oh, thank you,
- 19:04Marnie. All right. So,
- 19:08OK, so fun song lyrics,
- 19:11a whole lot of pop, cultural reference
- 19:12all mushed together, fun, you know,
- 19:15I'm amused by these kinds of things.
- 19:17So, so the hypothesis then being
- 19:22that's probably the means over here
- 19:24might be a little bit higher, right?
- 19:26Because anybody who didn't get
- 19:28it is just going to be exempted.
- 19:31And anybody who thinks it's funny or at
- 19:32least that that's that's my hypothesis.
- 19:34I thought, I think in terms of what we're at,
- 19:35we're at the way we're actually
- 19:37expecting the data to pan out.
- 19:39So we've got plenty of people thinking
- 19:41that not applicable, you know,
- 19:42they're like, I don't get it.
- 19:44And then some people like know, you know,
- 19:46very, very few people, whatever.
- 19:48Not very many people were as
- 19:49amused by this as I was.
- 19:51Let's put it that way.
- 19:53Now for the not applicable, when there's.
- 19:57When we don't have the not applicable,
- 19:58we've just got. This is not funny.
- 20:00Two, it is funny.
- 20:02We've got.
- 20:03Now this is consistent with your hypothesis,
- 20:06right?
- 20:06Like,
- 20:07because if we were to add
- 20:09the not applicable here,
- 20:11they'd show up here,
- 20:13right?
- 20:13Except
- 20:17what we've got here,
- 20:19and the way I'm interpreting this is when
- 20:21we have the not applicable option here.
- 20:24We should theoretically just
- 20:28see these two being equivalent.
- 20:30Instead, we've got a couple of
- 20:32different factors that might have
- 20:34influenced the pattern of responses.
- 20:36One is having that not applicable option
- 20:40might have told those people who thought
- 20:42it was a little bit amusing that it's
- 20:45actually a funnier than it is because oh,
- 20:48I'm in on this inside joke a little bit.
- 20:51Other people might not get it.
- 20:52That actually makes it a little bit funnier.
- 20:56Or it could be something as simple as
- 20:59the graphic of the five point scale.
- 21:02People do look at the number of
- 21:05response options, 4 versus 5,
- 21:07and we draw all kinds of inferences,
- 21:10especially when it comes to Likert scaling
- 21:12and a five point scale versus a four point.
- 21:14We look for middle a middle response option.
- 21:18Obviously the middle response
- 21:19option is the most popular.
- 21:21When we've got a 5 point scale,
- 21:26we see a little bit less of that
- 21:31when we've only got the four points.
- 21:36It's there are a lot of different poses.
- 21:38I don't know which one is correct.
- 21:39All I know is that including that
- 21:41non applicable option changes the
- 21:43pattern of results and this is
- 21:45highly significant if we actually
- 21:47put these into means to treat them
- 21:48not applicables as a missing value,
- 21:50it's highly significant like point OO1 and
- 21:52it happens over and over and over again.
- 21:55All right, shifting gears now,
- 21:59let's say we've got these.
- 22:03Now I kind of want to review
- 22:06some thoughts around real
- 22:08life research questions and
- 22:13how we're going to need to determine our
- 22:15measurement on both sides of the question.
- 22:17OK. So let's just say we've got
- 22:20a hypothetical research question
- 22:21about whether or not our policy or
- 22:24practice or procedure influences
- 22:26patient experience, right.
- 22:27We've got to now think about
- 22:31what about patient experience?
- 22:33Do we really care about?
- 22:34Are we looking at satisfaction?
- 22:35Are we looking at the competence
- 22:37of our of our medical staff?
- 22:39Are we looking at perceptions of
- 22:41care and nurturance that might be
- 22:42different from competence and so on.
- 22:44So there are a lot of different
- 22:46ways that we might aim to measure
- 22:48patient experience and there
- 22:52might be many existing measures
- 22:55to evaluate each of them.
- 22:59I don't this three shouldn't
- 23:01be there. That's an error.
- 23:05Once we figure out what it is we're
- 23:07exactly trying to focus on our measure,
- 23:08we next have to figure out whether or
- 23:10not there are questionnaires out there,
- 23:12measures out there that will do it.
- 23:13There are many different types of
- 23:15patient experience questionnaires.
- 23:16It developed in many different
- 23:19subfields or specializations,
- 23:20and it is up to you now to go
- 23:23searching and finding them.
- 23:25Here's how to do that.
- 23:27There's actually something called the Health
- 23:31and Psychosocial Instruments Database.
- 23:34GAIL has it. It's.
- 23:35I don't know how to pronounce it.
- 23:37Happy, I guess.
- 23:39Health and psychosocial instruments.
- 23:41Ask your medical librarian.
- 23:44They're remarkably helpful.
- 23:46You can also try cited reference searching
- 23:48and scope as your web of science.
- 23:50I prefer Google Scholar, even though our
- 23:52public health librarian doesn't like it.
- 23:54I'm just.
- 23:55But anyway,
- 23:55I feel bad kind of recommending it,
- 23:59given that librarians have
- 24:02identified many flaws with it.
- 24:04This is an incredibly helpful
- 24:06slide and resource. Go to Ovid.
- 24:10Psych Test is another option.
- 24:12Psych Info is another option,
- 24:14but Happy is really going
- 24:16to pull your info for you.
- 24:18The next question are the psychometric
- 24:21properties of these scales adequate?
- 24:23All right, now a little crash course
- 24:25in what psychometric properties are.
- 24:27Again, you first want to establish
- 24:30that your properties are
- 24:35reliable, then valid,
- 24:39and if you are looking at subscales,
- 24:41you also need to gauge the
- 24:44liability and validity of those.
- 24:51I'm sorry, I'm a little
- 24:52distracted because I'm.
- 24:52I'm concerned that I'm showing you
- 24:54the wrong slide show because I've
- 24:56got three slides open right now.
- 24:58Now This is correct.
- 24:59All right, this is fine.
- 25:04I I described inter item reliability
- 25:06a little bit a few slides back.
- 25:08Inter item liability is almost like a large
- 25:14correlation coefficient of all of the
- 25:18items comprising a particular scale.
- 25:20The by and large,
- 25:22the more items you have on a scale,
- 25:26the higher your reliability will be.
- 25:29But that doesn't mean that's a good thing
- 25:32because it does control somewhat for the N,
- 25:36the N being the number of items.
- 25:37You can artificially drive up an
- 25:40inter item reliability coefficient
- 25:42which is called Chromebox Alpha in
- 25:44most cases just by having extra and
- 25:46and potentially unnecessary items.
- 25:48What you always want.
- 25:49And so sometimes,
- 25:50like pseudo scientific jargon will say,
- 25:53oh, our scale is so much better because
- 25:55we have a coefficient alpha of .93,
- 25:57and the gold standard that's
- 25:58been used before this only has
- 26:00a coefficient alpha of .89.
- 26:02No, no, no, no,
- 26:05that's not really that impressive of a leap,
- 26:08especially if somebody's asking
- 26:10or measuring something with 35
- 26:13questions and somebody else can
- 26:15get it done in six and still
- 26:17have a good coefficient alpha.
- 26:19That's the one you want to choose.
- 26:23So internal consistency is the extent
- 26:25to which those items interrelate.
- 26:28You. Also that once you've established
- 26:31reliability, you can then talk about
- 26:33various types of validity content,
- 26:35validity and criteria and validity.
- 26:37And when you have all of these together,
- 26:39now you've got evidence of construct
- 26:43validity, reliability measures.
- 26:44There are particular different kinds.
- 26:46There's inter rater reliability which is
- 26:52if you have you know multiple individuals,
- 26:56for example evaluating a particular
- 26:59stimulus or interview or diagnosis.
- 27:02That's a little bit less relevant
- 27:05to scale development but you'll
- 27:07see it in the literature.
- 27:09Test, retest,
- 27:10reliability which you generally
- 27:12want to establish if possible.
- 27:14That's where when you're when
- 27:17you're looking at the properties
- 27:19of your measure you then want to
- 27:22re administer it to at least a sub
- 27:25sample of your population and look
- 27:27at the correlation looked at at how
- 27:30well these measures align but with
- 27:33each other from time 1 to time 2.
- 27:34Now again this is a,
- 27:35this is really in reference to
- 27:38questionnaire development and construction.
- 27:40But when you are choosing your
- 27:43measures you want those that have
- 27:45appropriate inter item reliability and
- 27:47test retest reliability that should
- 27:49be established if it's going to be
- 27:52a solid self report questionnaire
- 27:53that's used out there in the field.
- 27:55And these you know screen share
- 27:57this or not screen share just screen
- 27:59grab this particular slide.
- 28:00Because these are just the the
- 28:03reliability coefficients that you'll be
- 28:05using either alpha split half or ICC,
- 28:07which is a interclass correlation
- 28:10coefficient if you're using,
- 28:11you know, non continuous data.
- 28:15If you're looking at categorical
- 28:17outcomes or categorical decisions,
- 28:18then you might be looking
- 28:20at what's called the KR 20.
- 28:22These are just different statistics
- 28:23that ultimately are going to
- 28:25tell you the same thing.
- 28:26And then I also just want to
- 28:28give you some of the guidelines
- 28:31for what is considered adequate.
- 28:33Anything above .7 is going to look
- 28:35pretty good, especially again,
- 28:37sometimes you'll see like a three or four
- 28:40item scale with a .7 reliability coefficient.
- 28:43That's excellent that I would
- 28:45get very excited about that.
- 28:47Of course,
- 28:48you know you always want to see
- 28:50something in the point nines,
- 28:51but you're always wanting to
- 28:54balance against participant burden.
- 28:56You'd much rather have full and complete
- 28:58data than highly reliable data where
- 29:0120% of your respondents have dropped
- 29:02off by the end of the of the study.
- 29:07Validity. OK, I've spoken about
- 29:08reliability and and you almost
- 29:10think of reliability as being like
- 29:12the the likelihood that you can
- 29:14get the same response every time.
- 29:16That's the precision.
- 29:17Validity is whether or not it's true.
- 29:20So in order for something to be valid,
- 29:22it must first be established to be reliable.
- 29:26You can have something be very,
- 29:27very reliable, but wrong, right?
- 29:32So the bathroom scale can give
- 29:35you the same answer every time,
- 29:37but it might be 10 lbs off, and
- 29:41it's going to be 10 lbs off for every
- 29:44single participant that gets on it.
- 29:45So it's reliable. That reliably gives you
- 29:48consistent measures, but it's not valid.
- 29:54And valid validity is established by a
- 29:57bunch of different types of criteria,
- 29:59so there are different kinds.
- 30:00Face validity can generally
- 30:02get from the research group,
- 30:03or from focus groups,
- 30:05or from the Delphi panel.
- 30:07This is whether or not your measure
- 30:09is intending what or is measuring.
- 30:11The it's it's how obvious
- 30:15it is really does it?
- 30:17Is it measuring what it
- 30:18seems like it's measuring?
- 30:21Does anybody have any ideas of what
- 30:23this questionnaire is measuring?
- 30:28It's question one of a of a
- 30:30widely used validated measure.
- 30:31What do you think it's measuring?
- 30:45Any ideas?
- 30:51Don't be shy, guys. Chat or open. Or unmute.
- 30:57Yeah, seriously, Jordan, Yell.
- 30:58I'd like to be interrupted. I'd
- 31:02like to converse.
- 31:02I don't like to talk at all.
- 31:07Right. What do we have?
- 31:08We've got. This is depression.
- 31:10Yes. Excellent.
- 31:11This is the first question of
- 31:13the Beck Depression Inventory.
- 31:15Widely used. Excellent.
- 31:16So face valid? Yes, pretty obvious.
- 31:20Perhaps even to non specialists,
- 31:23even to patients.
- 31:24What this is probably measuring,
- 31:26right don't need a whole lot of of
- 31:28education and depression to guess.
- 31:30This is probably what it's about.
- 31:31It's at least measuring sadness.
- 31:36Does anybody know what
- 31:37this might be measuring?
- 31:38These are three items on the
- 31:41same subscale of another widely
- 31:44used psychiatric instrument.
- 32:00Any ideas,
- 32:07Ruchika, You unmuted. There we go.
- 32:10That's great. Oh, I'm sorry.
- 32:12I didn't. I didn't realize. Unmuted.
- 32:14But I was thinking about anxiety.
- 32:16Good, good, good. Good idea.
- 32:17So anxiety and depression.
- 32:19Or anxiety. Or depression.
- 32:20Maybe a mix of anxiety,
- 32:22depression or anxiety.
- 32:23So that right there,
- 32:26given that we have different ideas
- 32:30as to what this could be measuring,
- 32:32suggests that it's maybe not as face valid.
- 32:35And in fact I would not consider
- 32:37this to be a face valid measure.
- 32:39This is the depression scale of the MNPI,
- 32:42the Minnesota Multi Basic
- 32:44Personality Inventory.
- 32:45This measure was what's called
- 32:47empirically keyed, meaning,
- 32:49it is not that these items are not
- 32:51put together based on their value or
- 32:54their obvious correspondence with the
- 32:56construct that they're trying to measure.
- 32:59Rather,
- 33:00they individuals were classified
- 33:01according to the construct and then
- 33:04based on how they responded to these
- 33:06individual measures or the individual items,
- 33:08those items then mapped on to the
- 33:11creation of the scale as it happens.
- 33:14Appetite disturbance,
- 33:15sleep disturbance and mood of course
- 33:17are all prongs that relate to a
- 33:20clinical diagnosis of depression.
- 33:22However, not obviously so to everybody,
- 33:26right? Not face valid one more.
- 33:32Does anybody have any ideas
- 33:34what this might be measuring?
- 33:42Again, 4 items on the same subscale?
- 34:01Great idea. The question mark already
- 34:04tells me it's not face valid.
- 34:12This is the MMPIK scale,
- 34:15which is actually a correction scale and
- 34:18what it measures is social desirability.
- 34:22And when people score very highly on this
- 34:26scale, it invalidates their responses on the
- 34:29rest of the of the Large Assessment Battery.
- 34:33What this? At times I feel like,
- 34:35and this is all true. False, right?
- 34:37At times I feel like swearing false.
- 34:40Come on, criticism or
- 34:43scolding hurts me terribly.
- 34:44False. No, I'm good with it.
- 34:46It really helps me become a better person,
- 34:48you know, Come on, this is off.
- 34:49You know, if if people are saying false,
- 34:52false, false, they're
- 34:56trying to present themselves
- 34:57in a better light.
- 34:58What that means is that their
- 34:59response is on our side,
- 35:00something that's very not face valid,
- 35:04but was able to be determined again
- 35:06through that empirical keying,
- 35:08which I think is just fascinating.
- 35:12Other types of validity criterion, validity.
- 35:16This is really the meat of a
- 35:18lot of what we want to do.
- 35:19We're not usually just going out there
- 35:21trying to find a measure to measure
- 35:23something because of the sake of we
- 35:25want to make sure that we can truly,
- 35:27you know, know the status of truth
- 35:29of this particular construct.
- 35:31We're actually trying to relate
- 35:33to something meaningful,
- 35:34which means we want our scores
- 35:36on our measures to actually tell
- 35:38us something on down the line,
- 35:40whether it maps onto a particular
- 35:43diagnosis or, you know,
- 35:45recovery button, whatever.
- 35:47That would be concurrent validity if we're
- 35:50looking for our measure to relate to a
- 35:54diagnosis or some other gold standard.
- 35:56If we're looking for our screening tool
- 35:59to map onto an actual lengthier battery
- 36:06or something called predictive living,
- 36:08which is simply whether or not accurate
- 36:10knowledge of our questionnaire predicts
- 36:13something meaningful in the future,
- 36:15I love the old MCAT score and
- 36:18performance in medical school.
- 36:20Predictive validity?
- 36:22No, not a lot.
- 36:25We certainly know that's the case
- 36:27in the PhD sciences that GRE scores
- 36:29have like I believe it's about,
- 36:31I think it's at point O2 correlation
- 36:37coefficient with dissertation
- 36:38quality of resource, productivity.
- 36:40So of course we've got other issues
- 36:42like restriction of range and so on.
- 36:44But ultimately,
- 36:45if we're going to go through all the
- 36:47process of administering a measure,
- 36:49we want it to relate to something of value,
- 36:51something meaningful.
- 36:55These are the measures appropriate
- 36:57for your target population.
- 36:58And this is what I had mentioned
- 37:00previously about, you know,
- 37:01you might have something that's been
- 37:04validated for use in adults and
- 37:07it's not necessarily as translated
- 37:09to use in adolescence or children.
- 37:11And so you just want to make sure that
- 37:13if there's an existing construct out
- 37:15there that it has been validated for use
- 37:17in your particular population or that
- 37:19the population that you're studying.
- 37:22Hopefully that is the case.
- 37:23If not, you might have yourself
- 37:25a revalidation study,
- 37:28advice, advice, advice, advice.
- 37:29Once you've picked all of these things,
- 37:32I want to take you through how to
- 37:35avoid shortcomings in administration.
- 37:37And I've done most of these blunders
- 37:43myself over the past decades.
- 37:45I've done something like, you know,
- 37:48it's it's the best way to learn
- 37:49and it makes you meticulous.
- 37:50But I I, I collected thousands
- 37:54of participants of data only to
- 37:56discover after the fact that I had
- 37:59somehow left education out of the
- 38:01out of my demographic battery.
- 38:03So I had a really flimsy measure of socio
- 38:07economic status and educational attainment.
- 38:10And I mean it's just it's sickening,
- 38:13but it happens anyway.
- 38:14I want to to draw your attention to some
- 38:18kind of finely tuned things and and to
- 38:20help you kind of map out your research.
- 38:22Think of it,
- 38:23while you're setting up your questionnaires
- 38:25in Qualtrics or in Redcap or whatever,
- 38:28think forward about how you're
- 38:29going to analyse your data.
- 38:34I would caution you against using many
- 38:36open-ended questions if at all possible.
- 38:39If you do get a qualitative
- 38:42researcher on board early on,
- 38:44think about the scale of measurement.
- 38:46Are you going to be using True,
- 38:47false, Yes No, Yes, No,
- 38:50Maybe Yes No does not apply.
- 38:53Think about whether you're using
- 38:55multiple choice, rank ordering,
- 38:58ordinal data, or the good old fashioned
- 39:01Likert scale continuous data.
- 39:02I love the Likert scale.
- 39:05I don't know why.
- 39:06I find that it allows me
- 39:09to do a lot more data wise.
- 39:12That's just my preference and it's almost
- 39:16like why do I like cats as much as I do?
- 39:18I can't really explain it, I just know
- 39:20it's the case open-ended questions.
- 39:23I would just caution you against
- 39:26these or use them very sparingly.
- 39:29They might be appropriate for some things,
- 39:31especially some knowledge based things.
- 39:33Some things like you know,
- 39:34birth year, whatever else you know,
- 39:38idea of exposure.
- 39:39You know you might be forced to use
- 39:41them in particular circumstances,
- 39:43but the way you're going to be
- 39:47analyzing the information will
- 39:49dictate how you'll use them.
- 39:52If you are using multiple open-ended
- 39:55questions to ask opinion or experience,
- 39:58you're definitely going to need
- 39:59that qualitative researcher,
- 40:00because you're going to need to
- 40:02understand how to code open text data.
- 40:05I don't know how to do it personally.
- 40:08Some people,
- 40:09when they're wanting to do
- 40:10mixed methods research,
- 40:11they need to get that qualitative person.
- 40:13I can help on the quantitative
- 40:15side with questionnaire stuff.
- 40:16I can't.
- 40:16Once we start getting into
- 40:18analyzing open text stuff,
- 40:19I don't know how to do it that
- 40:22that's an entirely other parallel
- 40:25type of research approach.
- 40:28One option,
- 40:29if you feel that you must include
- 40:33open-ended variables is to do
- 40:35kind of a hybrid approach.
- 40:36So these are actually questions from
- 40:38the Food Addictions questionnaire,
- 40:40the Yale Food Addiction Questionnaire,
- 40:42where they ask people to indicate
- 40:44any specific foods that they in which
- 40:47they experience an addictive like
- 40:53addictive like qualities when
- 40:54eating the foods and people feel,
- 40:56you know out of control or that
- 40:58they they can't get enough of them.
- 40:59And so they they ask these very
- 41:02precise food items and then say, oh,
- 41:04and are there any others that that we missed.
- 41:07So that's an option as well seeing is
- 41:09there a question coming through. OK,
- 41:15a problem with open-ended,
- 41:17you run the risk of getting answers that
- 41:21are different from what you had intended.
- 41:25So you need to always make sure that
- 41:29you're providing really good instruction
- 41:32sets and direction for participants.
- 41:34Fixed response options when you're
- 41:36talking about fixed response options,
- 41:37which I always prefer.
- 41:38But again, that's just because they they so
- 41:41nicely lend themselves to data analysis,
- 41:43to empirical analysis.
- 41:44But you do have options within that,
- 41:47so the Likert scale,
- 41:49which I do love,
- 41:51can be a really nice one in terms of
- 41:54using what's called a visual analogue.
- 41:57Just providing people with two anchors.
- 41:58Now Qualtrics and Redcap will allow
- 42:01people to use a draggable scale and
- 42:03actually get it a range from zero to 10
- 42:07without having numbers involved at all.
- 42:09Something you need to consider when
- 42:11you are administering Likert scales.
- 42:16You want to make sure that you are
- 42:20following the response format of the
- 42:22scale as it was originally validated.
- 42:25If you're creating your own,
- 42:26you have to consider whether or
- 42:27not you're using four points or
- 42:29five points or seven points or 9,
- 42:31whether you're going to give
- 42:32them a middle response option,
- 42:33and whether or not you're going to
- 42:35label all those response options.
- 42:39When you're formatting these things,
- 42:41you want everything to
- 42:42be as clear as possible.
- 42:44Use white space A lot.
- 42:46Use page breaks a lot. Yeah.
- 42:50It's probably pretty unlikely
- 42:52that you're going to be doing many
- 42:54paper questionnaire administrations
- 42:55at this point in time. All my,
- 42:57I can't remember the last time I saw one.
- 42:59If you're using paper basics are, you know,
- 43:02only use the front side of the paper.
- 43:04It's wasteful it,
- 43:05it kind of kills us in this day of,
- 43:08you know, trying to be
- 43:12responsible environmentally.
- 43:13But people will skip that
- 43:16second page. They just will.
- 43:18They don't flip over the pages
- 43:23when I say code responses.
- 43:27This has actually been done and I see it a
- 43:32lot, this whole thing of making people hold
- 43:35in their head what is the strongly agree,
- 43:38what is strongly disagree and then undecided.
- 43:41Now we've got a neon people are having
- 43:43to like kind of keep on scrolling back
- 43:45to make sure that they're understanding
- 43:47you know what each column heading is.
- 43:50This is also problematic because that
- 43:53you now looks like a fifth point here,
- 43:56and so people are assuming that
- 43:58disagree is a middle point when in
- 44:00fact this entire column is graphically
- 44:05telling too much information.
- 44:08Another bad idea is making
- 44:10people code and put in the box.
- 44:13You want to reduce as much burden
- 44:15as possible from your participant,
- 44:17from your participants.
- 44:17That's both in terms of questionnaire length,
- 44:20but in also what they need to do.
- 44:25This is from the BDI.
- 44:27This is the way it's actually
- 44:29formally administered in the if you,
- 44:30you know go on to to the psych measures
- 44:33thing and order a packet of 100 BD is
- 44:35this is what it's going to look like?
- 44:37But all the time you see
- 44:41people put fours in the box.
- 44:44I think that we can safely
- 44:46interpret that to mean a three.
- 44:49But now what do you do for the next question?
- 44:50When they put a two in the box?
- 44:55I don't know.
- 44:56You have to consider that invalid.
- 44:57So instead, there's nothing wrong with
- 44:59just asking people to check the box,
- 45:02to check which one's beside there
- 45:03to click the particular button,
- 45:04as opposed to making them do all that work.
- 45:11Include instructions, but assume
- 45:12people are not going to read them.
- 45:15When people do read them,
- 45:16make sure they're appropriate.
- 45:18People get very annoyed when
- 45:20you say check the box and it's
- 45:22actually circle the number.
- 45:24Just keep instructions consistent.
- 45:27Also consider your formatting.
- 45:32This is so strange. I had a whole animation
- 45:35for this page. I don't know what's going on.
- 45:37Anyway, this, I just this is,
- 45:40this is a real life example from last week
- 45:43this everything looked great on the desktop
- 45:45and then I go to check it with my phone.
- 45:48No one's going to be able this is,
- 45:50this is so much work for
- 45:51the participant to try to.
- 45:52What does that even say?
- 45:55Constant, constant concerns and no concerns.
- 46:01This is terrible, right?
- 46:05It almost flew again,
- 46:06that comment about the paper.
- 46:08People aren't going to flip over the
- 46:13page. Oh, there's my animation. All right.
- 46:15Check your format every single time.
- 46:17Pilot it. Make your children take it.
- 46:20Make your colleagues take it.
- 46:22When you're or,
- 46:23think about the ordering of the of the items.
- 46:25This is a real example as well.
- 46:29I say use page breaks because
- 46:31these two were on the same page.
- 46:34Someone's following along.
- 46:35What I look like is important.
- 46:37Yes, I agree. Yes, I agree.
- 46:39You know, I don't know if I agree that much.
- 46:41I prefer not to send it.
- 46:43All of a sudden the response
- 46:45options have flipped on them.
- 46:47OK, you can avoid this the the
- 46:51errors that this will impose,
- 46:52and it will impose errors because people
- 46:55are they've gotten into their rhythm.
- 46:57They're using what's called a
- 46:59response set now, and to avoid these,
- 47:03so I'm saying, group themes together.
- 47:06You know,
- 47:06if you're asking about particular constructs,
- 47:07you're administering multiple questionnaires.
- 47:09Keep them that way.
- 47:11If you're using a validated questionnaire,
- 47:13do not shuffle the order of items.
- 47:17Keep the items administered
- 47:19in the way in which they are
- 47:22originally presented and validated,
- 47:25and then be sure to use page breaks.
- 47:27Page breaks will allow people to kind of
- 47:30reset for these flipping around changes
- 47:33in instructions or themes as well.
- 47:39When you're looking at at at
- 47:42question quality, always balance.
- 47:45You know your reliability and your
- 47:48validity against the burden on the
- 47:51participant, people will drop off.
- 47:56This is all alluding to the
- 48:01the types of reliability
- 48:03and the types of validity.
- 48:04But when you're looking at the
- 48:05at which measures you're going to
- 48:07choose and you're likely to find
- 48:08several that would that would relate.
- 48:10It's always just kind of a a balance
- 48:13against participants and and how well
- 48:15those psychometrics qualities look.
- 48:18Also, things to kind of ask yourself
- 48:21or have going in the background more.
- 48:23The lengthier, the more tedious,
- 48:25the more cumbersome,
- 48:26the more confusing.
- 48:27People are just going to disengage.
- 48:32I try to remove as many numbers
- 48:36and instructions as possible.
- 48:40People don't like numbers.
- 48:41It's heartbreaking, I know.
- 48:43All right, tell me some questions.
- 48:44Some problems with this question.
- 48:55I feel like some people might
- 48:56be overly optimistic.
- 48:59Yes, Assumes consistency.
- 49:02Assumes exercise.
- 49:03Assumes people understand what
- 49:05is meant by exercise. Excellent.
- 49:08You've got all the main the main things.
- 49:10Good, good, good.
- 49:12What does it mean? Guess what?
- 49:14I asked you all this question,
- 49:17except I had my little randomizer again.
- 49:20And what did I ask you?
- 49:22I asked about typical and I
- 49:24asked about just this past week
- 49:31time and time again. This is our pattern.
- 49:37I never understand what this is,
- 49:39but that also happens all the time.
- 49:41But people's the idea of their typical
- 49:45exercise is often a lot better.
- 49:48It's not often.
- 49:49It's again an AP value of like less than
- 49:52point OO one is better than their actual.
- 49:55Well, you know, I mean it was cold and rainy.
- 49:58There was that major storm that came through.
- 50:00It's the holidays.
- 50:01I've got to do so much shopping
- 50:03doesn't matter if I'm asking this
- 50:05in perfect spring weather in August,
- 50:07it doesn't matter.
- 50:07The same phenomenon occurs and we
- 50:09know this everybody wants to that
- 50:12social desirability thing of course,
- 50:13but we generally believe that our
- 50:15that our typical is a little bit
- 50:18better than our actual that causes
- 50:20difference in what we actually observe.
- 50:24Same thing with the frequency
- 50:27of our response options.
- 50:29OK.
- 50:30So asking on what's called a
- 50:32high frequency scale versus a low
- 50:34frequency scale contributes to a
- 50:36different pattern of responding.
- 50:38People are drawing inferences about
- 50:41these anchors and what they actually mean.
- 50:43This was done in a pain study
- 50:45or in a pain clinic,
- 50:47and it's likely that those who had the
- 50:50high frequency scale interpreted that
- 50:53to mean like a lower level of pain,
- 50:56like the meaning of what, you know,
- 50:58the daily headache or aches and
- 51:00pains or joint pain or whatever,
- 51:02versus the debilitating migraine
- 51:06in bed for the entire day,
- 51:08right.
- 51:09So people are drawing things
- 51:10based on the options you give
- 51:12them or drawing inferences.
- 51:16Same thing with something a little bit less,
- 51:20a little less objective, you know.
- 51:21So we think about physical pain as
- 51:23being probably about a, you know,
- 51:25you've got something that you feel
- 51:27acutely versus something a little
- 51:29bit more fuzzy and psychological.
- 51:31Yet we see here that providing people with
- 51:36numbers changed the pattern of responding.
- 51:39This is why I've gotten to the
- 51:40point that I try to remove all
- 51:42numbers are my questionnaires.
- 51:44Unless it's something that's been validly
- 51:46established as requiring the number.
- 51:48I asked you all the same thing.
- 51:50Do you like New Haven
- 51:549 point scale ranging from
- 51:57either 1:00 to 9:00 or -4 to 4?
- 52:00And here's what we see.
- 52:02It's just wild to me,
- 52:04that something that subtle.
- 52:06And again, I do the means analysis
- 52:08and I translate all of these
- 52:10to 1:00 to 9:00 with an easy
- 52:12additive transformation, right?
- 52:14We see a very different pattern.
- 52:16For some reason people don't want to
- 52:21select negative numbers in this context,
- 52:25which is so interesting.
- 52:26Anything less in the middle value
- 52:29should be an insult or whatever,
- 52:31not just like New Haven,
- 52:32but you know what I mean?
- 52:34Wild to me, labeling effects people
- 52:38also don't want to label themselves.
- 52:40This is a real story from real
- 52:43research conducted at Yale.
- 52:44They had done a whole lot of pilot
- 52:46research and determined that they had
- 52:47more than enough people in the community
- 52:50who drink sugar sweetened beverages.
- 52:52Then they were bringing them into the
- 52:53lab study and they just couldn't get
- 52:55enough people through the screening
- 52:56And they had found that, you know,
- 52:58the the plenty of people have,
- 53:00you know, energy drinks or whatever.
- 53:03And then people would call
- 53:04the screening and you know,
- 53:05the person's like in my office
- 53:07racking their brain.
- 53:07Why can't I get enough subjects
- 53:09for this lab study?
- 53:10I'm like how are you asking
- 53:11the question on the screen?
- 53:12Like we're
- 53:12asking if they have two or two or
- 53:14more sugar sweetened beverages?
- 53:15I'm like ask it open-ended.
- 53:17Ask them how many they have per week.
- 53:19Boom. Enrollment's covered.
- 53:20People don't want to put
- 53:22themselves in a box of like,
- 53:24oh, why are you asking that?
- 53:25I don't want to be in the pathological group.
- 53:29Now, asking this, do you have at least two?
- 53:33No, I do not. How many do you have?
- 53:35open-ended? 35% versus 22%.
- 53:40Isn't that wild? I
- 53:42know we're fascinating creatures.
- 53:44You see why I study psychology.
- 53:47Love it. All right, similar thing.
- 53:49First Force choice versus an open.
- 53:53Again, we've got these.
- 53:56We've got these irregularly
- 53:58spaced categories that sort
- 54:01of impose A Likert continuum,
- 54:03but they're not.
- 54:04They're uneven.
- 54:04I did this on purpose
- 54:08and we get very different pattern
- 54:10of responding from the first
- 54:11forced choice to the open-ended.
- 54:12Now, I said before,
- 54:14I I cautioned against using open-ended.
- 54:16You can do this with Qualtrics or
- 54:18Redcap by just having it of 0 to 30
- 54:20drop down or zero to 31 drop down.
- 54:22That'll cover that.
- 54:23Or even using a text box and
- 54:25letting them type it in,
- 54:27but validate it that the range of
- 54:28scores can only range from zero to 31.
- 54:34Food cravings again administered.
- 54:35This should be very, very straightforward.
- 54:36I gave some of you check the box
- 54:39of how many I gave. Some of you,
- 54:40you know go through and answer yes or no.
- 54:42How many of these have you?
- 54:44Have you people take information from
- 54:48the number of options you give them.
- 54:51If you give them just eight,
- 54:52they'll select two or three.
- 54:53If you give them twenty,
- 54:54they might select five or six.
- 54:56If you give them a hundred,
- 54:57they might select 30.
- 54:59Forcing people to say yes or no again,
- 55:02you get a very different
- 55:05pattern of responding.
- 55:06These are heuristics.
- 55:07They're little graphic things that
- 55:10people unconsciously consider when
- 55:13generating their answers for you.
- 55:17What is my point of all of this?
- 55:18Oh, I'm actually doing it on time this time.
- 55:22Amazing bias can be introduced accidentally
- 55:26by any number of subtle things.
- 55:29Be very careful. Pilot everything.
- 55:32Look at your raw data too.
- 55:35So give it to five friends or five
- 55:38colleagues to complete on their own,
- 55:40and then pull the spreadsheet and
- 55:42make sure the coding matches.
- 55:43Answer it yourself a couple
- 55:45of times on paper.
- 55:46You know, like print it out,
- 55:47do it on paper,
- 55:48go through and do it,
- 55:48And then make sure the scores are right.
- 55:50Because you can have all kinds of
- 55:52little glitches inside of Qualtrics.
- 55:53You know you can overcome them
- 55:55by recoding at the tail end,
- 55:56but it's just so much you can save
- 55:58yourself the headaches by getting it taken
- 56:01care of before you collect your data.
- 56:03Absolutely administer your questionnaires
- 56:04in the way they were originally validated,
- 56:07because these small little modifications
- 56:09can really mess things up.
- 56:11Even things like adding a Not applicable.
- 56:14I know you think that you're like
- 56:15cleaning up data and you think
- 56:17that you're coming to the patient
- 56:19where where they are.
- 56:20Allow people to skip questions
- 56:22that'll cover that.
- 56:23Unless, of course,
- 56:24the original questionnaire
- 56:25included or not applicable,
- 56:29and then power of the questionnaire.
- 56:32It's glorious.
- 56:32You can answer your research questions,
- 56:35You can satisfy your curiosity,
- 56:37You can screen.
- 56:38And of course you can win the debate on
- 56:41how you're going to spell your child's name.
- 56:43My sweet, sweet spouse knows that I
- 56:46specialize in psychometrics and yet,
- 56:48for whatever reason,
- 56:49has allowed his fate at the fate
- 56:52of our home to be subject to the
- 56:55demands of my online questionnaires,
- 56:57and still hasn't realized that I have
- 56:59figured out how to introduce bias to
- 57:01make myself right every single time.
- 57:03And that is it.
- 57:04That's
- 57:04all I wanted to it on time.
- 57:08Hooray, amazing.
- 57:09And with a sick child at home by the way,
- 57:12I might add, so super impressed.
- 57:14And you know, I feel like I need
- 57:17to take your course because I have
- 57:19introduced so much error in retrospect.
- 57:22But questions from,
- 57:23you know our audience on here,
- 57:30I have a question about
- 57:33the reliability of your respondent
- 57:36like when you said that
- 57:39there was that questionnaire
- 57:40that if if they scored highly it became
- 57:44basically invalidated their responses.
- 57:47So you know I think that how do you
- 57:55control for that? Yeah, I mean basically how,
- 57:58how could you you really assess and
- 57:59say you know I'm I've gather all this
- 58:01data but like how accurately does it?
- 58:04Absolutely. I think the problem is the,
- 58:07the power of the, I mean the, you know,
- 58:09with, I feel like with surveys you only
- 58:12get like a 10 to 20% response rate.
- 58:15Yeah, absolutely.
- 58:18Very complicated. An excellent question,
- 58:20very important. Michael and I
- 58:22have fallen victim to an invalid
- 58:27responses. We were collaborating,
- 58:28trying to work on a project,
- 58:30which I think should be
- 58:31resurrected by the way, Michael,
- 58:32but let's give it like 6 months.
- 58:37He was working on a measure
- 58:39of physician burnout.
- 58:39Right? Was it burnout?
- 58:41What were we working on?
- 58:44Physician trust.
- 58:45It was, it was a really important construct.
- 58:47But now I've fed bedside manner.
- 58:49Bedside manner. Oh my gosh. Yes.
- 58:53And it was really, really good.
- 58:55But we were, and we were trying
- 58:57to measure just by, you know,
- 58:59we weren't even compensating people,
- 59:01but we had too many invalid response patterns
- 59:06and realize that the data were correct.
- 59:09We can build in flags like if you are
- 59:12paying it, it's kind of like, you know,
- 59:13the whole captcha thing does, right?
- 59:16So you can build in things in between.
- 59:19You can actually build in Captchas
- 59:20in the middle of your survey.
- 59:22But I'll say things like,
- 59:23you know, to make sure our
- 59:25survey is functioning properly.
- 59:27Please select option C for this question.
- 59:30And that's actually a really good
- 59:32one to to reset if you've got
- 59:34different questionnaires from that
- 59:35strongly agree to agree and then
- 59:37the next bank might be reversed.
- 59:39I'll usually build in a page break.
- 59:41Ask something like that you know to to
- 59:43make sure that we're doing this correctly.
- 59:46Please select question three or
- 59:48to make sure our survey is is,
- 59:51you know, coding things correctly.
- 59:53Please select the the question you know.
- 59:56Please select the correct response
- 59:58for 2 + 4 or whatever and you'll
- 01:00:00just build in a couple of these
- 01:00:03little checks for attention.
- 01:00:04But they're actually ways to screen out.
- 01:00:07If you've been hit by, you know,
- 01:00:09people randomly responding and
- 01:00:10trying to get the completion code,
- 01:00:13very frequently,
- 01:00:13you know people will go on M Turk or one
- 01:00:17of the other kind of data collection.
- 01:00:19Services.
- 01:00:23And then, you know, people are out there
- 01:00:25just answering questions for money.
- 01:00:26So you need to make sure that you can build
- 01:00:29in little checks of attention like that.
- 01:00:33Probably have time for one more question.
- 01:00:35Julia, did you want to ask something?
- 01:00:37I saw you put yourself on on
- 01:00:41video. I need to see it.
- 01:00:44I was mostly just playing myself on video.
- 01:00:46I have I I I guess there's
- 01:00:49a lot on this topic,
- 01:00:51so I'm just curious just like if there
- 01:00:53is something that comes to mind for
- 01:00:54you to talk about other languages,
- 01:00:57whether that's like already validated,
- 01:00:59already gone through that process
- 01:01:01and translated and using other
- 01:01:02languages that we might not know.
- 01:01:04I'm in the process of doing
- 01:01:05a lot of like transition fact
- 01:01:06translation with cultural work and
- 01:01:08there's so many idioms and like
- 01:01:10the most commonly used things.
- 01:01:12So just curious overarching
- 01:01:14thoughts about multilingual.
- 01:01:15I mean it sounds like you nailed it,
- 01:01:17translation and back translation.
- 01:01:19Also consideration of the scaling
- 01:01:21itself because there are some cultures
- 01:01:24that the Likert scale or the four
- 01:01:26point scale really doesn't work.
- 01:01:28I'm not sure if the visual
- 01:01:31Analogue would work as well.
- 01:01:34So we've had kind of, you know,
- 01:01:37just collaborating with experts
- 01:01:38who are fluent in that language,
- 01:01:40ideally as their primary language
- 01:01:42and culture to weigh in on this
- 01:01:44and simply defer to them has been
- 01:01:46their approach for us when I've been
- 01:01:48involved in this particular process.
- 01:01:52Thank you. Great.
- 01:01:53Thank you, Marnie.
- 01:01:54Thank you again.
- 01:01:55Really appreciate it.
- 01:01:56Really appreciate your expertise
- 01:01:57sharing with us and as I said,
- 01:01:59doing it with somebody's to get home.
- 01:02:01So thank you.
- 01:02:03Thank you. Have a great
- 01:02:04day everyone. Thanks right.