I’ve been learning all about cognitive interviewing at Nat Cen this week…
Read on to find out more about what it is, how to do it, and how it can help your business…
What is cognitive interviewing?
It’s a qual method used to help develop and test quant questions. It figures out how people are interpreting and answering questions to help put together better questionnaires.
Can it be applied beyond that?
Absolutely, because its focus is on how people process communication, so it can be relevant for looking at the written word, advertising, web-design and much more
How does it work?
Respondents go through (parts of) a questionnaire as if they were being asked these questions in a real interview. Their responses are then unpicked to understand how they processed the questions and how they arrived at their answers. It uses special qual techniques to reveal what is going on cognitively…
What are the key techniques?
There are two key methods (and one I like..)
- Thinking aloud
- Probing
- Vignettes/ Little Stories
Thinking Aloud
The ‘Thinking Aloud’ technique gets respondents to expose mental processes whilst they are happening.
…imagine I asked you to tot up the number of windows there are in your house and give me the answer. In order to do that you need to do some mental processing… You might visualise the different rooms in your house, and count the number of windows room by room.
Thinking Aloud gets you to verbalise that process. So a respondent would say out loud (when totting up…)
Ok, so there’s the window in the hallway, and then one in the front room. In our kitchen there’s a window over the sink and there’s two in the dining room, upstairs there’s one in each bedroom, and we’ve got a three bed house, and then there’s a small window in the separate toilet and one in the bathroom. So that makes… umm…10 I think?
Thinking aloud about ‘Thinking Aloud’
It’s a technique that I have pulled out of my qualitative kit bag from time to time – it’s always useful to suss out what’s going on in someone’s mind…
But I think Behavioural Economists might have some challenges to the approach too…
Kahneman, the Daddy of Behavioural Economics, has developed the concept of System 1 and System 2 Thinking. System 1 Thinking is quick, intuitive decision making, and System 2 Thinking is more considered and conscious mental processing. The Thinking Aloud technique relies on System 2 type thinking in order to ‘reveal’ thoughts that could have actually been very intuitive and short-cut in nature. Some Behavioural Economists and Psychologists just don’t believe we know our own minds, and so they would reject Thinking Aloud out of hand.
Personally, I would use the technique to explore questions that require System 2 type thinking. (Questions that do get you to go through a mental process, like counting, or weighing things up). Sometimes people really can ‘revisit’ or reveal what’s going on in their mind. Importantly sometimes (I believe) they really can’t. So, I think use as appropriate.
The other techniques in Cognitive Interviewing…
Probing
Specifically, the aims of the cognitive interview are focused on four key areas and probes are developed accordingly. Cognitive interviewers are interested in:
- Comprehension – probing to see how well an idea was understood, or what was understood by an idea. The standard (and very effective) approach here is to ask someone how they would put the question in their own words or explain it to another person
- Retrieval – how well a respondent was able to recall or remember the issue they are being asked about. The thinking aloud technique is probably relevant here, but so too is probing on how easy or difficult recalling was, and how they went about thinking back in time, how retrieval happened…
- Judgement – probing on how confident a respondent felt about their answer, and how they came to formulate their own answer (thinking too about the filters they applied to their answer.
- Response Process – mapping their answer to the options they are given in the questionnaire
And finally…
Vignettes/ Mini Stories
This technique is more useful in the early stages of putting together a questionnaire (before it’s been tested). Say, for example, you wanted to see how respondents mentally mapped the idea of ‘work’ to see if this could be used as a universal term in a questionnaire, you could develop a number of little stories about different people to explore what ‘working’ means. An example…
Jan is a mum who is at home every day looking after her three young kids, cooking and cleaning for the family, taking the kids out to the park, making sure the household run smoothly – do you think of Jan as working etc…
Thinking Aloud about Cognitive Interviewing
Cognitive Interviewing is perhaps a bit of a fancy term for what qualitative research (at its best) does anyway, but now it’s been packaged up, and given a specific purpose (looking at questionnaire development and communications materials).
Maybe it’s no bad thing to have a label like this, so that we can say… “let’s do some Cognitive Interviewing to understand how our leaflet/ advert/ communications piece/ questionnaire is being processed…”
I suspect too that whilst Academics and Civil Servants are using Cognitive Interviewing to test big scale social research questionnaires, commercial market research agencies may not have discovered the benefits of Cognitive Interviewing yet.
From my two days learning and trying it out at Nat Cen, I think Cognitive Interviewing is well worth doing – for the development of better quant questionnaires, and in general terms to understand better how people process what is being put in front of them. So I’m up for a bit of Cognitive Interviewing!
If you’d like to know more, as always, do get in touch!
Kath
