Cognitive testing
The success of any survey depends to a large extent on the quality of the questionnaire. We use cognitive testing to assess the effectiveness of a survey question in eliciting the required information from its target audience.
Conducted as part of the developmental stage of the research process, cognitive testing allows an investigation into the way people understand, mentally process and respond to the question under scrutiny to explore whether the question, response options and accompanying instructions are interpreted in the way intended. It helps to answer questions such as: Is there a common understanding between researcher and respondent which will result in valid data? Are there any hidden barriers to understanding questions? Any areas that respondents find difficult or misunderstand can be improved before fieldwork commences, thereby improving the overall quality of survey data.
Cognitive testing typically involves carrying out in-depth, semi-structured interviews with a small sample of respondents, prior to conducting the pilot of the survey. As part of this, the question (or questions) is tested in conditions which, so far as possible, mirror those in which they would be administered in ‘reality’; for example, when testing a range of questions developed for the 2011 Census we asked people to complete the questionnaire as if it was Census day, without assistance, before interviewing them about the experience. Unlike more traditional research practices, there is less emphasis on the specific response to a question; instead, interviews are designed to elicit an understanding of how respondents interpreted the question and why they gave a particular response: What came into your mind when you were asked that question? Can you put the question in your own words? Was the question easy or difficult to answer? Is the layout helpful or off-putting? Why do you think the question is being asked?
The utility of cognitive techniques is not restricted to testing questions designed for use in survey questionnaires. They can also be used to test other documents, including claim forms (for benefits, allowances and pensions) and other official documents: we recently tested the usability of the ballot paper design that had been developed for the 2012 Local Government Elections to ensure that it allows voters to easily mark their preferences in the way they intend.
For further information, please contact Steven Treanor or Lorraine Murray.
Further reading
Research Synthesis: The practice of Cognitive Interviewing
Testing Survey Questions: New Directions in Cognitive Interviewing
Analysing Cognitive Interview data to Improve Cross-National Survey Questions
Cognitive Interviewing – A “How to” Guide: Reducing Survey Error through Research on the Cognitive and Decision Processes in Surveys
Cognitive Interviewing: A Tool for Improving Questionnaire Design