There can be big discrepancies between what people tell us in surveys and the truth, because respondents have a tendency to over and under claim certain answers. For example, Kantar has found that respondents are more likely to say they do socially desirable activities (like vote, recycle or buy organic food) and less likely to say they do socially undesirable activities (like smoking, watching reality TV or eating fast food) than they really do.
And whilst a lot of this misreporting is not necessarily conscious behaviour, it is important to limit dishonesty in your data set. Here are ten techniques you can apply to achieve more reliable data you can trust.
1. Use online panels that apply antifraud measures
There are increasingly sophisticated techniques, often utilised by organised groups, to fraudulently complete surveys. However, with this advanced technology comes the opportunity to build antifraud measures to tackle them. By employing barriers like identity validation, anti-bot testing and machine learning that predict, identify and act against the fraudulent behaviour of respondents, the impact fraudulent activity has on your final data set can be minimised. At Kantar we have also pioneered a range of honest detection protocol which are a series of questions that can determine the honesty of our panellist.
2. Take care with your pre-screening
If you are trying to target a particular audience it’s important to take care not flag that to the respondent what the survey is about. This might involve starting with one or two questions quite unrelated to the topic and presenting the question which identifies your target audience in amongst a set of competitive alternatives. For example if you were trying to target a group of people who watch a particular TV program, instead of asking people directly if they have watch that program or not you might present it in list of various other TV programs and ask respondents which they have watched.
3. Add opt-in selections for screening list choices
When presented brand choices, people can be tempted to pick the more prestigious brands over the brands they actually use. Why? Because it comes off more impressive to others.
A way to reduce this dishonesty, hide the prestigious brands in your list behind an “other” option. This requires respondents to proactively opt-in to selecting a premium choice. For example, you want to find out what make of car people own but discourage those who may be tempted to say they own a luxury car when they don’t. Rather than list all the brands in one list, show only the popular brands in the main list and reveal the luxury car brands if “other” is selected. If someone actually owns a luxury car, they will want to select “other,” while those who do not will be far less likely to.
4. Integrate excuses into the option choices
If you ask people outright if they do something or not, some people may be embarrassed to admit the truth. For example, asking a respondent if they smoke or if they recycle. Someone who only has an occasional cigarette and is trying to quit might be tempted to answer “no.” Someone who would recycle but doesn’t have a recycle bin might still answer “yes.”
We’ve found that by embedding common excuses for doing or not doing something into the question, it will enable people to give more truthful answers. For the smoking example, try providing option choices such as “I only smoke occasionally”, “I am trying to give up” or “I have recently given up”.
5. Use deflection priming
Deflection priming is a method where you ask a related question first that sets them up to respond more honestly to the question you really care about. For example, wanting to know if someone has a university education or not. This is a question that many people are tempted to lie about. However, if you ask someone if they enjoyed school or not first, and they say “no”, they will be slightly less likely to tell you that they went to university when you ask that next because the previous answer helps to explain why.
6. Apply competitive choice-based prioritisation
If you ask someone a direct question about a socially desirable behaviour or attitude, some people might be embarrassed to admit to not doing it or to holding a less socially desirable point of view. One way round this is to ask the same question in reference to a range of equally socially desirable options, and then ask people to pick out the options they agree with.
For example, instead of asking someone if they are concerned about global warming, ask what issues they are concerned about and provide an option list. By having multiple option to select from, they will feel less guilty about their response.
7. Implement comparative anchoring
Respondents tend to be very enthusiastic about new product ideas when you present them in a research setting. In a typical new product evaluation, you might expect that 70% of people will say they would consider buying the product – not very useful when you consider that less than one percent of new products ever gain more than five percent sales traction. A solution to this is to ask the question in a more comparative context. Instead of asking “will you buy this”, you might ask them what they currently buy and then ask them how likely they would be to buy this new product instead.
8. Try projective techniques
One of the most effective ways to understand real behaviours is to tackle it indirectly. Instead of asking people about their own behaviours or attitudes, ask them to observe and report on the behaviours and attitudes of people they know.
This methodological approach is derived from a classical psychology experiment. One group of students was asked whether they thought they would tidy up after a meeting, and 50% predicted they would. Another group was asked to predict how many students they thought would tidy up and they guessed 15%. The actual number of students that tidied up was 13%. This learning can be applied in the way you frame your survey questions too.
9. Implement behavioural questioning techniques
One of the biggest changes in our approach to gathering truthful insights has been a shift away from asking attitudinal questions and toward asking about behaviours that reflect those attitudes. For example, asking “Do you think it is important to sit down and eat meals with your family?” versus “How often do you sit down to eat meals with your family?” With the second question, we find more realistic responses.
10. Use open-ended questions
Asking prompted questions in surveys can embed ideas that bring about social-desirability bias. One of the best ways to get closer to the truth is to ask an unprompted open-ended question and examine what they say, or more interestingly sometimes, what they do not say.
Learn more
Download the full whitepaper “Getting Closer to the Truth” here or watch Jon's on-demand webinar for more on how you can apply these techniques and achieve honest answers.
For monthly survey design and sampling tips direct to your inbox, fill out the form below to subscribe.
