How would you feel about throwing out 20% of your completed survey questionnaires?

It sounds like a crazy proposition doesn’t it, but without proper quality control methods in place, you may end up doing just that, according to industry research.  

So how can you ensure that the data you collect in your B2B survey is of the highest possible quality?

There are actually quite a few strategies you can use when building your survey, such as ensuring you have the right sample size and demographic, using an engaging format for the questionnaire itself, knowing which questions to ask and how to frame them, which analytics to apply to the final data….the list goes on.

Here’s a strategy you may not have come across before: red herring questions. 

These questions form part of the quality-control process by highlighting several types of respondent behaviour. There are red herrings questions designed to spot the survey speedsters, others which can catch respondents who may have inflated their credentials, those who lost their attention throughout the survey, and a further category of questions to spot fraudsters and bots. 

Let’s kick off by taking a look at how you can ensure that the data from the last part of your survey is as reliable as the data from the first part of your survey. 

In other words, how can you keep an eye on accelerating  speedsters, the ones that aren’t highlighted by the survey timestamp because they start off with good intentions and speed up as they lose focus.

Which one of the following is your favourite sport?

1. Football
2. Soccer
3. Circus trapeze
4. Baseball

The above question is known as an “outlier answer” type of red herring. These are useful for keeping an eye on the  level of customer engagement throughout the questionnaire, which is particularly important for a long-form survey. It is designed to catch any answers which indicate that the respondent isn’t paying attention properly, or maybe even picking answers at random. Obviously, the best practice option is to keep your survey short and sweet, but sometimes a longer survey is unavoidable, and these outlier red herring questions will allow you to double-check the quality of your overall data.

The next category targets the problem of people with a tendency to overplay their skills.

Which of the following programming languages have you worked with in the last three months?

1. Viper
2. SQL
3. Ruby
4. Swift

Only someone who works in computer programming would be able to answer this question without difficulty. A layman may be familiar with several types of computer coding, but would be very unlikely to be able to read through this list and pinpoint that Viper is not a computer language. This type of question is a booby-trap red herring. It is designed to pick up any respondents who have managed to get past the initial screening process by inflating their credentials or exaggerating their knowledge, people who don’t see any harm in a little white lie. They may very well pick out answers at random which will act as a red-flag. For someone who is super-motivated by the promise of a valuable incentive at the end of the survey, they may even consider it worthwhile to look up the right answers to the survey questions, but this behaviour would usually be picked up by the timestamp.

Next we have the bot-buster questions. 

These are not as relevant in B2B surveys, because you will be targeting your sample in a more precise way. A consumer survey usually targets a much broader sample and the danger is that there may be people who are drawn to the incentives scheme and decide it is worthwhile to create a survey bot. These can usually be picked up with a simple free-form question such as this one

How many days are there in June and July together?

Answer:

There is a clever type of red herring question which works as an internal double-check.

When designing a survey questionnaire, it is essential to bear in mind human nature and mindset. People very rarely have an accurate perception of their own habits, especially if they have been pre-conditioned to shun particular common habits. A good example of this is in the following pair of questions, one appearing early on in the survey and the other appearing much later in the survey:

How many hours of television do you watch each week?

1. 0-4 hours
2. 5-12 hours
3. 13-20 hours
4. More than 20 hours

How often do you enjoy box-set binges (i.e. more than 5 episodes in one sitting)?

1. Never / rarely
2. Once a week
3. Twice a week
4. Three times a week or more

If a person has been conditioned to consider watching television as frivolous, they may prefer to think of themselves as someone who only watches a couple of hours a week, and so they choose option a for the first question. The later part of the survey would hopefully catch them unawares and focus on a different aspect of their television-watching habits, nudging them towards an answer closer to the truth. If the two answers do not correlate, then it is likely that the data from that respondent is less reliable and may be worth rejecting.

So here’s another question: if red herring questions are so great and have so many important uses, why aren’t all survey builders using them?

Red herring questions can be a great way to address certain survey requirements, but you need to be careful. The most important thing to bear in mind is how they come across to the respondents. Here are a couple of quotes from the consumer website SurveyPolice.com which will give you some insight into how blatant use of red herring questions can be perceived as condescending and demeaning:

“I don’t know if it’s bad survey creations or if it’s another “trick” to see if we are paying attention but either way it’s stupid and rather insulting. These are surveys, not quizzes!”

“I swear I’m thinking of making up a cryptic red herring type email for when I write in to these survey companies.. make them read through it before they get to my actual question so they can see what it’s like. I’ll go off in one direction and then at the end get to my real question which is totally different.”

It is important to cultivate a positive relationship with your respondents, so keep checking how your questionnaire will come across (one of the great reasons for a soft-launch; more on that another day!)

Another reason to be careful when using red herring questions is because of the possibility of false positives and negatives. Take another look at the first example:

Which one of the following is your favourite sport?

a. Football

b. Soccer

c. Circus trapeze

d. Baseball

Can you spot the potential problems? It could be that your honest, bona-fide respondent just so happens to love all circus sports and regularly takes part in a trapeze class; unlikely, but possible! This would give you a false negative, flagging up a questionnaire which is fully legitimate. On the other hand, a speedster who is rushing through the questionnaire clicking answers at random may miss the red herring option altogether, leading to a false positive, and allowing the data to pass through for analysis.

At Global Surveyz, we are familiar with the difficulties inherent in using red herring questions, and we have used our expertise to develop strategies for overcoming the problems, whilst still being able to garner the full benefits of red herring questions. We believe that the most important element in crafting a survey is ensuring the highest possible quality of the final data. 

To find out more about working with Global Surveyz on your B2B survey, call our team today and make sure your company can separate out the white lies from the red herrings!