Survey length is one of the most important considerations in quality research – generally the shorter the better, but you also have to balance this with meeting your objectives.
Survey length is one of the most important considerations in quality research – generally the shorter the better, but you also have to balance this with meeting your objectives. This is perhaps one of the most difficult challenges in doing research. While it’s generally the case that you want to get as much information as you can in one project, long surveys generally lead to poorer quality of response and subsequent data. Consider this; longer surveys have been shown consistently to:
- Lead to respondent fatigue which means lower levels of attention to your questions and therefore poorer data
- Lead to more terminations amongst those who have been good enough to start your survey
- Lead to a poorer experience with your survey and therefore the brand or organisation they represent
- Lead to lower response rates in the future making subsequent research more difficult.
It’s important to remember that all research relies on the good will of respondents – longer surveys generally lead to a diminution of this goodwill and that’s not a good thing for potential respondents or researchers.
It’s also important to consider that actual survey length is a bit of a nonsense because it is based on an average and we all own 1/7 of a Labrador on average! That is, in all surveys there will be respondents that finish faster or slower than others, which you need to consider, so try and define it in terms of a range such as 5-minute blocks – e.g. less than 5 minutes, 6-10 minutes, 11-15 minutes etc.
So what’s the best guideline for survey length? Generally speaking try to keep your surveys to 5-15 minutes and offer incentives for longer surveys. A short 5-10 minute survey is usually not an issue and most people will do these sorts of surveys given they are engaged with the subject matter or offered an appropriate incentive. Longer surveys can of course be conducted but you need to think hard about who you are researching and what’s in it for them.
The subject of incentives is a contentious issue in research and probably beyond this discussion (and deserving of a separate blog). Generally there are those that think that incentives are cash for comment and create a bias – however the alternate argument is that the lack of incentives also creates bias because it means some you miss out on certain segments of people – usually those who are more time poor and not willing to do a survey for nothing. The decision should always be made on the basis of three considerations:
1. Length of survey
It’s a good idea to consider offering an incentive for surveys over 10 minutes and to definitely offer an incentive for surveys over 15 minutes. For longer surveys over 20 minutes you really need to consider an incentive that is commensurate with the time taken to complete.
2. Type of respondent
If you are talking to a general consumer then you can probably offer an incentive such as a chance to win a pool prize or a token cash incentive that shows that you appreciate their time. If you are speaking to a senior person in an organisation or other time poor respondents then you really need to consider an incentive appropriate for their time.
3. Level of engagement/involvement
Low involvement categories or subjects generally mean respondents will give you less time while you might have a bit more room to move for high involvement categories – that is you have a bit more room to move. That is people are willing to participate longer in subjects that they care about like social issues or where they spend longer making decisions about such as cars and holidays. It gives you a bit of room but don’t abuse it!
Optimising Survey Length
Firstly optimise your questionnaire by ensuring that each question is matched to a specific objective. Think of it as each question earning the right to be there. If it’s a nice to know and you don’t have room it’s out!
Although single source data (all data collected from one respondent) is a wonderful thing from an analysis perspective it’s not necessarily from the respondents’ who have to sit through long and tedious questioning. If your objectives can’t be covered with a survey of suitable length then consider running a split sample (two identical samples covering different things) or consider running two separate studies over time with objectives split between them.
If your survey is a bit longer than you are comfortable with think about making the survey itself more engaging by using techniques like visually interactive question types using drop and drop, image choosers, image ratings, interactive videos, sliders and the like. Most survey software platforms now offer these but you need to be careful as sometimes they too can add to the task and don’t always work on mobile devices – especially small screen devices like phones.
Giving respondents the opportunity to save as draft is also a useful technique if you are worried about the length of your survey. Basically you give respondents a come back later code which enables you to remind them (and thank them for participating so far) to come back later and finish the survey in 2 or more sittings without losing any continuity in the data.
Finally it’s a very good idea to tell respondents realistically how long the survey will take and their progress throughout the survey via a progress bar. Designing you survey into logical sections also can help as can thank and encouraging respondents throughout the survey journey.
Hopefully these guidelines and tips and tricks will help you create shorter more engaging surveys. It’s our experience that questionnaires tend to always end up longer than they should be so making sure you are aware of survey length and always do what you can to reduce and optimise your instrument will help create better responses and better quality data that your decisions are going to be made on. Best of luck.