Uncategorized

How Long Should a Survey Be

How long should a survey be?

A questionnaire/survey should be formed as short as it is possible and you should ensure that the researcher is able to collect valid and reliable answers to all the questions asked of each visitor.

A survey or a questionnaire length is often referred to as the length of interview or LOI, which is typically measured by the average time that a respondent takes to complete the whole survey or questionnaire when it is managed in a similar aspect to that in which it will ultimately be fielded.

To be honest there is no limit of objective on may be or any limit for maximum number of questions that it may contain.

The mental pressure that a questionnaire places on a visitor will vary not only on the basis of an individual respondent but also it varies on the basis that how a questionnaire itself has been designed or customized, how and when it is administered, what kind of questions it contains and many other factors.

What should you consider when writing a questionnaire?

Actually there is no hard and fast rule for governing a questionnaire length but there are few general guidelines or rules that a good researcher will always try to keep in mind while customizing a survey or a questionnaire for common market research contexts.

These rules take into account the fact that as survey length increases, the response given by the respondents decreases.

Also, the response validity, that is, the data quality eventually decreases to the point which any further questions of the visitor are not worthwhile, as the individual’s attention span and the ability of giving reliable and valid answers has been totally exhausted.

Common rules of thumb How Long Should a Survey Be

From the starting times of academics professional research practice, the researchers have found that the following rules of thumb below for length of interview which generally ensures that the questionnaires are kept sufficiently short to gather high quality responses from all the respondents to all the questions,

while still offering sufficient room for thorough questioning on most of the topics of interest to the researchers and their clients.

In general, the length of the questionnaire according to length of interview (LOI) should be limited to: 

  • Time limit for self administered online surveys should be of 10 to 15 minutes.
  • Time limit for semi structured telephone interviews should be of 20 to 30 minutes.
  • Time limit for in depth, structured telephone or face to face interviews should be of 30 to 60 minutes.

The benefits of pretests

In research circumstances where the above rules are not applicable or relevant, the perfect way to assess whether a given survey is too long or not is simply to pretest it before it is actually fielded, that is,

by managing it to a small sample of test visitors and gauging their ability to offer high quality responses to every question it contains within a reasonable time frame.

If your pre testers can complete the entire survey within an identical or similar context to that in which you are actually planning to field it then the chances are, most of your ultimate target respondents will be able to do the same.

How much time do the visitors willing to spend on your survey?

How much time would you be preparing to spend for completing a survey distributed by your Human Resources team about employee satisfaction? a customer satisfaction survey about a current shopping experience?

Or from your friends who hosted an event that you attended and you want to give your feedback?

Firstly, understanding your audience while constructing a survey is very important and it also helps you to take decision about a survey length and content granularity.

As each audience who are responding to your survey have different motivations about responding to survey, so their tolerance for how long a survey is will vary.

What we analyzed:

Actually we wanted to know that how the length of the surveys as measured by the number of questions impacts the time which respondents spend while completing the surveys.

We kept in look in some random sample of roughly 100,000 surveys which were of 1-30 questions in length, and we analyzed the median amount of time that visitors took to complete the surveys to understand the relationship.

After analyzing, we came to know that in ideal circumstances and over a large, randomized sample of the responses, the average time it takes for answering a question should not vary depending on the length of the survey,

so in conclusion a linear relationship between the number of questions in a survey and the actual time it takes to complete a survey should exist.

What we learned:

Nowadays it is not a surprise that the relationship between the number of questions in a surveyor a questionnaire and the time spent by a respondent on answering each question is not linear.

It is actually inversely proportional with each other, which means, the more questions you ask on your survey, the less time your respondents spend, on average, answering each question in your survey.

When the visitors in your survey, in methodological term, start ‘‘satisficing’’ or “speeding” through your survey, then the quality and the reliability of your data can suffer.

After analyzing, we found that, on average, the respondents take just over a minute to answer the first question in a survey which includes the time spent reading any survey introductions, and spend actually about 5 minutes in total in answering a 10 question survey.

However, the visitors take more time per question in case of responding to shorter surveys as compared in answering the longer surveys.

If you have question in your mind that do the longer surveys always contains less thorough answers?

The answer is not always. It actually depends on the type of survey, the audience and the relationship of the audiences to survey or among other factors.

But, the analyzed data shows that the longer a survey is, the less time the respondents spend on answering each question.

It is counted that for surveys longer than 30 questions, the average amount of time the respondents spend on each question is nearly the half time as compared to on surveys with less than 30 questions.

On addition to the decreased time spent while answering each question as the surveys grew in the length, we saw the surveys leaving rates increases for the surveys that took more than 7-8 minutes of time to complete with the completion rates decreasing anywhere from 5% to 20%.

The allowance for lengthier surveys were greater for the surveys that were school or work related and decreased when they were customer related.

What this mean for you:

You must take the survey completion time into consideration when you design or form your next survey.

Make sure that you are maintaining and balancing your audience profile and the survey goals with the total number of questions which you are asking so that you can get the best data possible for the decisions that you need to make, and if you do write a survey which has a low response rate then you must make sure that you send it to enough recipients to receive a good sample size.

How many days does it take for the respondents to respond to your survey?

After creating the survey, start checking by sending an email. Check the survey posted to Facebook, Linkedln and Twitter. Also check by logging in to SurveyMonkey to see the results but, the question is: How Long should you really wait before beginning to analyze your survey result?

How long, on average, does it actually take for the majority of the respondents or visitors to complete a survey? So we wanted to find out how patient you have to be to know the result.

What we analyzed:

We choosed a random sample of about five lakh individual respondents who got survey invitations using the SurveyMonkey email gatherer in the year 2009 and 2010. We kept out the first week of January, the last week of November and the last two weeks of the month December to balance the United States holidays.

We restricted our analysis to the surveys that collected the responses between 50 and 500, so if any survey got a thousand or millions of responses (yes, some receive millions of responses) will not skew the results.

And we actually include those responses to surveys that are created by SurveyMonkey professional accounts because free accounts can only have up to 100 responses per survey.

What we learned:

Maximum of the responses to the surveys using an email collector were collected in the first few days after the email invitations were sent:

  • Within 1 day the responses were gathered was 41%
  • Within 3 days the responses were gathered was 66%
  • Within 7 days the responses were gathered was 80%

And only 11% of the responses were gathered during the second week of the survey period, and another of 4% during the third week.

What does this mean for you?

It does not matter if you peek during day one but you might want to wait until day seven to run a proper and a significant analysis, and if you are trying to create more responses then you might prefer sending out a remainder email after a week to see if that email triggers a bump from your laggards.

What happens when you add one more question on your survey?

It is definitely not a surprise that if you ask more questions, the fewer respondents who started the survey or the questionnaire will complete the full questionnaire.

But it will be better asking fewer questions in case, if you want or need to receive responses from a certain number of audience, and you have a limited visitor or sample size.

In many cases, the surveys cannot be completed with just a few questions, but if the response rate matters for you then keeping the survey compact can definitely help.

We analyzed the response of random 100,000 surveys to know the drop off rates with the increase length of the survey, which were conducted by SurveyMonkey.

We keep an eye on both the number of questions and the number of pages to understand the drop off rates from start to finish for the surveys from 1 to 50 questions.

However, the limit of qualifier was that, the visitor will have to submit a response on at least one page so that we conduct our study, we looked at about 2000 surveys with 1 question, 2000 surveys with 2 questions, 2000 surveys with 3 questions and many more, which ends on looking 2000 surveys with 50 questions.

Result we got after analyzing

As expected, the survey which contains more number of questions have the higher respondents drop off rates from start to the end. The analyzed data suggests that if a respondent begin answering a survey, the sharpest increase in drop off rate occurs with each additional questions up to 15 questions. 

What does this mean?

If you are trying to optimize for more completed survey responses then you must try to keep your survey as short as you can. You can skip the logic which will help shorten a survey to allow the visitors to navigate to onlt to the survey questions which are relevant to them.

Conclusion

We can say that the shorter survey are definitely better and gives more responses than the longer ones, and they should not be longer than 15 minutes.

The main concern when the survey long is the poor data quality and the fatigue effects not the survey participation.

People appear to have the thing in them of completing something which they start, but if your survey goes too long for the respondents then the respondents focus on just completing the survey that they started rather than providing the thoughtful answers to your questions.

Arkadip

Published by
Arkadip