In the two previous posts we talked about why you should use an even number of items in your response scales, and when to allow “N/A” and “I Don’t Know” as options.
Now let’s talk about the optimal number of response options to use in your opinion-related, scaled questions to get the most bang for the buck.
(Questions about fact don’t apply here as there isn’t a judgement call required – the answer is the answer.)
A battle between the forces of Accuracy and Validity
The choice of the best number of options (on a scale) is a balance between two opposing forces which I’ll call:
- “Variance/Range” (the Champion of Accuracy), and
- “Just Noticeable Difference (JND)” (the Champion of Validity/Truth).
So then on opinions you’re looking for a response scale with the widest range possible (which helps with the stats), while keeping the range narrow enough to maintain a noticeable difference between each choice.
Variance: Your stats and reporting want as many options as possible
The more options you have, the more “pin-point” and accurate the response might be.
This is why an “Employee Net promotor Score” (ENPS) “I would recommend working here to a friend” is on a scale from “1 to 10” rather than “Yes/No“.
JND: BUT, too many options can make your results misleading
There is an entire area of psychology and statistics regarding the Just Noticeable Difference (JDN), but in essence it comes down to this: if I were to ask you to grade how white your shirts are when they come out of the wash on a 0-100 scale, can you REALLY tell the difference between an 87 and an 88? If not, then the scale is two wide and you’re getting a false sense of a range of opinion that just isn’t there. (In statistics this is called “False Variance“.)
So what’s best?
In the above example there is no way for a normal human to tell the difference between 87 and 88, but if I ask the same question on a 2-pont scale (“white” and “not white”) then the results become useless, and on top of that the respondent would have no idea how to answer.
So start here (using an even number of responses of course):
- If you use a 4-point scale, can people tell the difference between 3 and 4? If so go up to 6.
- How about the difference between 4 and 5 now? Can you still differentiate? If so go to 8.
- How about the difference between 6 and 7? …
- Etc. etc. until you can’t determine a noticeable difference between option choices.
We ourselves generally choose a 6 response scale for questions on opinion, as we believe it gives the highest range of responses, while still maintaining a JND.
To recap
- The choice of the number options is a balance between two opposing forces:
- Accuracy (“Variance/Range“), vs.
- Validity (a “Just Noticeable Difference“)
- From the “statistical” point of view, the more choices you have the more “Accurate” your results, and the more statistical measures you can use.
- From the “human/psychological” point of view, the fewer choices you have the more “Valid” the difference between choices may become.
- To determine how many you should use, you want the largest possible range of options where you can still determine a difference between them.
Hope this was of help, but as always give us a shout if you would like to discuss!
Next up – Avoiding unactionable questions
Thanks for reading, and if you’re interested in discussing a survey for your organization call us at 1-604-219-7876, email us at enquiries@veritaengage.com, or just book a discovery call for a one-on-one chat.