Should I “randomize” the order of my survey questions?

In this post I’m going to cover the pros and cons of randomizing the question order in your employee survey, and why … well … it depends.

In the end the intent is to balance clarity and ease-of-use on the part of the respondent against any response biases or data validity issues that might be caused by your choice of which way to go.

What do I mean by “randomize the order”?

So what I’m NOT talking about, and what I feel you should generally not do in an Employee Survey:

• DON’T randomize the question order for all questions for all respondents – everyone should see the same survey. Unless you are talking about very large sample sizes, randomizing the question order just adds a new level of uncertainty to the results. (“But what about “Early Question Bias” you say? Happy to discuss this at length, but in this case I feel the cons outweigh the pros.)
• DON’T randomize or change the scales or options. Conceptual ideas of “positive” and “negative” should always flow in the same direction throughout the survey. I could write an entire post on why this is so – but in short it would again add a level of confusion and uncertainty to your results. (… and anyone who makes the argument that “it forces the respondents to concentrate” please send them to me – I will have words with them.)
• DON’T spread the demographic questions throughout the survey (ex. Location, Department, Age, Gender, etc.). Keep them in one place if you can. (At the beginning or end?  I’ve addressed this in a separate post as “it depends”.)

Doing any of the above might make the survey results suspect from a “stats/validity” point-of-view.

The pros and the pros

So let’s get into it … now that we know we are ONLY talking about the pros and cons of randomizing the order of the non-demographic questions in employee surveys, but everyone gets the same survey. Which is best?

I generally have a preference, which I’ll discuss in a sec – but in truth it really depends on a case-by-case review of the pros and cons below that touch on aspects of psychology, normal human behavior, accuracy of memory (“recall bias“), and some basic stats.  (I will get into the many types of survey bias in a later post in excruciating detail – so you might what to skip that one – I won’t take it personally.)

PROS of randomizing the question order:

1. Randomizing might avoid “context bias”. In other words are there concerns that the respondents will get in an intellectual or conceptual “rut” in their thinking by putting certain questions together?
2. It might avoid some degree of “favor bias”. If I like my manager, and all my “my manager” questions are grouped together, I might barely read the questions and give them all a “6” (“Strongly agree”).
3. It might avoid “pity bias” or “let’s even things out bias”.  If questions on my manager or colleague are grouped together, some people will make a calculation as to whether they are being “too nice” or “too harsh” overall. If I have answered negatively to the last 3 questions regarding HR then I have to give them SOME positive feedback right? They’re such nice people after all.
4. Spreading the question groups out can mitigate “order bias”. As we are all human, the first questions in a survey will always get more consideration on the part of the respondent than the last.

PROS of  NOT randomizing the question order:

1. If it’s a very long survey you can use “question order bias“ to your advantage by placing the most important questions first.
2. It can make the survey a bit easier to read and fill out overall.
3. Grouping the questions can provide clarity of “concept area” and “context” when there are multiple questions on the same subject. You can even add question area titles on the survey page like “About my manager…” to save them having to read it over and over, and to save the respondent the intellectual energy of having to revert back to the context. (This may lead to favor or pity bias as above though.)

What do WE do?

Though we’ve done both given the particulars of the survey and respondent group – we generally lean toward not randomizing the questions, but grouping them by driver (ex. “benefits”) or area (ex. “The senior leadership team …“)

Every survey is a little different though – and our forte is customization after all – so we ourselves might group or not group depending on what we think is best given the above. 

Something I’m missing here?  Totally disagree with me and want to vent?  … or just want to chat on this fascinating topic?  Give me a shout if you have any questions or comments.

Next topic?

Next up: “Employee Survey Pro Tip Series 18 – What “Demographic” questions should I use, and where should I put them in the survey?.“

Thanks for reading, and if you’re interested in running a survey for your organization call us at 1-604-809-8570 or email us at enquiries@veritaengage.com.