Driving Impact Through The “Observer Effect.”
I am going to make a bold prediction: that each of us will be asked in the next year to take a real stinker of a survey. You know the type: too many questions (or too few), awkward wording, confusing rating scales, and to top it off, having no idea about what happens with the results. In this article, we highlight one key area that survey teams and committees should consider before embarking on a new survey process: the “Observer Effect.”
The observer effect occurs when the ACT of measuring something affects the results. Some examples: in particle physics, the act of studying an electron results in the electron changing its path; in anthropology, the act of introducing an anthropologist into a secluded tribe results in a change in the tribe’s social structure; and the act of counting the cookies in the Rumery’s household cookie jar usually results in one less cookie.
In survey research, the observer effect can be your best friend, or your worst enemy. Many times survey researchers want these observer effects and may even count on them. For example, surveys are often used in part to communicate new expectations to those taking the survey, such as with 360 degree-feedback surveys where the act of measurement can lead to the setting of new and desirable expectations of manager behaviors.
However, observer effects can also lead to unintended and often negative effects. In particular, imagine the effects caused by one of those stinker surveys.
1) Fatigue. Raters filling out surveys that have too many items and items that are hard to interpret.
2) Confusion. Raters misinterpreting the messages that the survey researchers are trying to communicate.
3) Frustration. Raters getting their hopes up that change will occur based on the results, only to see that nothing changes.
4) More Surveys. Confronted with results that are difficult to interpret, survey researchers have one solution: more surveys!
So what can you do to manage observer effects? The answer is: survey like you mean it! Take the time and effort needed to put together a thoughtful survey that communicates the messages you want and that enables you and others to take action.
Some tips:
- Know the key messages you want to convey. Build a survey process that reinforces the key messages and avoids unintentional messages.
- Build a clean survey. Draft items that are easy to understand and pair the items with an appropriate rating scale. Be mindful of the number of items you are asking. Make sure to pilot the survey with a representative sample of raters.
- Be thoughtful when interpreting results. Take the time needed to fully analyze the data before interpreting the results. Look for clues regarding how different rater groups (raters in Asia, for example) may be interpreting items in an unanticipated way or using the rating scale differently.
- Follow up, repeatedly. Finally, communicate frequently with raters about the results of the survey and what actions are being taken based on the results. Get people involved in developing solutions to issues raised with the survey.
In closing, we hope you find this article useful. If you are currently working on a survey and struggling with some of the issues outlined here, let us know! Surveys can be real stinkers, but if done right they can also drive real change in your organization. But to have this kind of impact, you need to survey like you mean it!