“Do you have time to answer a few questions?” is a phrase we have all heard before. It is one of several typical lead-in lines for a survey. Others just jump right into the survey, trying to avoid giving you an opportunity to decline the survey.
Some people will take their time and answer every question with care, while others will make up answers that don’t represent their opinion. Another portion of the population will simply not answer any surveys. It makes you wonder how valid surveys are.
Surveys of an entire population, such as, a neighborhood or an organization, would likely be relatively accurate. These surveys reach out to each individual; thus, creating an entire composite for the entire group. Also, these people have a stake in answering questions honestly. There still can be reasons that the data is distorted. For example, if the survey is about management of a company and people feel that their answers are identifiable, they may not answer honestly.
Although surveys can be designed to sample a small number of people and still be representative of the population, this often is not the case. The reason for this is because great care must be taken in selecting those who are surveyed and because equal care must be taken in selecting and stating questions in the survey.
Surveys must question individuals fairly and must allow for individuals that do not know anything about the topic. For instance, one day my husband had a call where he was asked if he knew anything about longhorn sheep. Despite his answer stating he did not know anything about them, they continued to ask him specific questions that required knowledge of longhorn sheep. Of course, they were questions that required you to select an answer and “I don’t know” wasn’t an option. Clearly, his opinion and that of anyone else that knew nothing of longhorn sheep provided no value and rendered the survey invalid.
Questions also cannot be loaded or stated in a way that is looking for a specific answer. Sometimes this is accomplished by wording a question in a way that almost no one would disagree with it. By itself, this is okay. However, these statements are often re-stated along with the percentage in agreement in a way that gives a different impression that the actual statement in the question. Additionally, surveys can be written such that the question clearly supports one choice over another. Political surveys are probably the biggest offenders of this technique. However, if the survey is designed to clearly support one candidate, it is not valid. This is why a poll done by a particular party is not the most accurate poll.
Questions also must actually be meaningful to the survey. For instance, one health survey asked the question, “Have you ever been told by a doctor that you have <disease/condition>?” The problem with this question is that although a doctor may have told you that you had the disease or condition, the doctor or a specialist may have later decided that you did not have it. There is no value in this answer although it is likely being used to count or categorize people with the given disease or condition.
This same survey was only restricted to people on landlines and you were considered an invalid participant if you answered on a cell phone. I suppose this was done to ensure that you lived within the state. However, this means that entire populations of people were being left out of the survey, including younger adults and poorer individuals, who tend to have phone service only via a cell phone.
The next time you are asked to participate in a survey, pay close attention to how they reached you, the questions they ask, and the way they ask them. All of those things will tell you a little something about the validity of the survey.
Likewise, when you hear or read the latest survey results, pay close attention. It is important before you believe in a survey that you know the source and the quality of the information. Ask yourself if the organization doing the survey has a stake in the outcome. It is also helpful, when possible to find out method of data collection, the survey size, and the specific questions that were asked.
A survey is simply a survey. Pay attention and observe. Then draw your own conclusions.