To introduce students to factors that can affect the accuracy of opinion surveys.
This lesson is part of a group of lessons that focus on the social, behavioral, and economic sciences. These lessons are developed by AAAS and funded by the National Science Foundation Grant No. SES-0549096. For more lessons and activities that take a closer look at the social, behavioral, and economic sciences, be sure to check out the SBE Project page.
A prominent feature of modern discourse is the opinion survey or poll, in which the opinions of a population are reported based on responses obtained from a sample of that population. No survey is perfect, but some are more reliable representations of group opinion than others. The best surveys are conducted using methods based on scientific principles and best practices. Students therefore need to learn how to evaluate opinion surveys—how to judge in each instance whether the data has been collected and analyzed in a manner that reduces error.
By the end of middle school, students should have had experience in collecting data, compiling statistics, preparing an analysis, and analyzing the statistical arguments of others. Of particular importance to opinion surveys, they should know that the larger a well-chosen sample is, the more accurately it is likely to represent the whole and that, by contrast, there are many ways of choosing a sample that can make it unrepresentative of the whole. (Benchmarks for Scientific Literacy, pp. 228-229 and p. 299.)
By the end of high school, most students have been amply exposed to opinion surveys in the real world, either by paying attention to the news or by participating in some phenomenon of popular culture such as a radio call-in survey, a reality TV show voting process, or an online poll of some sort. Their academic studies should have helped them understand how opinion polls can be rendered inaccurate by various factors such as bias in selection or response. They should understand the relationship between sample size and population size, and the importance of choosing a random sample to avoid intentional or unintentional bias. Most importantly, they should approach opinion surveys with skepticism, being aware that people who conduct surveys are sometimes ignorant of scientific principles and best practices, may be careless, or may deliberately manipulate their data and analysis in order to advance a preconceived viewpoint. (Benchmarks for Science Literacy, p. 230 and 300.)
In order to do this lesson, students should have obtained an understanding of these concepts:
- Elementary descriptive statistics (percentages and measures of central tendency such as mean, median, and mode)
- Probability and chance as it pertains to selecting a random sample from a population
- How to read tables and graphs, particularly those that illustrate percentages
It must be emphasized that the goal of this lesson is to serve as an introduction to opinion surveys and help students understand the factors that affect the accuracy of opinion surveys. Teaching students how to produce accurate and unbiased opinion surveys is beyond the scope of this lesson.
Ask students to help you create a list of opinion surveys. They should be able to mention several polls, such as those conducted by Internet sites, TV shows, and polling or news organizations and those conducted by fellow students for science fair projects or the school newspaper.
Next, read out loud or put on the board this definition of opinion survey:
- An opinion survey is an inquiry into public opinion conducted by interviewing some participants from a larger population, where each of those selected is chosen randomly (entirely by chance).
Ask students which of the surveys on the list you created were based on a random selection of the target population. Many that they have mentioned will probably NOT fit the definition, particular any call-in or Internet polls or polls informally conducted by students, because the participants were not randomly selected.
Next, state that random sampling is a key factor behind reliable survey results, but other factors are important as well. Ask students to name any other factors they believe might affect the accuracy of survey results. Answers may include: question wording, order of questions, interviewer skills, interviewer bias, nonresponse, mathematical errors, etc. Accept all responses and encourage students to support their statements. The purpose of this activity is simply to draw out what students know or believe about the factors that affect the accuracy of opinion survey results. Through discussion, students should realize that many factors can affect the accuracy and reliability of opinion surveys. Allow about 10 minutes for this discussion.
Next, explain that you will conduct an informal, nonscientific opinion survey as a way to demonstrate the factors that can affect the accuracy of a survey’s conclusions. Write on the board a question that you are certain will generate some differences of opinion within the class, such as:
- Do you believe that grade point average (GPA) is an accurate predictor of future success in life?
Ask students to raise their hands if their answer to the question is “yes.” Record the number of students. Then ask students to raise their hands if their answer to the question is “no,” and record that number. Note any abstainers.
Ask the class to help you convert that data into a poll result, such as:
- Sixty percent of students believe that GPA is a useful predictor of future success in life.
Then lead a discussion of these questions:
- To what extent does this survey conclusion reliably reflect group opinion? (It reflects the opinions of the group of students in the class. Everyone in the class [the entire population], rather than just a sample, participated in the poll.)
- What factors might reduce the reliability of this statement? (All students [in the grade, school, community, etc.] were not polled, nor a representative sample of all students; only the students in the class were polled. People may be more likely to agree than to disagree with a statement that is presented, whatever it is. If a number of students abstained from participating for whatever reason, then the reported result may differ from the actual views of the population. The statement is vague; students may have responded based on differing interpretations of the phrase “useful predictor of future success in life.” Students may not have provided their honest opinion, for a variety of reasons, such as the polling situation was not confidential [that is, everyone could see how each other was voting]. The teacher may have miscounted the show of hands.)
- If another class were polled, would the response be the same? (There is no way of knowing, because the students participating in this survey were not randomly selected from the pool of all students.)
- How could we rewrite this statement to make it more accurate? (“Sixty percent of students in this class believe that GPA is an accurate predictor of future success in life.” Students may choose to rewrite the statement so that it is less subject to various interpretations [i.e., “Do you believe that grade point average (GPA) is an important factor colleges consider when deciding which applicants to accept?”] and then revote.)
Allow about 15 minutes for this activity.
To further introduce students to the factors that affect the accuracy of poll results, direct students to work in small groups to find the answers to a set of questions. First, divide the class into thirds and assign each group one of these articles:
You may choose to print out these readings or ask students to read them online via the Opinion Surveys student esheet. Also, print out the Opinion Survey Quiz ahead of time to give one copy to each student.
Organize students into groups of three or more; a student from each of the above groups should be included in each triad, because each of the above three readings provides information for some but not all of the Quiz questions.
Direct the students to read or scan their articles and then complete the Quiz together in their groups of three. You may ask students to turn in their Quizzes or to discuss answers in class. The Quiz Answer Sheet indicates where the resources provide information that can be used to answer each question.
Next, distribute copies of or display on an overhead a contemporary poll report that you have selected, such as one from a newspaper or on the website of The Gallup Poll or The Pew Charitable Trusts. Guide students through an analysis of the poll with such questions as:
- Who conducted the poll?
- How were the survey respondents chosen?
- What is the sample size?
- Is the sample representative of the whole?
- What is the sampling error for each response?
- Has this poll been weighted, and if so, why?
- Is there any evidence of bias?
- Is there evidence of other factors that might skew the poll results?
You may choose to lead discussions of two polls—one that is done by a creditable polling organization using responsible practices, and another that is NOT done in a manner that produces reliable results.
As homework, direct students to work independently or in their small groups to complete one of these assignments:
- Make a poster display of a published opinion survey and indicate the information the pollster has provided to help readers determine whether the survey was conducted in a manner that reduces error. For example, is information provided about sample size, question wording, margin of error, etc.?
- Write a brief essay on one of these topics:
- Online polls: Can they be scientific?
- Why surveys sometimes get it wrong
- The different purposes of surveys (with examples)
- A famous polling incident (such as when polls predicted that Thomas Dewey would defeat Harry S. Truman in the 1948 U.S. Presidential election)
PollingReport.com bills itself as “an independent, nonpartisan resource on trends in American public opinion.” It has polling information on a wide variety of topics for students to look at after they have examined some of the issues involved.
Angus Reid Consultants provides a daily summary of published polling results from all over the globe. Students may wish to use this site to access surveys for their homework assignment or for further study of the topic.
There are a number of Internet sites that enable visitors to participate in polls and even to create their own polls. These are very popular with young people; your students probably already have engaged in an Internet-based survey or will eventually. Therefore, rather than warning against the phenomena, you might encourage your students to go look at such a survey and critically examine it to see whether it is conducted scientifically (most are not). One such website is YouPolls.