Behind The Numbers:
Polling As A Public Service

By Stephen Medvic

On October 31, 1936, the editors of Literary Digest made a bold prediction: Republican Alf Landon would be the next president of the United States. Beginning in 1916, the Digest poll had correctly predicted five straight presidential elections. At a time before the development of scientific polling, this record was astonishing.

But in 1936 they got it wrong, even though more than 2 million people responded to their poll. The primary cause of the Digest’s error was the magazine’s practice of mailing mock ballots to people based on telephone and automobile registration. In 1936, during the Great Depression, those with phones, and particularly those with cars, were disproportionately Republican. Since the Digest’s sample was skewed, its prediction was flawed.

While the Digest was making a monumental mistake, three young pollsters independently — and correctly — predicted a landslide for Franklin Roosevelt. One, George Gallup, was even able to foresee the Digest’s misstep as early as July of that year. These new pollsters brought a scientific approach to polling and would revolutionize the practice of measuring public opinion.

Nevertheless, the science of polling was far from perfected. In 1948, polls would again mistakenly project a winner, forecasting that Harry Truman would lose the presidential election. The disasters of 1936 and 1948 taught pollsters a great deal about how to get representative samples for polls, providing the means to eventually establish sophisticated methods for measuring public opinion. As they did, those individuals with a keen interest in public opinion, such as politicians and the media, increasingly turned to polls to “take the pulse of the people.”

Politicians’ polls and those conducted by the media differ in one important respect, however. The distinction is important for evaluating the role of polls in our society because it highlights the different purposes to which polls can be put to use. Politicians (as well as interest groups and corporations) conduct private polls, the results of which help them craft campaign (or marketing) strategy. The public is not privy to the results of private polls. Media polls, on the other hand, are public and are used in reporting on issues and elections.

Universities with survey-research capabilities conduct public polls, either in conjunction with a media outfit or independently. University polls (polls used solely for academic research are typically referred to as “surveys”) not only inform the public about opinions on pressing issues of the day, they also support academic research into attitudes about a variety of subjects.

Into The Field

The Social Science Research Center (SSRC) at Old Dominion University, under the direction of Jeffrey Harlow, opened its doors in 1998. Since then, it has been providing multiple services to clients ranging from Habitat for Humanity and the American Cancer Society to the Center for Pediatric Research. For polling purposes, the SSRC is equipped with 14 computer-assisted telephone interviewing (CATI) stations and uses state-of-the-art, data-collection software. Currently, there are only two other active academic survey centers in Virginia, one in Charlottesville at the University of Virginia and the other in Blacksburg, at Virginia Tech.

When people read the results of the latest poll in the newspaper, most are unaware of the process behind the numbers. Informed citizens should understand how surveys are conducted so that they can guard against invalid results based on biased or poorly executed research. What follows is a behind-the-scenes look at that process.

There are a number of decisions to be made before a poll can “go into the field.” First, you have to decide which population’s attitudes you’re interested in measuring. Do you want to tap the views of all citizens or just likely voters? Are you interested in all Virginians or only those in Hampton Roads?

Next, you have to decide how big your sample should be. How many people should be interviewed? Part of this decision is based on cost; obviously, the larger the sample, the more expensive the poll. But a more important consideration is what pollsters call the “margin of error.” The margin of error is the range within which the population’s actual distribution of responses is likely to fall, given a certain level of statistical confidence.

For example, assume a sample’s distribution of responses is 60 percent for Candidate X and 40 percent for Candidate Y. With a margin of error of plus or minus ( +/-) 5 percentage points, the actual split between candidates among the entire population could be anything from 55 for X and 45 for Y to 65 for X and 35 for Y. Obviously, pollsters are interested in being as precise as possible, so they attempt to get a sample size large enough to produce a small margin of error. But because they require large sample sizes, small margins of error are expensive; thus, the pollster may have to trade precision for affordability.

At the SSRC, we typically try to keep the margin of error at around +/- 4 points. To do so, we only need a sample size of 600 for a statewide poll. Like most people, many of my students have a hard time believing that the views of more than 7 million Virginians can be captured with a certain degree of accuracy by interviewing only 600 people. Nevertheless, as long as the sample is drawn in a random manner, a relatively small number of people can indeed represent the views of an entire population.

Of course, in order to do telephone interviews, we need a list of phone numbers. There are basically two ways to draw a sample of phone numbers for a poll. The first is to allow the computer to randomly generate numbers to be used with specified area codes and three-digit exchanges. The second, and the one we use at the SSRC, is to purchase lists of phone numbers from businesses that specialize in gathering such lists.

The Final Phases

Once decisions about sample size have been made, the questionnaire is written. Cost is once again a factor at this stage because the more questions you ask, the more expensive it will be to complete the poll. More importantly, the pollster has to write a questionnaire that will not influence the respondents’ answers. For instance, the order of questions can affect how a person responds to some items on the poll. And question wording, which is more art than science, can have an even bigger impact. Questions must be balanced and should avoid “hot button” phrases. Experiments have shown, for example, that as many as three-quarters of the respondents to a survey will oppose more funding for “welfare;” yet the same amount will support more spending on “aid to the poor.” Questionnaire writers have to take great care to describe items on a poll in an emotionally neutral way.

Only two steps are left — but they are big ones. First, the poll has to be conducted. Without going into any detail, it is essential to recognize the importance of well-trained staff for the quality of a poll. Callers who vary from the script or incorrectly record responses threaten the validity of the results.

Finally, the results must be analyzed and reported. With respect to the latter, all poll reports must indicate a number of pieces of information about the poll, including the dates the poll was conducted, the size of the sample, the margin of error and the question wording. Without these, it is impossible to know how much faith to place in the results.

SSRC pollers have asked Hampton Roads residents about the impeachment of President Bill Clinton, and residents throughout the state about their views on campaign finance reform and their preferences at election time. When we conduct a poll, we contribute valuable information about pressing current events to researchers, elected officials and the public at large. So if your telephone rings one day and it’s someone calling from the SSRC, please take a few minutes to share your opinions with us. We’ll try not to call during dinner.

Stephen K. Medvic is an assistant professor in Old Dominion’s Department of Political Science and Geography and a research fellow at the Social Science Research Center.