TCS Daily

In Defense of Exit Polls

By Arnold Kling - November 5, 2004 12:00 AM

I am not a pollster, but I do teach statistics. And the question of where the exit polls went "wrong" in the 2004 election is one that I can use as an example in my classes. Basically, what I would argue is that exit polls can be perfectly conducted and still be "wrong" in the middle of the day.

Consider the hypothetical example shown in the table. The exit polls are extremely accurate. The exit poll at precinct A predicts that Kerry will win 60 percent of the vote there, and that is exactly what he gets. The exit poll at precinct B predicts that Kerry will win 45 percent of the vote there, and that is exactly what he gets.

What the exit polls do not know--and cannot know until the polls close--is the number of voters in each precinct. Once we know how many people voted in each precinct, we can use the exit polls to call the election. But in the middle of the day, when we don't know how many people voted in each precinct, the exit polls have no value. You can report the results, but the results are meaningless. I recommend clicking each button on the form in turn to see what I mean.

Candidate Precinct A Precinct B Total

Number of Voters Percent Number of Voters Percent Number of Voters Percent
Kerry 60 45
Bush 40 55

Stratified vs. Random Sample

To make an accurate prediction, you use a random sample. The idea is that by picking a set of voters at random, you are likely to have an unbiased, representative sample of voters.

Sometimes, statisticians do not use random samples. One alternative is called a stratified sample. In a stratified sample, you select a specific number of people in a given category. For example, you might choose to interview 50 black voters and 50 white voters. Then, if the true proportion of black voters in the election is 15 percent, you re-weight your results to get an accurate estimate of the actual total.

For pure prediction purposes, a random sample is easier to use than a stratified sample. However, you might pick a stratified sample if you want to get accurate analysis of a small subgroup of the population, such as minority voters. When the Federal Reserve does its survey of consumer finances, it takes a stratified sample that over-samples wealthy individuals, in order to get a better picture of asset holdings among that group.

Exit Polling is Stratified

In the case of exit polling, I cannot see any way of conducting a random sample. That is because until the election is over, you do not know what the voting population looks like, so you do not know what constitutes a random sample.

Exit polling is necessarily stratified. You have to plan ahead of time which precincts to sample and how many voters in each precinct to sample. This plan produces a stratified sample by definition.

To use an exit poll to predict the outcome of an election, you need to adjust the weights on your stratified sample. As the table above illustrates, the only information that an exit poll can provide is an estimate of the percent of people in each precinct who voted for each candidate. The exit poll cannot tell you how many voters there were in each precinct.

After the polls close, if the election officials tell you how many people voted in each precinct, then you can use exit polls to accurately predict the outcome of the election. Thus, exit polls can play a valid roll in helping news organizations "call" the election--after the polls close, but before the votes have been counted. Of course, real-world polling is subject to sampling error, so that in a close election, such as Florida in 2000, an exit poll cannot determine the winner.

Mid-day Noise

In 2004, exit poll results were "leaked" in the middle of the day, before the polls closed. At that point, you have a stratified sample without any idea of how to re-weight it to reflect the voting population. I do not know what weights were used in the mid-day leaks--perhaps the totals were weighted by the number of people who voted in those precincts in 2000, or perhaps the weights had even less basis in reality. In any event, the mid-day leak constituted noise, not information.

In the wake of the 2004 election, I have heard many people attack exit polling. Some have suggested that it ought to be banned. I think that a better alternative would be for the news media to be educated as to how exit polling works. That way, they might explain why exit poll results released in the middle of the day are noise, even though once the polls close the exit polls provide a valuable signal.


TCS Daily Archives