TCS Daily

Push Me, Poll You

By Iain Murray - December 4, 2003 12:00 AM

Despite making it through the House of Representatives, the omnibus Energy Bill couldn't survive a Senate filibuster and so discussion has been put off to January. The Bill had been castigated by environmentalists and fiscal conservatives alike, despite being loaded with measures designed to appeal to them such as the mandate for the use of ethanol as a fuel additive or the repeal of the disastrous Public Utility Holding Company Act.


One wonders whether a poll by Zogby International might have had anything to do with the success of the filibuster. It found that "a majority of likely voters (55%) feel it would be better if Congress did not pass this particular bill, knowing what the legislation contains." That last qualifying phrase is important, because it is the key to understanding whether what this poll found was legitimate or not.


The poll was conducted for The Wilderness Society, and the question it asked was:


If you knew that the energy bill Congress is considering exempts energy companies from key parts of the Clean Air and Clean Water Acts, makes it easier for oil and gas companies to drill on public lands, and gives up to 20 billion dollars in subsidies and tax breaks to the oil, gas, nuclear, and coal industries, which of the following most closely reflects your opinion about such a bill?


There is a lot wrong with this question, and to understand precisely what, we should consider the rules of good poll questions, as enumerated by Robert Worcester, the British polling maven and chairman of MORI.


Worcester cites several rules for good poll questions. A question must be logical (allowing a simple answer), precise and even-handed, without a preferred answer. The Zogby question fulfills those basic requirements. A good question must not assume knowledge, must not supply so much information as to be leading and must not be too complex. This is where the Zogby questions starts breaking rules like plates at a Greek wedding.


It assumes knowledge -- how many likely voters know what "key parts" of the Clean Air and Water acts are in play? The question is undoubtedly complex, being 70 words long. And it is surely leading -- no explanation is given as to why the oil and gas companies are to be allowed to drill on public lands or as to what the purpose of the "tax breaks" for the industries are (and there is no mention of other subsidies to promote environmentally friendly causes).


One might even argue that the question breaks a final rule against the use of colloquialisms and pejoratives. It is hard to argue that use of the term "tax break" does not come loaded with a huge amount of psychological baggage.


Perhaps the best way of illustrating the inadequacy of the question is to turn it around, and ask whether something like this formulation would have been viewed as fair:


If you knew that the energy bill Congress is considering would repeal Depression-era Acts that make blackouts more likely, would help reduce American dependence on foreign oil and provide grants to help fossil fuel industries produce more environmentally-friendly fuel, which of the following most closely reflects your opinion about such a bill?


That question would probably have produced the opposite result, but would have been just as wrong.


Still, the question is not quite as bad as the one Worcester often cites as the worst of all time, from a 1937 Gallup Poll in the UK. Respondents were asked, "Are you in favor of direct retaliatory measures against Franco's piracy?" In the space of eleven words, this question breaks five rules of question design: It is not balanced, it assumes knowledge (who is Franco?), it does not use everyday language (what are "retaliatory measures"?), it employs a pejorative (piracy) and it is vague -- retaliatory measures could range from dressing down the Ambassador to declaring war!


The Wilderness Society and their pollsters will have to try harder next time.


TCS Daily Archives