TCS Daily

Decisions Under Uncertainty

By Arnold Kling - February 10, 2004 12:00 AM

"Tim Russert: But can you launch a preemptive war without iron clad, absolute intelligence that he had weapons of mass destruction?

President Bush: Let me take a step back for a second and there is no such thing necessarily in a dictatorial regime of iron clad absolutely solid evidence."
-- Meet the Press, February 8, 2004

I enjoy teaching statistics in high school even more than I enjoy teaching economics. Part of the reason is that the Advanced Placement exam in statistics is so much better than the exam in economics. You can pass the economics AP just by memorizing some verbal jargon and graphical tricks. For statistics, you really need to understand the subject.

Last year's statistics exam, for example, had an excellent question about decision-making under uncertainty, which is what President Bush faced in Iraq. Instead of giving students an artificial exercise to crank out the answer based on a formula, the question described a realistic scenario and then asked:

"Define the parameter of interest and state the null and alternative hypothesis...

In the context of this situation, describe Type I and Type II errors and describe the consequences of each"

I want to show how this thinking applied in Iraq. However, first let me illustrate it with another real-world case -- the Presidential election in Florida in the year 2000.

Uncertainty in Florida

In Florida, there was doubt about the result of the election. First, the networks came out with an estimate that Al Gore had won. Later, they came out with an estimate that George Bush had won. Then there were counts and recounts.

What was the parameter of interest? In statistics, a parameter (not to be confused, as it is often is by laymen, with perimeter), is an unknown quantity. In this case, the unknown quantity might be described as "the proportion of voters in Florida who intended to vote for Gore when they cast their ballots." This parameter is unknown for a variety of reasons. Partially-punched ballots made voters' intentions less than clear. Moreover, even if there had been no "hanging chad" issues with the vote counting, the "butterfly ballot" raised the question of whether voters intending to vote for Gore wound up accidentally voting for Buchanan.

We can never know the value of a parameter. All we have are statistical estimates. In the Florida election case, the estimates included exit polls, the initial vote count, various recounts, and even a post-election audit conducted by news organizations. However, none of these estimates enables us to know the true parameter -- the proportion of voters who intended to vote for Gore when they cast their ballots.

In statistics, Type I error and Type II error are the two ways that a statistical estimate can go wrong. In the case of the Florida election, an estimate that Gore lost, when in fact a higher proportion of voters intended to vote for him as they cast their ballots, might be a Type I error. Conversely, a Type II error would be an estimate that Gore won when in fact a lower proportion of voters intended to vote for him.

In the case of the election, the cost of Type I and Type II errors was approximately symmetric. In many instances, those costs are asymmetric. The classic example of asymmetric errors is a criminal trial. Sending an innocent person to prison, which is a Type I error, is considered a much more serious mistake than letting a guilty person go free, which is a Type II error. We express the asymmetry in the phrases "innocent until proven guilty" and "the prosecution must prove its case beyond reasonable doubt."

The AP Test for Iraq

Let us ask the AP statistics test questions about Iraq. First, what is the parameter of interest?

In the case of Iraq, the unknown quantity is whether, if left alone, Saddam Hussein's regime would have eventually killed Americans or blackmailed our leaders with weapons of mass destruction (WMD). Ultimately, that is the unknown parameter about which we are concerned.

Next, let us describe Type I and Type II errors. A Type I error would be to back down from confronting Iraq and subsequently suffer a WMD attack. A type II error would be to invade Iraq when in fact we would not have been hit with WMD even if we left the regime alone.

The consequences of a Type I error -- an attack on Americans using WMD -- would go beyond even the death and destruction that would be involved. The response, in terms of military action and domestic security, would be very costly, both in terms of lives and in terms of compromises to our freedom and privacy.

The consequences of a Type II error would include loss of lives during the war and its aftermath. Also, we are left with a significant responsibility in helping Iraqis rebuild their state. By the same token, one could argue that a Type II error has benefits, such as ending the mass murders committed by the Hussein regime and giving Iraqis an opportunity to establish a better government.

We will never know the unknown parameter -- what the Hussein regime would have done vis-a-vis weapons of mass destruction had we not invaded. However, the failure to find weapons stockpiles increases the probability that we committed a Type II error and reduces the probability that by backing down we would have committed a Type I error.

Making the Decision

Prior to invading Iraq, we had no way to be certain about whether or not Iraq possessed an arsenal of WMD. It is not possible to have perfect intelligence in that context. As CIA Director George Tenet put it, "By definition, intelligence deals with the unclear, the unknown, the deliberately hidden. What the enemies of the United States hope to deny, we work to reveal... In the intelligence business, you are almost never completely wrong or completely right."

The goal of UN resolution 1441 was to get the Hussein regime to disarm in a transparent way. The regime failed to do so. In fact, the report of David Kay, the American inspector who was disappointed that he was unable to find weapons stockpiles, indicates that some weapons programs were still active, even though they apparently failed to produce usable weapons.

To wait to invade until we had ironclad intelligence that Iraq had WMD stockpiles would have been to take a huge risk of Type I error. The costs of such a mistake would have been unacceptably high. To avoid going to war, I would argue that we needed clear assurance that the Hussein regime was co-operating with UN resolution 1441. To give the regime the benefit of the doubt would have been a dubious approach for dealing with the uncertainty.

If the United States has committed a Type II error, meaning that Iraq never posed a threat, then the blame should not fall on our intelligence. It should fall primarily on the Iraqi regime for its continued concealment, deception, and defiance. There is no way that we could have relied on intelligence alone to dismiss the notion that Iraq was a threat. Only the regime itself, by fully co-operating with inspectors and by dismantling suspect programs, could have averted war.

Bush vs. Russert

In an essay on the role of Harvard Business School in shaping President Bush's thought process, Thomas Lifson writes,

"The very first lesson drummed into new students, as they file into the classrooms of Aldrich Hall, is that management consists of decision-making under conditions of uncertainty. There is never perfect information, and decisions often have to be made even when you'd really prefer to know a lot more. Given this reality, students are taught many techniques for analyzing the data which is available, extracting the non-obvious facets, learning how read into it the reasonable inferences which can be made, while quantifying the risks of doing so, and learning the costs and value of obtaining additional data."

The question from Tim Russert that I quoted at the beginning of this essay implies that we should not have invaded Iraq unless we had "iron clad absolute intelligence" that Iraq had weapons stockpiles. Russert, and many other Americans, want to see the decision in black-and-white terms.

Former Treasury Secretary Robert Rubin described the challenge of making decisions in an environment with unknown parameters. In his portrait of Rubin in the New York Times, Jacob Weisberg wrote, '"At Harvard and Yale Law School I learned to think about the uncertainties and the ambiguities of life intellectually," he says. "When I got to Goldman, Sachs, I learned it was a matter of financial life and death to learn to be probabilistic. If you thought in absolutes and black-and-whites, sooner or later you got wiped out. The odds would catch up with you."'

Decision-making under uncertainty means living with probabilities, not absolutes. Tim Russert needs to take a class in AP Statistics.


TCS Daily Archives