TCS Daily

Positively False

By Iain Murray - November 11, 2002 12:00 AM

A recent National Academies panel report on the effectiveness of lie detectors highlighted one of the biggest problems we face when dealing with data. Polygraphs, the panel concluded, were not sophisticated enough to help in screening out potential risks to national security. If there were ten security risks in a pool of 5,000 people to be screened, then a test designed to catch as many of them as possible - eight out of the ten - would also flag up over a thousand innocent parties as security risks. If the polygraphs were set to minimize the number of innocent parties misidentified, then while they would only find about 40 such cases, they would also find only two of the real risks.

In data-gathering terms these errors are known as false positives - where the test is set to find the presence of something, but finds it where it does not exist - and false negatives - where the test fails to find actual examples of what it is looking for. These errors bedevil the understanding and usefulness of data.

One example is in the question of how many times privately-owned guns are used defensively each year in the prevention of crime. Every time someone has run a survey to answer this question, the answer has been surprisingly large, from around 765,000 incidents to over 3.5 million (there are about 450,000 gun crimes every year, although that number is not directly comparable). As very few defensive uses of guns are reported to the police, these numbers seem to some to be inappropriately high.

The designers of one survey for the National Institute of Justice, Philip Cook of Northeastern University and Jens Ludwig of Georgetown, that found about 1.5 million defensive uses in the previous year decided that the survey instrument was too broad, thereby allowing many false positives. For instance, a defensive gun use from 14 or 15 months ago might be misremembered as a use from within the previous year.

Other survey designers, such as Gary Kleck of Florida State University, argue that the surveys may actually underestimate the use of guns defensively, because people who own guns illegally (in contravention of a restraining order, for instance) or who carried them without a permit, for instance, may not wish to admit their use. The surveys, Kleck and his fellows argue, are therefore balanced when it comes to the possibility of both false positives and false negatives. It is clear that in this area, at least, there is substantial room to doubt the accuracy of any of the surveys, although there does seem to be a preponderance of evidence that guns are used defensively a lot more than official statistics tell us (about 200,000 incidents a year).

In other areas, surveys seem purposefully designed to allow for false positives. In one particular example, a claim was made that a survey by Mary Koss' Ms. Foundation had found "one in four female respondents had an experience that met the legal definition of rape or attempted rape." Yet one of the questions in the survey was as follows:

Have you had sexual intercourse when you didn't want to because a man gave you alcohol or drugs?

The wording here is problematic, as Neil Gilbert of UC Berkeley has pointed out. "What does it mean," he asks, "to have sex when you don't want to 'because' a man gives you alcohol or drugs?"

Positive answers to the question do "not indicate whether duress, intoxication, force, or the threat of force was present; whether the woman's judgment or control was substantially impaired; or whether the man purposefully got the woman drunk in order to prevent her from resisting his sexual advances." He goes on to argue that the question "could have been clearly worded to denote the [general] legal standard of 'intentional incapacitation of the victim.'" The question as asked, however, provided "no way to detect whether an affirmative response corresponds to the legal definition of rape."

In surveys such as this one, false positives are actively encouraged by the survey design. Yet attempts to measure the rate of sexual assault have been bedeviled by false negatives, too. Until 1991, the federal government's National Criminal Victimization Survey (NCVS) estimate about 130,000 rapes and attempted rapes a year. Every indication was, however, that the NCVS was missing many thousands of cases annually because the question allowed, or even encouraged, false negatives because it did not ask about rape directly. When the question was changed to a direct question, and at the same time also emphasized that rape is not just committed by strangers, the number of sexual assaults jumped to 310,000.

False positives and negatives affect every form of data gathering from opinion polls (people not admitting that they intend to vote for certain candidates) through to even the seemingly objective world of lie detection. Always remember that even the best designed survey may include a significant source of inaccuracy.



TCS Daily Archives