TCS Daily

Airport Profiling's Hidden Controversy

By Carroll Andrew - January 26, 2004 12:00 AM

Airport security is making the news. An automated system for checking the identities of airline passengers against terrorist watch-lists is scheduled to begin operation soon. Proposals for implementing a second generation of the Computer Assisted Passenger Prescreening System (CAPPS II) and for information sharing between airlines are generating controversy, particularly amongst privacy advocates.

Conspicuously absent from the airport-security headlines, however, is any reevaluation of the Department of Transportation's ban on the use of racial and ethnic profiling in the passenger screening process. The debate about profiling has stagnated because the opposing sides are separated by a pair of hidden assumptions -- the idea that correlation does not imply causation and the idea that algorithms cannot duplicate the human mind's ability to process information -- that sound more relevant to a computer-science class than they do to a politically charged homeland security debate. To understand how these two assumptions define the profiling debate and why ignoring them makes reaching a consensus impossible, begin by considering the costs that profiling involves.

It'll Cost Ya

Decisions about racial and ethnic profiling involve balancing three costs. The first cost, the cost that motivates the discussion, is the number of innocent lives lost to terrorist acts. We shall refer to this cost as the fatality count. Next, the security measures needed to minimize the fatality count require expenditures of time, resource, and effort. We shall refer to these costs as the overhead. Finally, whenever a non-terrorist is subjected to a security screening, an additional cost is paid beyond the overhead. This cost includes elements like the value of freedom, the history of relations between law-enforcement and members of specific minority and ethnic groups, and the histories of the minority and ethnic groups themselves. We shall refer to this cost as the negative examination cost. All three costs must be considered in an honest and complete discussion of profiling. There can be disagreement about their relative magnitude, but none of them can be wished away.

If racial and ethnic profiling involved only a fatality count and an overhead, its practice would be much less controversial. This is because, although they might disagree on the appropriate balance, the opposing sides can roughly agree on how to evaluate a fatality count and an overhead. The fatality count increases with every terrorist murder. The overhead increases every time a new layer is added to security procedures. Consider car accidents instead of terrorist murders and consider longer travel times instead of new security layers; it becomes readily apparent that the idea that balancing a fatality count against an overhead entails all of the controversy of setting the speed limit. To isolate and understand the controversial aspects of profiling, we must look to the negative examination cost.

The negative examination cost depends upon a complex interaction between our rights as individuals, our responsibilities as citizens, our responsibilities as human beings, the motives of the examiner, and the attitude of the person being examined. Proponents and opponents can analyze identical security procedures and reach wildly different conclusions about the size of the negative examination cost incurred. Profiling opponents feel that history clearly teaches that singling out individuals based on physical characteristics imposes a cost so high that any benefit is negated. Profiling proponents feel that those singled out should consider more recent history, understand that additional scrutiny is motivated by a reasonable concern about the lives of the innocent, and take no offense. To date, there has been very little acknowledgement of this complexity in the profiling debate. Both sides have, in different ways, treated the negative examination cost as something that can be easily assessed.

Profiling proponents make statistically compelling cases that profiling effectively balances the overhead against the fatality count, implicitly treating the negative examination cost as something small enough to be ignored. One-size-fits-all treatment of the negative examination cost, however, renders such analyses incomplete. Given recent evidence, it is plausible that frequent, aggressive audits of certain charitable organizations would interdict terrorist funding, balancing the fatality count against the overhead required in an effective manner. Yet, airport profiling proponents would not endorse profiling the boards of directors of charitable organizations as the first step in the audit process. Inefficient use of resources is not the concern; the difference in attitude towards profiling reflects a set of beliefs about freedom. In other words, there is broad societal agreement that the negative examination cost of interfering with the operation of a charitable organization is much greater than the negative examination cost of searching an airline passenger. Profiling proponents accept that there are different negative examination costs in the different realms where profiling could be applied, and that airport security is one of the few places where the negative examination cost is frequently negligible. Likewise, profiling proponents should also accept that, even within the single realm of airport security, the negative examination cost might not always be small enough to be ignored.

Profiling opponents favor an approach equally extreme. They separate the negative examination cost from the other two costs, pretending that they are not related. Profiling opponents would outlaw any security procedure that does not treat everyone in the same manner, theoretically eliminating the negative examination cost. This analysis is also incomplete. The negative examination cost, the fatality count, and the overhead cannot be separated from one another. Policies eliminating any negative examination cost lead to one of two undesirable possibilities. Minimizing the negative examination cost while also minimizing the fatality count requires detailed screenings of everyone, prohibitively raising the overhead. Minimizing the negative examination cost while maintaining a feasible overhead limits detailed screenings to a small, spot-selected group of people. Due to the small percentage of terrorists relative to the general population, most terrorists would never be adequately screened. It does not require missing many terrorists to tragically raise the fatality count.

Both extremes outlined above take for granted the idea that the negative examination costs can be treated the same in every situation. Neither approach generated with this assumption leads to an acceptable profiling policy. Procedures that ignore the negative examination costs leave ambiguity about what is acceptable and what is not. Procedures that equalize all negative examination costs raise the other costs to unacceptable levels. The idea that negative examination costs can be treated in a one-size-fits-all manner must be replaced.

Correlation and Causation

Negative examination costs are complex because human interactions are complex. The cost, however, is not the only quantity that depends on the complexity of human interaction. The benefit does as well. This is where we encounter the first hidden assumption of the profiling debate, the fact that no pre-determined set of procedures can duplicate the human mind's ability to process information. In many important ways, human information processing is vastly superior to any pre-determined set of procedures, e.g., programs that process lists of symptoms cannot replace doctors or control algorithms connected to a set of instrument gauges cannot replace pilots. In the same way, neither CAPPS nor CAPPS II can effectively replace human security screeners. Any effective system for reducing the fatality count depends upon the complexity of human interaction that is responsible for the negative examination cost; the negative examination cost cannot be removed from the profiling equation

Human information processing does have its limitations. The required overhead prohibits universal application. Random searches waste valuable expertise by applying the limited pool of expertise in areas where it is not really needed. Any strategy to fully utilize the information processing capacity of the human mind requires that detailed examinations only be performed in areas where terrorists are most likely to be. First separate out subjects who possess characteristics that have the highest correlation to terrorists. Examine the remaining subjects in a bit more detail, and separate again. After each iteration, if security personnel use their expertise to eliminate non-terrorists from consuming further scrutiny, the most detailed screenings will be applied in the areas where terrorists are most likely to be. Staying with the assumption that effective separation cannot result from a pre-determined set of rules, the success of this depends on the day-to-day decisions made by security agents on the front lines.

So how does a security screener determine which subjects merit heightened scrutiny? Answering this question forces us to confront the second hidden assumption of the profiling debate, the principle that correlation does not imply causation. An individual's stance on profiling is determined, in large part, by the side of the correlation/causation inequality he or she chooses to emphasize. Profiling proponents rightly argue that there are superficial characteristics, e.g. language or national origin, that correlate with terrorist behavior. Given the harm that a terrorist can do, the mere existence of a correlation justifies profiling; the absence or presence of a causal relation is irrelevant. Profiling opponents rightly argue that superficial characteristics such as race and national origin do not cause terrorist behavior. Given a society which values individual rights, law-enforcement agents must not single out any individual until they observe something that could be causally connected to a specific crime.

Much of the disconnect in the profiling debate arises from the fact that the different sides are talking about different quantities. Yet, the subtleties of causal uncertainty are in no way limited to the field of law-enforcement or the subject of profiling; every decision about the future based on evidence from the past involves the murky relationship between correlation and causation. We seem to be able to get along in other aspects of our life. Consider an innocuous example, the process of choosing a restaurant. You are more likely to go where you have been served good meals and less likely to go where you have been served bad ones. Is this the result of correlation or causation? Strictly speaking, the existence of bad meals in the past will not cause bad meals in the future. There may be a causal connection, but you will need to know many details to be sure. Will the same cook be on duty? Were the supplies bad? Have management practices changed?

Causation is always more difficult to prove than correlation, as proving causation requires collection and analysis of many correlations. Yet it is not often that we have all the time we need to collect enough data. In the restaurant choice, for example, there are just a few hours between meals, not nearly enough time to research every detail that will affect the quality of the upcoming meal. Correlations exist, and they hint at causes, but we cannot always count on an unambiguous proof of causes. So we act using correlations as our guide, not because we want to, but because we have to. Correlation is enough in just about every theatre of human activity, from the innocuous example of restaurant selection, to decisions in areas as diverse as marketing, stock-trading, college-admissions, weather forecasting, etc. The fact that you cannot be sure of all of these details does not mean that you should not use your past experiences to help you decide. Acting on correlations with no real proof of causation is not unique to the practice of profiling.

The acceptance of the original CAPPS program by some profiling opponents (like Transportation secretary Norman Mineta) is an admission that they are willing to use correlations that do not prove causes. CAPPS is just another form of profiling. Why are the CAPPS criteria acceptable while race or national origin criteria are not? CAPPS proponents may honestly believe that they are looking for behaviors that are causally connected to specific terrorist acts, but their choice of correlations does not prove causation any more strongly than racial correlations do. Their set of variables that correlate to terrorist acts is merely different, not better. Just as most Arabs are not terrorists, neither are most people who pay cash.

Why the Exception?

If people regularly use correlation rather than causation to make their decisions, is there a reason to make an exception for profiling? Given that there is no automated system or set of rules that can effectively detect terrorists without human intervention, the answer to this question depends upon whom you trust to use correlations to balance the cost. Ultimately, the impasse in the profiling debate is not about what you think about the proper balance between correlations and costs. The impasse is about what you think about the ability of others to properly balance correlations and costs.

Implicit in the argument of profiling proponents is the assumption that police officers and other law-enforcement agents can be trusted to weigh the costs of negative examinations against the potential fatality count when acting on the correlations relevant to terrorism. Implicit in the argument of profiling opponents is the assumption that police officers cannot be trusted to take the costs of negative examinations into account when acting on the correlations relevant to terrorism. (This assumes that profiling opponents do not really believe in the argument that law-enforcement expertise contributes nothing when deciding who to carefully scrutinize.)

Profiling opponents have awarded themselves a special place in the consideration of the negative examination cost. They oppose profiling on principled grounds while doubting that everyone can comprehend the importance of the principle. They consider their ability to take the negative examination cost into consideration something that is not shared by the general population, especially security agents. It is not hyperbole to say that profiling opponents have profiled security personnel. They think that police officers are different enough in some way that makes them unable to evaluate the negative examination cost. Built into their policy prescription is the assumption that no security agent can be trusted to consider the negative examination costs of his actions.

Since it is profiling opponents that are asserting that security personnel lack an ability that others possess, i.e. the ability weigh the potential fatality count against the cost of a negative examination, it is incumbent upon them to explain why. Is this something unique to here and now, or is it true for all times? If the problem is unique to the here and now, how do we fix it? Do homeland security and other law enforcement agencies need to make a greater effort to gain the public trust? Do profiling opponents believe that day-to-day involvement in front line security operations will eventually corrupt anyone's ability to find the proper balance, or do they believe that the possibility of a few rogue agents in the field makes the use of profiling too risky? Making broad generalizations because of a few bad apples based on a shared characteristic is wrong. It is just as wrong when that characteristic is wearing a uniform instead of race.

Carroll Andrew Morse is a Ph.D. physicist who now works as a statistical modeler developing anti-fraud and anti-money laundering systems.


TCS Daily Archives