Forget the anecdotes and assumptions. Under the No Child Left Behind Act, federal education dollars are supposed to fund only programs proven effective by "scientifically based research." That's spotlighting a problem: A lot of what passes for education research isn't reliable or rigorous, and many education professors aren't keen on the scientific method.

Education has a "dirty little secret," writes Jeffrey Mervis in the June 11, 2004 Science Magazine:

"No program has yet met that rigorous standard, because none has been scientifically evaluated and shown to be effective. (A related secret is that there's no consensus on the type of evaluation studies that are needed.)

Bush's Education Department wants controlled studies, like the tests that determine whether a new drug is safe and effective. Is Panacea Z more likely to cure ignorance than Brand X? It would be nice to know before investing millions of dollars. And yet the research often provides no guidance.

In May, the National Research Council tried to determine the effectiveness of middle-school math curricula developed by the National Science Foundation and by commercial publishers. After identifying 698 studies of 19 curricula, NRC concluded it was impossible to decide which programs work and which don't. While about 20 percent of studies met NRC's minimal standards, no one program was backed by sufficient research to prove its effectiveness.

Most of the NRC's 212-page report discussed how to evaluate education programs in a scientifically valid way, Mervis writes. "The problem is complicated by the many factors that influence student achievement: students' previous knowledge, their teachers' quality of training, the level of resources available, the degree of parental and community support, and so on.

In July, the Education Department unveiled the new, improved What Works Clearinghouse which reports on which educational programs, products, practices and policies are backed by research, and evaluates the strengths or weaknesses of the studies.

For example, the clearinghouse looked at 300 studies on peer-assisted learning (students tutoring each other) and found 15 that met evidence standards; 176 studies didn't pass the screen and 109 are still being reviewed.

Of 70 studies on two middle school math curricula, only one met evidence standards fully, another met the standards with some reservations and 20 are still being reviewed.
Controlled studies are harder to do with children than with lab rats, especially if everyone assumes that the experimental group is getting something special that should be available to everyone.

Evaluating some questions requires following students for many years, but students move around so much it's hard to keep track of them. Some studies -- for example on the effectiveness of bilingual vs. English immersion classes -- get muddled because teachers aren't following the model they're supposed to be using. Controlling for home factors also is challenging. If the study is small, a few atypical students or teachers can throw off the results, critics say. If it's large, it costs a fortune.

Some educators say there's no point in doing controlled studies: The evidence will be ignored by policy makers. Or they complain that schools will focus on measurable outcomes -- test scores -- and ignore what's hard to measure.

Yet without scientific rigor, education researchers can't answer any of the interesting questions.

"Education is often degraded by the use of pseudoscience or weak science or anecdote in lieu of better methods," writes Grover J. (Russ) Whitehurst, research director for the U.S. Education Department, in a Chronicle of Higher Education story.

Academics in the American Education Research Association typically do "qualitative" and "ethnographic" research, notes the Chronicle. Like anthropologists, they describe what goes on in a classroom or at a school, but don't provide any data that makes it possible to figure out whether one approach works better than another.

"Fewer than 10 percent of AERA members are knowledgeable about randomized trials," Robert F. Boruch, a Penn education and statistics professor, tells the Chronicle. "And even fewer have actually worked on a randomized trial."

As a result, education professors have been frozen out of major new studies, often in favor of private research firms like Mathematica, which is evaluating software that claims to boost reading and math skills, and MDRC, which has specialized in job training and welfare reform. Labor economists, statisticians and psychologists have the skills to do controlled studies. For the most part, the education professors do not.

Of course, some are converting to rigorous research. It's where the money is. And plenty of academics really do want to know what works.

But there's a deep well of hostility to cold, hard, number-heavy science, poisoned further by liberal elites' loathing of the Bush administration. Though the move to controlled studies started in the Clinton administration, it didn't take off till Bush pushed through No Child Left Behind, which greatly increased federal education funding and insisted that all federally funded programs be research-based. If Bush's guys want scientific rigor, it must have something to do with Halliburton, right?

Joanne Jacobs blogs on education at JoanneJacobs.com. She's writing a book on a start-up charter high school. She is a TCS contributor.