Better Than Fine

I said recently, when reviewing Cordelia Fine's Delusions of Gender that there remains to be written a fantastic, comprehensive demolition of arguments for cognitively based sexual differences. Allow me to recommend Rebecca Jordan-Young's Brainstorm: The Flaws in the Science of Sex Differences. My problem with Fine's book was one of style rather than substance, but if this review is anything to go by, Jordan-Young's book may be the one to go to for clarity as well as comprehensiveness.

On a slightly related note, let me also recommend a couple of articles on the scientific method that recently landed on my desk: Jonah Lehrer, writing in The New Yorker magazine, discusses a phenomenon known as "the decline effect":

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.

It's also worth reading Lehrer's follow-up blog, in which he responds to the letters that The New Yorker received about his article.

But even those theories that do get replicated are shadowed by uncertainty. After all, one of the more disturbing aspects of the decline effect is that many results we now believe to be false have been replicated numerous times. To take but one example I cited in the article: After fluctuating asymmetry, a widely publicized theory in evolutionary biology, was proposed in the early nineteen-nineties, nine of the first ten independent tests confirmed the theory. In fact, it took several years before an overwhelming majority of published papers began rejecting it. This raises the obvious problem: If false results can get replicated, then how do we demarcate science from pseudoscience? And how can we be sure that anything—even a multiply confirmed finding—is true?

The other article that caught my attention was this one by Benedict Carey in the New York Times on the publication of a research report in a psychology journal that claims to show the existence of ESP. The broader argument was what interested me.

For decades, some statisticians have argued that the standard technique used to analyze data in much of social science and medicine overstates many study findings — often by a lot. As a result, these experts say, the literature is littered with positive findings that do not pan out: “effective” therapies that are no better than a placebo; slight biases that do not affect behavior; brain-imaging correlations that are meaningless.

By incorporating statistical techniques that are now widely used in other sciences — genetics, economic modeling, even wildlife monitoring — social scientists can correct for such problems, saving themselves (and, ahem, science reporters) time, effort and embarrassment.

Although, of course, even if those results are replicated, it doesn't necessarily mean they're correct.