The Tragic Disintegration of Science

I haven't gone through all of the links yet but RMJ pointed out this article which documents the devastating failure under rigorous review of the psychological article of faith, "ego depletion".  After a description of the original experiment as done by Baumeister and Dianne Tice of Case Western Reserve University, Daniel Engbur says:

The authors called this effect “ego depletion” and said it revealed a fundamental fact about the human mind: We all have a limited supply of willpower, and it decreases with overuse. Eating a radish when you’re surrounded by fresh-baked cookies represents an epic feat of self-denial, and one that really wears you out. Willpower, argued Baumeister and Tice, draws down mental energy—it’s a muscle that can be exercised to exhaustion.

After the "self denial" experience, the original paper asserted to show that those who had used up their "will power" in eating radishes wouldn't be able to stick with a very difficult, allegedly impossible, and, I'll add, meaningless, task as long as those who hadn't used up their reserve of "will power" eating chocolate chip cookies.  It doesn't say so but there is an assumption that isn't universally true built into the experimental design, there are, actually, people who don't have a sweet tooth, even those who don't like chocolate chip cookies.  I'm kind of sick of the ubiquitous flavor, to tell you the truth. You don't have to prefer radishes to cookies, you can not want either of them and, so, not eating one or the other might not represent any kind of act of will.

Anyway, after that first study was published a huge number of subsequent studies all purporting to show the "ego depletion" phenomenon were studied.

Psychologists discovered that lots of different tasks could drain a person’s energy and leave them cognitively depleted. Poverty-stricken day laborers in rural India might wear themselves out simply by deciding whether to purchase a bar of soap. Dogs might waste their willpower by holding back from eating chow. White people might lose mental strength when they tried to talk about racial politics with a black scientist. In 2010, a group of researchers led by Martin Hagger put out a meta-analysis of the field—a study of published studies—to find out whether this sort of research could be trusted. Using data from 83 studies and 198 separate experiments, Hagger’s team confirmed the main result. “Ego depletion” seemed to be a real and reliable phenomenon.

Only, twenty years into the research juggernaut, there are problems with, not only the research it rests in, there is trouble with the basic idea, itself.

But that story is about to change. A paper now in press, and due to publish next month in the journal Perspectives on Psychological Science, describes a massive effort to reproduce the main effect that underlies this work. Comprising more than 2,000 subjects tested at two-dozen different labs on several continents, the study found exactly nothing. A zero-effect for ego depletion: No sign that the human will works as it’s been described, or that these hundreds of studies amount to very much at all.

For anyone who doesn't already realize it, this issue and many others in the field of psychology and social science has the potential to call most, perhaps all of that "science" into question.  If you can't trust the review process, the choices of those who do the research, review their colleagues and publish their data, you can't rely on any of it.

Baumeister’s theory of willpower, and his clever means of testing it, have been borne out again and again in empirical studies. The effect has been recreated in hundreds of different ways, and the underlying concept has been verified via meta-analysis. It’s not some crazy new idea, wobbling on a pile of flimsy data; it’s a sturdy edifice of knowledge, built over many years from solid bricks.

And yet, it now appears that ego depletion could be completely bogus, that its foundation might be made of rotted-out materials. That means an entire field of study—and significant portions of certain scientists’ careers—could be resting on a false premise. If something this well-established could fall apart, then what’s next? That’s not just worrying. It’s terrifying.

Considering how much power has been given to psychology by courts, by legislators, in the making of laws and regulations, their influence in education, in the media construction of the common consensus, yes, I would say discovering how unreliable, how open to the beliefs and expectations of researchers the entire thing is, finding out how unreliable it is is rather terrifying.  But not nearly as terrifying as what will, I am certain, to come from this, it will be entirely ignored because it, like the fact of global warming, is inconvenient to a deeply ingrained, deeply interested group.  This pseudo-science will continue to be dutifully reported on Morning Edition, in the New York Times, cited by columnists and pundits (the article points out that Baumeister and the NYT conservative "libertarian" columnist John Tierney have co-authored a book on the topic) and, worst of all, judges and politicians, school administrators and teachers.

And, as the article points out, the problems with the whole idea began to be noticed nine years ago.

Evan Carter was among the first to spot some weaknesses in the ego depletion literature. As a graduate student at the University of Miami, Carter set out to recreate the lemonade effect, first described in 2007, whereby the consumption of a sugary drink staves off the loss of willpower. “I was collecting as many subjects as I could, and we ended up having one of the largest samples in the ego-depletion literature,” Carter told me. But for all his efforts, he couldn’t make the study work. “I figured that I had gotten some bad intel on how to do these experiments,” he said.

Only, when he reviewed the meta-analysis of the literature, he found out that the whole thing was unreliable.

To figure out what went wrong, Carter reviewed the 2010 meta-analysis—the study using data from 83 studies and 198 experiments. The closer he looked at the paper, though, the less he believed in its conclusions. First, the meta-analysis included only published studies, which meant the data would be subject to a standard bias in favor of positive results. Second, it included studies with contradictory or counterintuitive measures of self-control. One study, for example, suggested that depleted subjects would give more money to charity while another said depleted subjects would spend less time helping a stranger. When he and his adviser, Michael McCullough, reanalyzed the 2010 paper’s data using state-of-the-art analytic methods, they found no effect. For a second paper published last year, Carter and McCullough completed a second meta-analysis that included different studies, including 48 experiments that had never been published. Again, they found “very little evidence” of a real effect.

I will probably write more on this after I've gone through more of the links.  This one by Michael Inzlicht is good because it honestly says how serious the discovered problems being discovered are and how basically they discredit the status of what is published as science.

Our problems are not small and they will not be remedied by small fixes. Our problems are systemic and they are at the core of how we conduct our science. My eyes were first opened to this possibility when I read Simmons, Nelson, and Simonsohn’s paper during what seems like a different, more innocent time. This paper details how small, seemingly innocuous, and previously encouraged data-analysis decisions could allow for anything to be presented as statistically significant. That is, flexibility in data collection and analysis could make even impossible effects seem possible and significant.

What is worse, Andrew Gelman made clear that a researcher need not actively p-hack their data to reach erroneous conclusions. It turns out such biases in data analyses might not be conscious, that researchers might not even be aware of how their data-contingent decisions are warping the conclusions they reach. This is flat-out scary: Even honest researchers with the highest of integrity might be reaching erroneous conclusions at an alarming rate.

And, as we are finding out, the problems with science aren't restricted to the social sciences which have always both claimed and been granted huge exemptions in the requirements of what it claims as having the status of reliable knowledge.

I strongly suspect that this has a lot to do with the elevation of science in both the popular and academic imagination as a better religion than religion.  And that it is directly related to the explosion in the rate at which scientists needing to publish research has expanded in the post-war period.  There was every professional reason to move papers into publication.  While the asserted nature of religion that is popularly held among those who are educated is an ahistorical and anti-religious ideological cartoon, they have, essentially, granted everything they critique about religion to the study of the material universe, what science is supposed to do.  If I were in the business of doing that kind of pop cultural analysis I'd probably claim that the medieval world elevated Christianity to that position, the renaissance elevated its imagined classical culture to replace it and, then, the enlightenment elevated science to that position.  Which I'd reject as being more than the crudest and most vulgar of characterizations of extremely complex historical epochs, but that's exactly what those folks do. Only none of those absolute authorities work when viewed that way.   It is dangerous to grant any of them anything like absolute reliability but I think that danger is at its highest when it is science.

Science as a profession is open to all of the worst features of a corrupt clergy, allowing the exigencies of professional interest, competitive publication, and faculty politics and comity to overtake the alleged ethical and professional standards and rule the actual practice of science probably was the venue through which psychology was allowed to be called "science" and allowed to practice appallingly lax methods and standards of review.  And scientists, from the start, have claimed a reliability of their product "knowledge" that most of religion never claimed, as it honestly called what it dealt in as "belief".  

Science makes predictions,  I'm no scientist but I will predict that if this process of reimposing rigor expands and continues large areas of biology will also fall under it, I think that natural selection and those claims made within biology based in it are potentially as vulnerable as psychology has been.  Indeed, since the 1960s under the influence of Hamilton, Wilson, and, popularly, Dawkins etc. the worst of the standards of psychology have been introduced directly into biology.  And a lot of the problems with that are already obvious, the problems with that have, as well, been being pointed out for the past forty years to little effect.  Maybe, now, it will get a more rigorous review.  It will, though, take at least a generation for that to disappear from the media where those who read that stuff in their college days are still true believers in it.  And it will take even longer for those who rely on the media to be let in on the bad news that their entire world view is based on junk science.   Max Planck's famous observation that scientific progress relied on the death of those who held old ideas would seem to be a reliable truth.

There are ironies galore in this, one of those being that an even higher level of rigorous, no, hostile review has already been applied to one aspect of human behavior, that is the one which is automatically rejected, every part of the critique under which huge swaths of psychology are falling has been done on the controlled research into psychic phenomena.  I think any fair reading of that research and review would show that its results are more reliably in line with the requirements of science than the large majority of claims presented by conventional psychology.   Most ironic, of all, is that many of the quasi-professional "skeptic" of that research are professors of psychology.   It makes you wonder how many of them have taught the bogus science as reliable to their students and presented such science in their professional publication.

This is a tragedy for legitimate science which is so important to our lives, to the life of the entire Earth.  That area in science which has to be protected from the scandal most of all is environmental science, especially the science around climate change.  I am, afraid, though, that the very financial and professional interest mentioned above will guarantee that it is made to pay a disproportionate price for the inevitable discrediting of science as people discover that it was not what they were sold. That is the tragedy of the disintegration of science, the hubris of so many scientists is the force that has powered that tragedy, it will take down those with integrity as well as those who are to blame.  At least that's my greatest fear.