Can We Trust Psychological Research?

A recent scandal suggests that data manipulation is all too common in psychology studies

  • Share
  • Read Later
Getty Images

Psychologist Dirk Smeesters does fascinating work, the kind that lends itself to practical, and particularly business, applications. Earlier this year, the Dutch researcher published a study that showed that varying the perspective of advertisements from the third person to the first person, such as making it seem as if we were looking out through the TV through our own eyes, makes people weigh certain information more heavily in their consumer choices. The results appeared in the Journal of Personality and Social Psychology, a top American Psychological Association (APA) journal. Last year, Smeesters published a different study in the Journal of Experimental Psychology suggesting that even manipulating colors such as blue and red can make us bend one way or another.

Except that apparently none of it is true. Last month, after being exposed by Uri Simonsohn at the University of Pennsylvania, Dr. Smeesters acknowledged manipulating his data, an admission that been the subject of fervent discussions in the scientific community. Dr. Smeesters has resigned from his position and his university has asked that the respective papers be retracted from the journals. The whole affair might be written off as one unfortunate case, except that, as Smeesters himself pointed out in his defense in Discover Magazine, the academic atmosphere in the social sciences, and particularly in psychology, effectively encourages such data manipulation to produce “statistically significant” outcomes.

(MORE: The Latest Trend: Blaming Brain Science)

Dr. Smeesters is not being accused of fabricating data altogether. He ran studies, but allegedly excluded some data so as to achieve the results he wished for. Insidious as this may sound, some recent analyses of psychological science suggest that fudging the math to get a false positive is all too easy. It is also far too common, as Leslie John and colleagues have shown.

Nor is all of it, or even most of it, purposeful. As Etienne LeBel and Kurt Peters eloquently put it recently in the Review of General Psychology, the problem is not that social scientists are willfully engaging in misconduct. The problem is that methods are so fluid that psychologists, acting in good faith but having natural human biases toward their own beliefs, can unknowingly nudge data in directions they think they should go. The field of psychology offers a staggering array of competing statistical choices for scholars. I suspect, too, that many psychologists are sensitive to comparisons with the “hard” sciences, and this may propel them to make more certain claims about the results even when it is irresponsible to do so.

(MORE: Lessons from the Lab: How to Make Group Projects More Successful)

Then there are the more obvious pressures, including the old “publish or perish” issue in academia. Getting results that don’t support a study’s hypothesis published is a rare event. Given that academic jobs and whether we are hired and fired tend to rely mainly on publications and grants, many scholars may feel pressured to be sure their results are “statistically significant.” Similarly, if a scholar has just convinced the federal government that, say, cartoons are a possibly impending danger to children everywhere and to give him or her a grant for a million dollars to prove it, it’s difficult to then come back years later and say,“Nope, I got nothing.” Some scholars function as activists for particular causes (or take funding from advocacy groups). And of course statistically significant results tend to grab headlines in ways that null results don’t.

Many psychologists are aware of these issues and very concerned about them—in fact, most of the concern about this problem has been raised from within the scholarly community itself. This is how science works, by identifying problems and trying to correct them. Our field needs to change the culture wherein null results are undervalued and scholars should submit their data along with their manuscripts for statistical peer review when trying to get published. And we need to continue to look for ways of moving past “statistical significance” into more sophisticated discussions of how our results may or may not have real world impact. These are problems that can be fixed with greater rigor and open discussion. Without any attempt to do so, however, our field risks becoming little more than opinions with numbers.

MORE: What Counts As Crazy? 

10 comments
dwwwww
dwwwww

1. You can keep your data away from everyone on the ground of "confidentiality".

2. you know you can't do anything with your project other than attending a few regional conferences (probably poster) unless you have p<0.05, eye catching conclusions. 

3. it is extremely easy to manipulate data (e.g. accidentally mistype a number; removing a particular subject's data due to whatever reason).

4. The whole discipline is infested. No one will bother to question your results unless it is obviously made-up. (or for some reason they are pissed at you and want to destroy your career).

5. After you fail to replicate someone's findings, guess what?  you can't publish your findings! 

6. rule #1 of sampling: your sample should roughly represent the population you are studying. Undergrads do not represent any population except undergrads...

7. Didn't get the p-value you need?  No worries, go back and collect some more data!  you are bound to get something significant.

...

This list goes on.

In the end it is a battle of desire vs conscience of being a psychological researcher.

Janne P. Hukkinen
Janne P. Hukkinen

Statistical data review is a good idea but I think there are a lot of problems in terms of data-privacy and how to keep the field trustworthy in the eyes of laymen. We should develop data anonymity procedures to design a best possible solution to these problems.

Fatesrider
Fatesrider

Psychology is a subjective science to a very large degree.  It's certainly not exact and open to an extremely large degree of subjectivity.  It's possible to get objective data out of a psychological study, but it depends very much on the interpretation of behaviors.  That interpretation can vary from researcher to researcher, leading to conclusions that may or may not be warranted.  So for the most part, in psychology, generalities are about the best one can hope to find, assuming there are multiple people interpreting the reactions and observations rather than just one and a consensus is reached among them.

Another thing to keep in mind is that researchers are often married to a hypothesis they wish to expand into a valid scientific theory, or make a hitherto unmade discovery.  That subjectivity alone makes all research done by individuals (or even dedicated groups) merit extremely close scrutiny.  Case in point, the energy industry funded "studies" allegedly discrediting climate change as a human-caused phenomenon in which "researchers" deliberately cherry-picked trees from the data forest and said they were representational data proving climate change wasn't happening (and then, later when the evidence became overwhelming, that it wasn't initially caused by fossil fuel emissions).  Both assertions were falsified because the researchers had an agenda which was to support the fossil fuel industry.

(Ironically, the most recent study, also funded by the energy companies - the Berkeley Study - was put together by climate change skeptics to review the data and conclusions used by the three other major climate change studies.  It was staffed by scientists in relevant fields who had their own questions about the conclusions of those studies.  When they reviewed the data and methodology, they found no errors and agreed that the conclusions were both warranted and valid.)

And while a non-result isn't exciting, it is probative.  If a

perspective difference in advertising indicates there's no effect on how

things or a message is interpreted by the consumer, that is relevant

information, even if it lacks a "wow" factor.   (Personally, I loathe

and despise first person perspectives in ads.)

Studies are supposed to be peer reviewed and signed off by outside, dispassionate, knowledgeable experts in the field before they're published.  It seems that the problem here is less about the bias of the studies and more about the failure of the checks and balances built into the system which allows these studies to be published in the first place.

Danyz
Danyz

Whoa. You start with the problems of psychological studies and then switch to climate change studies? A confused melding of two different topics here...

Cmdr_Casey_Ryback
Cmdr_Casey_Ryback

Gee, if you know so much, how many petrabytes are required to run a supercomputer model of climate?

How many v'bles? What are your assumptions? Have you 'smoothed' any data? How biased are you? How much taxpayer money did Bwarney Fwrank (D) get you?

Oh, and did you claim you are American Indian? When, in fact, you have no hard-copy evidence and never spent one day on a reservation? But you know you can get extra affirmative action points for claiming American Indian status?

The lie is clear (D). And shouting "Bush's fault" has gotten old.

Lauren Allen
Lauren Allen

This is why results must be reproducible to be valid and good studies frequently don't reach firm conclusions, but instead identify directions in which further study is needed. There's already a check to balance the temptation to just write up anything you want. The fact that these guys are being caught and their work removed is a positive thing. Everyone out there wants a perfect cause amp; effect situation, but if they have two brain cells to rub together they know that isn't possible. Anyone who's taken a base level stats course can see right through most poor studies. The biggest problem with research is that the media doesn't understand how to interpret it. 

This is seriously like complaining that testing encourages cheating.

therantguy
therantguy

Give me a break. It is totally on purpose. A) Publishing is important for things like tenure B) Studies that have no significant results don't get published C) See A

It is utterly naive to think that people don't make rational choices and fudge/eliminate or downright make up data. In fact, at the Masters level, I have yet to EVER see a study for an thesis that did NOT have fudged data. It is far easier to write a paper with interesting results than no results.

If anything is going to ever change, the research data has to be done separately from the paper. And the data collectors should have no idea what the researcher is looking for.

its_Joby
its_Joby

Hey! I saw you on Slate! Nice logo