Multiple comparisons and the "Axe" effect 

  
Like many of us, I'm always on the lookout for good examples to use in undergraduate methods courses. My high school chemistry teacher (a former nun) said that the best teaching examples involved sex, food, or money, and that seems like reasonable advice for statistics as well. In that vein, I noted a recent article on the "Axe effect" in Metro: 

 
'Axe effect' really works, a new study swears 

 Researchers in the U.K. asked women to rate the attractiveness of men wearing Axe's British counterpart, Lynx, against those who were wearing an odorless placebo. 

 On a 7-point scale, men wearing Lynx scored a 4.2, 0.4 point higher than those wearing the placebo. 

 But here's the catch: The women did not meet the men face-to-face. They watched them on video. 

 So what explains the discrepancy in ratings? Men wearing Lynx reported feeling more confident about themselves. So the difference in attitude appears more responsible for getting you lucky than the scent itself.   

 This story was not just reported in a subway tabloid; a long article appeared in the  Economist . (Although at least the Metro story reported an effect size, unlike the Economist).  

 Is there an Axe effect? The news stories are reporting on a study in the International Journal of Cosmetic Science, "Manipulation of body odour alters men's self-confidence and judgements of their visual attractiveness by women". The researchers recruited male students and staff members from the University of Liverpool, randomly assigned some of them to use deodorant or a placebo. They then took photographs of the men as well as videos of them pretending to chat up an attractive woman. The photos and videos of the men were evaluated by "a panel of eight independent female raters" for attractiveness and self-confidence.  

 
   
     Medium 
     Attractiveness 
     Confidence 
   
   
     Photo 
     Not significant 
     (not asked) 
   
   
     Video, no sound 
     Significant! 
     Not significant 
   
   
     Video w/ sound 
     Not significant 
     Not significant 
   
 

 There may be an Axe effect on women's perception of men's attractiveness (but not self-confidence) if they see them on video if they can't hear them.  Or it might be a fluke.  This seems like a classic multiple comparison problem.  With five tests, it is not that unlikely that one of them would be (barely) statistically significant.  The proposed mechanism for the one "effect" (which attracted all of the media attention) was increased self-confidence on the part of the male subjects, so it seems a little odd that an effect would be found on perceived attractiveness and not on self-confidence.  We might be more confident that something is going on if the effect sizes were reported for the non-significant results, but they don't appear in the paper. So, the Axe effect may be for real, but only if you keep your mouth shut.