Misunderstandings among Experimentalists and Observationalists 

 We had several discussions a while ago on this blog about balance test fallacies, and an early version of a paper on the subject that Kosuke Imai, Liz Stuart and I wrote.  Kosuke, Liz, and I also had a number of interesting discussions with people in several other fields about this topic, and we've found much confusion about the benefits of the key portions of the major research designs.  Observationalists seem to have experiment-envy, which is in at least some cases unwarrented, and experimentalists have their own related issues too.  To sort these issues out (largely or at least at first for ourselves), we have now written a new paper that tries to clarify these issues and also incorporates the points from the previous paper (material from the previous paper is the last few pages of this one).  We'd be very grateful for any comments anyone might have. 

 " Misunderstandings among Experimentalists and Observationalists: Balance Test Fallacies in Causal Inference " by Kosuke Imai, Gary King, and Elizabeth Stuart.

 Abstract

 We attempt to clarify, and show how to avoid, several fallacies of
  causal inference in experimental and observational studies.  These
  fallacies concern hypothesis tests for covariate balance between the
  treated and control groups, and the consequences of using
  randomization, blocking before randomization, and matching after
  treatment assignment to achieve balance.  Applied researchers in a
  wide range of scientific disciplines seem to fall prey to one or
  more of these fallacies.  To clarify these points, we derive a new
  three-part decomposition of the potential estimation errors in
  making causal inferences. We then show how this decomposition can
  help scholars from different experimental and observational research
  traditions better understand each other's inferential problems and
  attempted solutions.  We illustrate with a discussion of the
  misleading conclusions researchers produce when using hypothesis
  tests to check for balance in experiments and observational studies.