More on Cheating 

 In my  last post , I solicited comments on ways to cheat when using a design-before-analysis framework for analyzing observational studies.  My claim was that if one does the hard work of distinguishing intermediate outcomes from covariates (followed usually by discarding the former) and of balancing the covariates (often done by discarding non-comparable observations) without access to the outcome variable, it should be hard(er) to cheat.   Felix suggested one way  that should work but that should also be fairly easy to spot:  temporarily substitute in a "good" (meaning highly predictive of the outcome variable) covariate as the outcome and find a design that achieves the desired result, then use this design with the "real" outcome.   In a comment, Mike suggested another way :  do honest observational studies, but don't tell anyone about those that don't come to desired results. 

 Here's my thought:  in many observational settings, we have a strong prior that there is either an effect in a particular direction or no effect at all.  In an anti-discrimination lawsuit, for example, the issue is whether the plaintiff class is suffering from discrimination.  There is usually little chance (or worry) that the plaintiff class is in fact benefiting from discrimination.  Thus, the key issue is whether the estimated causal effect is statistically (and practically/legally) significant.  With that in mind, it seems like a researcher might be able to manipulate the distance metric essential to any balancing process.  When balancing, we have to define (a) a usually one-dimensional distance metric to decide how close observations are to one another, and (b) a cutoff point beyond which we say observations are too far from one another to risk inference, in which case we discard the offending observations.  If one side of a debate (e.g., the defendant) has an interest in results that are not statistically significant, that side can insist on distance metrics and cutofff points that result in discarding (as too far away from their peers) a great many observations.  A smaller number of observations generally means less precision and a lower likelihood of a significant result.  The other side can, of course, do the opposite. 

 I still think we're way better off in this world than in the model-snooping of regression.  What do people think?