Is randomization really a panacea? 

 In the spirit of Weihua's  post  on inference in non-random and random settings, I found this  paper  by Anup Malani on selection bias in  randomized  studies.  The thirty second point is that patients' prior beliefs over outcomes of a study matter and as the probability of being in the "good" arm (to the extent that there is one) increases, the marginal patient will expect a smaller effect from the study.  What this means is that the study population is  not  representative of the overall population and that real-world outcomes will always be worse than those observed in an experiment since in the real world the probability of receiving the "good" treatment is 1. 

 Malani's model proposes three potential treatments--none, conventional, and experimental--and each treatment has a heterogeneous effect across the population; for simplicity I will assume that the outcome is binary (alive/dead, employed/unemployed, elected/unelected, etc.).  Individuals have beliefs about the probability of a good outcome for each treatment and there is also a distribution of true probabilities of a good outcome for each treatment.  The key assumption is that these two distributions are not independent, thus knowing an individual's preferences over the three potential treatments provides information about true outcomes (as long as it doesn't provide too much information).  The exciting part of this paper is that when one thinks of an experiment, it is simply a lottery among these alternatives, perhaps a lottery between no treatment and the experimental treatment.  Then anyone who believes no treatment is better than conventional treatment should enroll in the study, regardless of the lottery.  The interesting case is when should individuals who believe that conventional treatment is better than no treatment take the risk of entering the study and potentially being assigned to no treatment?  Intuitively this happens when an individual believes that the expected benefit of entering the study is greater than taking conventional treatment, but this benefit depends on the lottery.  Therefore, in a simplified form of Malani's argument, as the probability of assignment to the new treatment increases I will enroll in the study with a weaker belief in the new treatment.  Malani then shows that under plausible assumptions this yields selection bias and demonstrates that this bias could explain much of the benefit observed in randomized studies of anti-ulcer drugs. 

 Thus even randomized studies are not immune to selection bias. 

 What does one do to address this problem?  Malani proposes allowing subject to choose which control they would like to receive--thus when a subject is enrolled, she can choose a lottery over no treatment and new treatment or over conventional treatment and new treatment.  This then provides estimates of the treatment effect for individuals who prefer conventional treatment to no treatment, which may be the most relevant for policy purposes.