• Prior to 2002, based on observational studies, it was routine for physicians to prescribe hormone replacement therapy for post-menopausal women to prevent myocardial infarction. In 2002 and 2004, however, published RCTs from the Women's Health Initiative claimed that women taking hormone replacement therapy with estrogen plus progestin had a higher rate of myocardial infarctions than women on a placebo, and that estrogen-only hormone replacement therapy caused no reduction in the incidence of coronary heart disease.

Many papers discuss the disadvantages of RCTs. Among the most frequently-cited drawbacks are:

Limitations of external validity

The extent to which RCTs' results are applicable outside the RCTs varies; that is, RCTs' external validity may be limited. Factors that can affect RCTs' external validity include:

• Where the RCT was performed (e.g., what works in one country may not work in another)

• Characteristics of the patients (e.g., an RCT may include patients whose prognosis is better than average, or may exclude "women, children, the elderly, and those with common medical conditions")

• Study procedures (e.g., in an RCT patients may receive intensive diagnostic procedures and follow-up care difficult to achieve in the "real world")

• Outcome measures (e.g., RCTs may use composite measures infrequently used in clinical practice)

• Incomplete reporting of adverse effects of interventions

Costs

RCTs can be expensive, for a mean cost of US$12 million per RCT. Nevertheless, the return on investment of RCTs may be high, in that the same study projected that the 28 RCTs produced a "net benefit to society at 10-years" of 46 times the cost of the trials program, based on evaluating a quality-adjusted life year as equal to the prevailing mean per capita gross domestic product.

Relative importance of RCTs and observational studies

Two studies published in The New England Journal of Medicine in 2000 found that observational studies and RCTs overall produced similar results

Two other lines of reasoning question RCTs' contribution to scientific knowledge beyond other types of studies:

• If study designs are ranked by their potential for new discoveries, then anecdotal evidence would be at the top of the list, followed by observational studies, followed by RCTs.

• RCTs may be unnecessary for treatments that have dramatic and rapid effects relative to the expected stable or progressively worse natural course of the condition treated.

Difficulty in studying rare events

Interventions to prevent events that occur only infrequently (e.g., sudden infant death syndrome) and uncommon adverse outcomes (e.g., a rare side effect of a drug) would require RCTs with extremely large sample sizes and may therefore best be assessed by observational studies.

Difficulty in studying outcomes in distant future

It is costly to maintain RCTs for the years or decades that would be ideal for evaluating some interventions.

Pro-industry findings in industry-funded RCTs

Some RCTs are fully or partly funded by the health care industry (e.g., the pharmaceutical industry) as opposed to government, nonprofit, or other sources. A systematic review published in 2003 found four 1986-2002 articles comparing industry-sponsored and nonindustry-sponsored RCTs, and in all the articles there was a correlation of industry sponsorship and positive study outcome. One possible reason for the pro-industry results in industry-funded published RCTs is publication bias.

Therapeutic misconception

Although subjects almost always provide informed consent for their participation in an RCT, studies since 1982 have documented that many RCT subjects believe that they are certain to receive treatment that is best for them personally; that is, they do not understand the difference between research and treatment. Further research is necessary to determine the prevalence of and ways to address this " therapeutic misconception".

Statistical error

RCTs are subject to both type I ("false positive") and type II ("false negative") statistical errors . Regarding Type I errors, a typical RCT will use 0.05 (i.e., 1 in 20) as the probability that the RCT will falsely find two equally effective treatments significantly different..

Cultural effects

The RCT method creates cultural effects that have not been well understood. For example, patients with terminal illness may attempt to join trials as a last ditch attempt at treatment, even when treatments are unlikely to be successful.

Criminology

A 2005 review found 83 randomized experiments in criminology published in 1982-2004, compared with only 35 published in 1957-1981. The authors classified the studies they found into five categories: "policing", "prevention", "corrections", "court", and "community". Focusing only on offending behavior programs, Hollin (2008) argued that RCTs may be difficult to implement (e.g., if an RCT required "passing sentences that would randomly assign offenders to programmes") and therefore that experiments with quasi-experimental design are still necessary.

Education

RCTs have been used in evaluating a number of educational interventions. For example, a 2009 study randomized 260 elementary school teachers' classrooms to receive or not receive a program of behavioral screening, classroom intervention, and parent training, and then measured the behavioral and academic performance of their students. Another 2009 study randomized classrooms for 678 first-grade children to receive a classroom-centered intervention, a parent-centered intervention, or no intervention, and then followed their academic outcomes through age 19.

International development

RCTs are currently being used by a number of international development experts to measure the impact of development interventions worldwide. Development economists at research organizations including Abdul Latif Jameel Poverty Action Lab have used RCTs to measure the effectiveness of poverty, health, and education programs in the developing world. While RCTs can be useful in policy evaluation, it is necessary to exercise care in interpreting the results in social science settings. For example, interventions can inadvertently induce socioeconomic and behavioral changes that can confound the relationships (Bhargava, 2008).

For some development economists, the main benefit to using RCTs compared to other research methods is that randomization guards against selection bias, a problem present in many current studies of development policy. In one notable example of a cluster RCT in the field of development economics, Olken (2007) randomized 608 villages in Indonesia in which roads were about to be built into six groups (no audit vs. audit, and no invitations to accountability meetings vs. invitations to accountability meetings vs. invitations to accountability meetings along with anonymous comment forms). After estimating "missing expenditures" (a measure of corruption ), Olken concluded that government audits were more effective than "increasing grassroots participation in monitoring" in reducing corruption.

However, similar conclusions can also be reached by suitable modeling of the data from longitudinal studies. Overall, it is important in social sciences to account for the intended as well as the unintended consequences of interventions for policy evaluations.

• Drug development

• Hypothesis testing

• Impact evaluation

• Jadad scale

• Statistical inference

• Bhargava, Alok. Randomized controlled experiments in health and social sciences: Some conceptual issues. Economics and Human Biology, 6, 293-298.

• Darwin, Charles. The effects of cross- and self-fertilization in the vegetable kingdom. London: John Murray, 1876

• Domanski MJ, McKinlay S. Successful randomized trials: a handbook for the 21st century. Philadelphia: Lippincott Williams & Wilkins, 2009. ISBN 978-0-7817-7945-6.

• Faraday, Michael. Experimental researches in chemistry and physics. London: Taylor and Francis, 1991.

• Fisher, R.A. The design of experiments. Edinburgh: Oliver and Boyd, 1935.

• Jadad AR, Enkin M. Randomized controlled trials: questions, answers, and musings. 2nd ed. Malden, Mass.: Blackwell, 2007. ISBN 978-1-4051-3266-4.

• Matthews JNS. Introduction to randomized controlled clinical trials. 2nd ed. Boca Raton, Fla.: CRC Press, 2006. ISBN 1-58488-624-2.

• Nezu AM, Nezu CM. Evidence-based outcome research: a practical guide to conducting randomized controlled trials for psychosocial interventions. Oxford: Oxford University Press, 2008. ISBN 978-0-19-530463-3.

• Solomon PL, Cavanaugh MM, Draine J. Randomized controlled trials: design and implementation for community-based psychosocial interventions. New York: Oxford University Press, 2009. ISBN 978-0-19-533319-0.

• Torgerson DJ, Torgerson C. Designing randomised trials in health, education and the social sciences: an introduction. Basingstoke, England, and New York: Palgrave Macmillan, 2008. ISBN 978-0-230-53735-4.

• Bland M. Directory of randomisation software and services. University of York, 2008 March 19.

• Evans I, Thornton H, Chalmers I. Testing treatments: better research for better health care. London: Pinter & Martin, 2010. ISBN 978-1-905177-35-6.

• Gelband H. The impact of randomized clinical trials on health policy and medical practice: background paper. Washington, DC: U.S. Congress, Office of Technology Assessment, 1983. (Report OTA-BP-H-22.)

• REFLECT (Reporting guidElines For randomized controLled trials for livEstoCk and food safeTy) Statement

• Wathen JK, Cook JD. Power and bias in adaptively randomized clinical trials. M. D. Anderson Cancer Center, University of Texas, 2006 July 12.