Randomized controlled trial - Applied Fields and Critiques

Criticisms and Applications of Randomised Controlled Trials Introduction Randomised controlled trials (RCTs) are widely considered the gold standard for establishing causal relationships in research. However, they are not without limitations. This section explores the methodological criticisms of RCTs, the key assumptions they rely on, and how these considerations apply across different fields including economics, transportation, criminology, and education. Methodological Limitations of RCTs Recent systematic reviews have identified several important weaknesses in how RCTs are conducted in practice. A 2018 review of the ten most cited randomised controlled trials found recurring problems including poor balance of background characteristics between groups, difficulties implementing blinding effectively, and questionable statistical assumptions. These aren't minor technical issues—they represent fundamental challenges to the validity of RCT findings. Understanding these limitations is essential for properly interpreting RCT results. Seven Key Assumptions and Their Limitations RCTs rely on several critical assumptions that are often not fully met in practice. Each of these represents a potential source of bias in trial results. Time-Period Assessment Bias One important limitation is that RCT results are always tied to the specific historical and social context in which they occur. An intervention effective in 2010 might produce different results if tested in 2025. This time-period assessment bias arises because the broader environment—economic conditions, cultural norms, technology availability—affects how interventions work. A researcher cannot simply assume that results are timeless. Background Traits Remaining Constant RCTs assume that participants' characteristics don't change meaningfully during the study period. However, many studies last months or years, and people naturally change over time. Their skills, beliefs, health status, and circumstances may shift in ways that affect the outcome, independent of the treatment. The Average Treatment Effect Limitation Perhaps the most commonly discussed limitation is that RCTs typically report a single average treatment effect—one number summarizing how much the intervention helped on average. This hides a crucial reality: the intervention likely works differently for different people. Some participants might benefit greatly, while others see minimal benefit or even harm. The average masks this heterogeneous treatment effect. A trial showing that a program improves test scores by 5 points on average might actually help some students by 20 points while hurting others by 10 points. Understanding this heterogeneity matters for real-world policy. Treatment at Individual vs. Group Level Most RCTs assign treatment to individuals and measure individual outcomes. But many real-world interventions don't work this way. When a city implements new public transit, everyone in that city experiences it—you cannot isolate one person's experience from others'. The simple-treatment-at-individual-level limitation reflects that interventions often operate at group or contextual levels, but RCTs frequently ignore these spillover effects and contextual interactions. The All-Preconditions-Fully-Met Assumption RCTs assume that all necessary conditions for the causal mechanism to work are present. In real-world settings, this is rarely true. An education program might be designed to work only when teachers are adequately trained, but if implementation is rushed and training is poor, the causal chain breaks. The all-preconditions-fully-met assumption often fails silently, producing weak results that researchers may misinterpret. Continuous Outcome Measures The quantitative-variable limitation concerns how RCTs handle continuous measures (like test scores, income, or health metrics). The statistical techniques used make assumptions about how these variables are distributed, and when those assumptions are violated, results can be misleading. Limited Comparator Options Finally, the placebo-only-or-conventional-treatment-only limitation arises when trials compare an intervention only against a placebo or only against standard care. This ignores other potentially effective alternatives. A trial showing that Treatment A beats a placebo doesn't tell you whether Treatment B might work better. Real decision-makers often need to choose between multiple options, not just "try this or do nothing." Understanding RCT Structure and Common Implementation Issues The diagram above illustrates the typical structure of an RCT and where problems commonly arise. Understanding this flow helps clarify why every RCT produces some bias. The process begins with enrollment: participants are assessed for eligibility and those excluded never appear in results. The randomization step then divides eligible participants into groups. However, three common problems occur during allocation and follow-up: Did Not Receive Intervention: Some participants assigned to treatment don't actually receive it, undermining the comparison Discontinued Intervention: Others stop participating partway through Lost to Follow-Up: Participants disappear before the outcome is measured, often for reasons related to the treatment itself Finally, the analysis phase excludes participants for various reasons ("Not Analyzed"). Each of these steps introduces potential bias. Importantly, bias can come from factors like selection bias (who drops out), measurement bias (how outcomes are assessed), and reporting bias (which findings are published). Understanding that every RCT is susceptible to these biases is essential for interpreting results appropriately. Criteria for Appropriate RCT Use: Transportation Science Example Not all research questions are suitable for RCTs. Researchers in transportation science have developed useful criteria for determining when RCTs are appropriate for testing behavioral interventions: 1. Novel Intervention Timing The intervention should not be applied to an entire unique population before a randomised trial is conducted. If an intervention has already been rolled out to everyone, randomization becomes impossible. 2. Comparable Settings The intervention must be implementable in a setting that truly resembles the control group's environment. Testing a new transit app where the control group has no smartphone access creates an unfair comparison. 3. Isolable Effects The effect of the intervention must be separable from other activities happening simultaneously. If you're testing a congestion pricing policy but implement it alongside ten other traffic changes, you cannot isolate what caused any improvements. 4. Short Time Lag The intervention should produce observable effects relatively quickly. Measuring outcomes years later introduces many confounding factors that obscure the intervention's direct impact. 5. Known Mechanisms Causal mechanisms should be understood or testable, and they should not involve strong feedback loops where the treatment and control groups influence each other. If everyone learns about a policy test, the control group may change behavior in response. 6. Stable Relationships Causal mechanisms should be robust to different contexts and should operate similarly regardless of which group receives the intervention. This is a demanding requirement that is often violated in practice. These criteria highlight a key insight: RCTs work best for relatively simple, contained interventions with clear mechanisms, not for complex social changes. <extrainfo> Applications in Specific Fields Economics and Development Policy Economists have embraced RCTs as the first-best method for identifying causal effects in microeconomic research, contributing to what scholars call the "credibility revolution" in empirical microeconomics. This has influenced development economics substantially. For example, randomised experiments in Kenya showed that providing textbooks to students modestly improved test scores, offering quantifiable evidence for education policy. The Mexican Progresa/Oportunidades conditional cash transfer program demonstrated through RCT methodology that cash transfers tied to school attendance and health behaviors can increase attendance and improve health outcomes. Field experiments have become prominent in economics because they allow researchers to test theories in real-world settings, providing better external validity than purely laboratory studies. To improve transparency and reduce selective reporting of results, the American Economic Association now maintains a public registry for randomised controlled trials, allowing researchers to register studies before data collection begins. Criminology Implementing RCTs in criminology faces unique challenges. Randomising offenders to different rehabilitation or deterrence programs may be impractical or unethical, forcing researchers to rely on quasi-experimental designs instead. Despite these constraints, two decades of randomised experiments in criminal justice have produced valuable insights: some deterrence strategies have limited impact, while focused deterrence approaches have shown success. However, randomised evaluations of offending behavior programs reveal that many produce only modest or mixed effects on recidivism, emphasizing the importance of rigorous trial design to avoid overestimating program effectiveness. Education Educational research has increasingly adopted RCTs, with a comprehensive review of studies from 1980–2016 documenting growing use alongside persistent implementation challenges. Specific examples include a randomised trial of the First Step to Success early intervention program, which demonstrated improvements in academic and behavioral outcomes for at-risk students. Longer-term research shows that universal preventive programs introduced in first grade can positively affect achievement through high school, suggesting that early interventions may have durable effects. Transportation Research Systematic reviews identify multiple RCT-tested interventions that reduce car usage, such as congestion pricing and improved public transit. However, critics argue that complex travel behavior may limit the feasibility of strict experimental designs, challenging RCT applicability in this field. This tension between RCTs' theoretical rigor and real-world complexity is a recurring theme across applications. </extrainfo> Key Takeaway: Bias is Inevitable A fundamental conclusion emerges from this discussion: every randomised controlled trial produces some bias. Selection bias, measurement bias, reporting bias, and the various assumptions discussed above mean that no RCT is perfectly executed. This doesn't invalidate RCTs as a research method, but it does mean researchers and readers must interpret findings carefully, remaining aware of potential limitations and avoiding over-confidence in single study results. The goal is not to achieve perfect trials—an impossible standard—but to understand how each trial's specific limitations affect its conclusions.