Evidence-based policy Study Guide

📖 Core Concepts Evidence‑Based Policy (EBP) – Decision‑making that is grounded in rigorously established objective evidence rather than ideology, anecdote, or intuition. Key Elements – Good data, strong analytical skills, and political backing for using scientific information. Claiming “Evidence‑Based” – Requires (1) comparative evidence vs. at least one alternative, (2) alignment with the organization’s policy preferences, and (3) a clear explanation of how evidence + preferences justify the claim. Roots – Modeled after evidence‑based medicine, which applies research findings to clinical choices. Methodology Core – Test a theory, construct a counterfactual (what would happen without the policy), measure impact with appropriate indicators, examine direct & indirect effects, identify uncertainties/external influences, ensure replicability, and align with a cost‑benefit framework. Types of Evidence – Quantitative: numerical data from peer‑reviewed studies, surveillance systems, program records, surveys. Qualitative: non‑numerical data from observations, interviews, focus groups; useful for persuasive narratives. No inherent hierarchy: both can be equally valuable. Pew Results First Framework – Five components: program assessment, budget development, implementation oversight, outcome monitoring, targeted evaluation. Cost‑Benefit Analysis (CBA) – Economic tool that tallies economic, social, and environmental impacts to guide decisions that maximize societal welfare. Critiques – EBP may under‑estimate policy complexity, over‑rely on randomized controlled trials (RCTs) that lack real‑world relevance, and focus narrowly on single‑factor interventions instead of systemic reforms. --- 📌 Must Remember EBP ≠ Ideology – Evidence, not belief, drives the policy. Comparative Evidence Required – A single “policy works” claim is insufficient. Counterfactual is mandatory – Always ask: What would happen without this policy? Both Direct & Indirect Effects matter – Ignoring indirect pathways can mis‑state impact. Quantitative ↔ Qualitative – Neither type automatically trumps the other. CBA Formula (simplified): $$\text{Net Benefit} = \text{Total Benefits} - \text{Total Costs}$$ RCT Limitation – Not always feasible or externally valid for policy settings. Pew’s Five‑Component Checklist – Use it as a quick self‑audit for any EBP project. --- 🔄 Key Processes Define the Policy Theory – What mechanism makes the policy effective? Gather Evidence Collect quantitative data (metrics, surveys). Collect qualitative data (interviews, observations). Construct Counterfactual – Use control groups, historical baselines, or statistical models to estimate “no‑policy” outcomes. Measure Impact – Choose appropriate indicators (e.g., mortality rate, graduation rate). Separate Effects Direct: immediate outcomes attributable to the policy. Indirect: spill‑over or secondary outcomes. Identify Uncertainties & External Influences – Document data gaps, confounders, and sensitivity analyses. Ensure Replicability – Document data sources, code, and methodology for third‑party verification. Align with Cost‑Benefit Analysis – Quantify net welfare change; compare to alternatives. Report & Communicate – Produce policy options, compare pathways, and justify the preferred choice with evidence + organizational preferences. --- 🔍 Key Comparisons Evidence‑Based vs. Ideology‑Based Evidence‑Based: data‑driven, testable, comparative. Ideology‑Based: belief‑driven, anecdotal, non‑comparative. Quantitative vs. Qualitative Evidence Quantitative: numeric, statistical inference, good for measuring magnitude. Qualitative: narrative, contextual insight, good for understanding mechanisms & stakeholder perspectives. RCTs vs. Broader Evidence RCT: high internal validity, limited external validity, costly, sometimes infeasible. Observational/Qualitative: broader context, lower internal validity, useful when RCT impossible. Narrow Intervention Focus vs. Institutional Reform Narrow: targets single causal factor, quick wins, may miss systemic drivers. Institutional: tackles underlying structures, longer horizon, often more sustainable. --- ⚠️ Common Misunderstandings “More data = better policy” – Data must be comparative and relevant; irrelevant numbers add noise. Quantitative always superior – Qualitative insights can reveal why a policy works (or fails). RCT results automatically generalize – External validity must be assessed; policy context may differ. Evidence alone decides policy – Political support and preference alignment are required. CBA captures everything – Social and environmental impacts may be hard to monetize; acknowledge limits. --- 🧠 Mental Models / Intuition “Compass Model” – Evidence is a compass pointing toward the best direction; ideology is a map that may be outdated. “Counterfactual Lens” – Always view outcomes through the “what if we didn’t act?” lens; without it, impact attribution is speculative. “Evidence Triad” – Data + Analysis + Policy Preference = justified claim. --- 🚩 Exceptions & Edge Cases No feasible RCT – Use quasi‑experimental designs (difference‑in‑differences, regression discontinuity) or robust qualitative case studies. Sparse quantitative data – Lean on mixed‑methods; let qualitative narratives fill gaps, then triangulate with limited numbers. Highly politicized issues – Even strong evidence may be overridden; prepare communication strategies that align evidence with stakeholder values. Rapid‑response emergencies – Limited time for full CBA; use rapid impact assessments and update as data accrues. --- 📍 When to Use Which Quantitative metrics → When outcomes are measurable (e.g., mortality, test scores). Qualitative insights → When you need to understand stakeholder perceptions, implementation barriers, or causal pathways. RCT → When you can randomize safely and the setting mirrors real‑world conditions. Quasi‑experimental → When randomization is impossible but you have comparable groups or time series. Full CBA → For major budgetary decisions where net societal welfare is the primary objective. Rapid assessment → Early stages of crisis response; prioritize speed over completeness. --- 👀 Patterns to Recognize Comparative language – “versus alternative,” “relative effectiveness.” Counterfactual phrasing – “what would happen without…”, “baseline scenario.” Direct/Indirect effect split – Statements about “primary outcomes” and “secondary spill‑overs.” Uncertainty acknowledgment – “confidence intervals,” “sensitivity analysis,” “external factors.” Replication cue – Mention of “third‑party verification,” “open data,” “code sharing.” --- 🗂️ Exam Traps Distractor: “A policy is evidence‑based if it uses any data.” – Wrong: must have comparative evidence and justification with preferences. Distractor: “Quantitative evidence is always superior to qualitative.” – Wrong: hierarchy does not exist; both can be essential. Distractor: “RCTs are the only acceptable method for EBP.” – Wrong: RCTs are valuable but not always feasible or appropriate. Distractor: “Cost‑benefit analysis guarantees the best policy.” – Wrong: CBA may miss non‑monetizable impacts and can be biased by assumptions. Distractor: “If a policy shows a positive impact, the counterfactual is irrelevant.” – Wrong: without a counterfactual you cannot attribute the change to the policy. ---