Introduction to Program Evaluation

Introduction to Program Evaluation What Is Program Evaluation? Program evaluation is the systematic process of gathering information about how a program is designed, implemented, and what results it produces. Think of it as asking critical questions about a program: Is it running as intended? Is it actually working? Is it achieving what stakeholders hoped it would? The main purpose of program evaluation is to provide evidence-based feedback to help stakeholders—program managers, funders, participants, and policymakers—make informed decisions. These decisions might include whether to continue a program, invest more resources, make improvements, scale it up, or end it altogether. Why Evaluation Matters: Three Key Benefits Accountability. Evaluation demonstrates that money and effort are being spent wisely. Funders and the public want to know their investments produce real results. Learning. Evaluation identifies what works, for whom, and under what conditions. This knowledge helps program staff improve operations and informs better program design in the future. Evidence for Policy. Evaluation findings provide concrete data that guide decisions about scaling programs, adopting new policies, or allocating resources in specific ways. Who Uses Evaluation Results? Different stakeholders rely on evaluation information for different reasons: Program managers use findings to improve how services are delivered day-to-day Funders use findings to decide whether to continue funding or invest elsewhere Participants and clients benefit when evaluations identify gaps in service quality Policymakers and community leaders use evaluation evidence to assess public impact and inform broader policy decisions Types of Program Evaluation Understanding the two main types of evaluation is crucial because they answer different questions and are used at different stages of a program's life. Formative Evaluation: Looking at Process Formative evaluation focuses on how a program is being delivered. It examines whether the program is actually being implemented as originally planned and whether participants are receiving the services they're supposed to receive. Key questions in formative evaluation include: Are program activities happening as intended? Are participants actually showing up and engaging? What barriers or challenges are staff encountering? How satisfied are participants with the services they're receiving? Formative evaluation typically occurs during the early and middle phases of a program. It uses the findings to make real-time improvements—staff can adjust operations, remove barriers, and refine how services are delivered. Example: A new after-school tutoring program could conduct formative evaluation by observing classes and interviewing tutors. If they discover that attendance is low because students don't have transportation, staff can arrange rides or change meeting times. Summative Evaluation: Looking at Outcomes Summative evaluation focuses on results and impact after a program has been running for a sufficient period. It asks the bigger-picture question: Did the program actually achieve what it was supposed to achieve? Key questions in summative evaluation include: Did the program reach its stated goals? Did student test scores improve? Did vaccination rates increase? Did outcomes actually change for participants? Summative evaluation typically occurs at the end of a program or after a major phase when there's enough time and data to assess whether outcomes have changed. The findings inform strategic decisions about whether to continue, scale up, or terminate the program. Example: After the tutoring program has run for a full school year, a summative evaluation might compare students' test score gains to a similar group of students who did not receive tutoring, showing whether the program actually improved academic achievement. Key Differences Between Formative and Summative Evaluation | Aspect | Formative | Summative | |--------|-----------|-----------| | Focus | Process and implementation | Outcomes and results | | Timing | Continuous, during program | Single point in time, at end | | When to Use | Program is new or under revision | Program has run long enough to show results | | Methods | Qualitative (observations, interviews) | Quantitative (statistics, comparisons) | | Purpose | Make immediate improvements | Inform strategic decisions | Tricky Distinction Alert A common confusion: students sometimes think formative evaluation is only about asking for feedback. In reality, formative evaluation is specifically about evaluating the program's implementation process, not just getting general feedback. Similarly, summative evaluation isn't just a final test—it's a systematic assessment of whether the program's intended outcomes were achieved, often using comparison groups to show that the program actually caused the changes. The Program Evaluation Cycle Evaluation follows a logical, structured sequence. Understanding these steps helps you see how all pieces of an evaluation fit together. Step 1: Define Purpose and Questions Before collecting any data, evaluators must be crystal clear about what they're trying to learn. This step involves: Writing an evaluation purpose statement that identifies who will use the findings and why Developing specific evaluation questions that guide all subsequent decisions Evaluation questions are typically concrete. Instead of asking "Is the program good?", evaluators ask things like "What percentage of participants completed the program?" or "Did student attendance improve compared to the previous year?" Step 2: Develop the Evaluation Design The design matches the evaluation questions with appropriate methods. This step specifies: Data collection methods such as surveys, interviews, observations, document review, or statistical analysis Sampling strategy: Who will provide information? Everyone or a selected group? Timeline: When will data be collected? Measurement instruments: What specific tools will be used (questionnaires, observation protocols, etc.)? The key principle: the design should directly answer the evaluation questions you identified in Step 1. Step 3: Collect Data This is when evaluators actually gather information from program staff, participants, program records, or external sources. Data collection might happen through: Questionnaires completed by participants Focus group discussions One-on-one interviews with staff Direct observations of program activities Extraction of data from existing databases or records Important ethical consideration: Evaluators must address informed consent (people know they're being evaluated and agree to participate) and confidentiality (protecting people's privacy) during data collection. Step 4: Analyze Data Now the information gathered must be examined for patterns and meaning. Analysis looks for connections to the evaluation questions. Quantitative analysis (with numbers) might involve: Calculating averages, percentages, or rates Comparing groups using statistical tests Determining whether changes are meaningful or just chance Qualitative analysis (with words) might involve: Coding interview transcripts—labeling sections with themes Identifying patterns in what people said Summarizing key insights from observations The analysis determines whether program goals were actually met and identifies strengths and areas needing improvement. Step 5: Report and Use Results Findings must be communicated clearly to stakeholders in formats they can actually use: Written reports with clear conclusions Presentations to decision-makers Dashboards or infographics Executive summaries for busy administrators Actionable recommendations are crucial—the report shouldn't just describe findings, but explain what should be done differently based on those findings. When stakeholders review the report, they decide how to apply it: Should the program be refined based on formative evaluation findings? Should it be scaled up if summative evaluation shows strong results? Should it be terminated if results are poor? Bringing It All Together Now that you understand program evaluation's definition, types, and process, recognize that these elements work together: A formative evaluation uses the evaluation cycle to improve a program that's currently operating A summative evaluation uses the same structured cycle to determine whether the program achieved its goals Both types answer purposeful questions through systematic processes that involve stakeholders and lead to actionable decisions The core principle underlying all program evaluation is this: Evaluation exists to provide evidence that helps people make better decisions about programs and policies.