Sampling (statistics) - Survey Design Errors Weights and Random Generation

Understanding Sample Selection and Survey Quality Introduction Whether you're collecting data through a survey or designing an experiment, how you select your sample fundamentally affects the quality of your results. This unit covers the practical mechanics of random sampling, the choices you make in sample design, and—critically—the errors and biases that can creep into your data even with careful planning. Understanding these concepts will help you recognize which estimates are trustworthy and which require careful interpretation. Generating Random Samples To select a truly unbiased sample, you need a method that doesn't introduce personal preference or systematic patterns into your selection process. Random Number Tables provide one straightforward approach. Published tables contain numbers arranged randomly, allowing researchers to use them as a neutral tool for sample selection. For example, if you have a population of 500 people numbered 001 through 500, you could randomly open the table and read down the columns, selecting only those numbers that fall between 001 and 500. Pseudo-Random Number Generators are mathematical algorithms built into computers that create sequences of numbers that appear random and behave like random numbers for practical purposes. These are especially useful for large samples, where hand-selecting elements from a random number table would be tedious. Key Sampling Design Choices Once you've committed to random selection, two important decisions remain: whether elements can be selected more than once, and how large your sample should be. With-Replacement versus Without-Replacement Sampling In with-replacement sampling, once you randomly select an element, you "put it back" before the next selection. This means the same person, household, or object can theoretically appear in your sample multiple times. In without-replacement sampling, once an element is selected, it's removed from the pool and cannot be chosen again in that particular sample. This is more common in practice because you typically want each sample unit to represent itself once. The choice between these approaches matters mathematically but is often decided by practical considerations: without-replacement is more efficient for data collection since you're not contacting the same person twice. Determining Sample Size You cannot simply guess at an appropriate sample size. Instead, you must specify three things before consulting a sample-size table: Effect size: How large a difference or relationship do you want to detect? Significance level (alpha): What is your acceptable probability of incorrectly rejecting a true null hypothesis? (Usually 0.05) Power (1 – beta): What is your acceptable probability of detecting a true effect when it exists? (Usually 0.80 or higher) Once you've decided on these values, you locate the row corresponding to your desired power level, find the column matching your estimated effect size, and read off the minimum required sample size at the intersection. Errors and Biases in Sample Surveys Even with perfect random selection, your results can be wrong. Understanding where errors come from—and which ones you can control—is essential. Sampling Errors versus Selection Bias These terms describe related but distinct problems. Random sampling error is the natural variation that occurs simply because you're working with a sample rather than the entire population. If you randomly selected different people, you'd get slightly different results. This is expected and unavoidable—but it decreases as your sample size increases, which is why larger samples are more reliable. Selection bias is different and more serious: it occurs when your actual selection probabilities differ from those you assumed. For example, if you attempted to use random-digit-dialing to survey households but systematically missed households without landline phones, you've introduced selection bias because those households had zero probability of selection when you assumed they had equal probability to all others. Non-Sampling Errors Beyond sampling errors, several other problems can distort your data: Coverage errors arise from mismatches between your sampling frame (the list you select from) and the actual population you want to study. Over-coverage means your sampling frame includes people or units outside your target population. For instance, if you try to survey "current employees" but your list includes people who recently left, you have over-coverage. Under-coverage occurs when your sampling frame omits elements that belong to the target population. Historical examples include survey frames based on telephone directories, which systematically excluded people without listed numbers. Measurement error happens when the actual responses differ from the truth because respondents misunderstand questions, lack information, or find certain topics uncomfortable to discuss honestly. Processing error is introduced by your team's mistakes—incorrect data coding, entry errors, or miscalculation—rather than respondent error. Non-response bias is the distortion that occurs when people who don't respond to your survey differ systematically from those who do. If outgoing, confident people are more likely to respond than introverted people, any question about social comfort will be biased. Understanding Non-Response Non-response comes in two varieties, and they require different solutions. Unit non-response occurs when you never hear from a selected individual at all—they refuse the survey, you can't locate them, or they're unavailable during the survey period. These people contribute no data whatsoever. Item non-response is more limited: a participant responds to your survey but skips one or more specific questions, perhaps because they find them intrusive or unclear. Strategies for Mitigating Non-Response Since non-response bias can seriously distort results, invest in preventing it: Improve survey design by using clear language, logical question flow, and professional administration. Offer incentives—even modest ones like gift cards or entry into a prize drawing—to increase participation. Conduct follow-up attempts to reach non-respondents via different methods or times. Additionally, when you do reach non-respondents, collect brief data that lets you compare them to respondents—"Are you generally satisfied with your job?" This reveals whether non-respondents differ meaningfully. Use weighting adjustments when you have population benchmarks (the true percentages of different groups in your population). If your final sample has too few young adults, you can statistically inflate young adult responses to match the true population proportion. Apply imputation methods for item non-response by using statistical techniques to estimate missing answers based on patterns in responses to related questions or similar respondents' answers. Survey Weights: Correcting for Unequal Selection Sometimes different groups in your population have different probabilities of being selected—whether by design or accident. Survey weights are numerical adjustments that restore proper representation. When Weights Are Necessary Apply survey weights whenever different units had unequal probabilities of selection. Consider a few scenarios: Stratified sampling with unequal representation. If you intentionally oversampled a rural population because it's geographically dispersed, those rural responses would otherwise have too much influence on your results. Weighting scales them down to their true population proportion. In this stratified sample with urban and rural strata, the rural stratum may be weighted up (multiplied by a factor greater than 1) if it was undersampled, or weighted down if it was oversampled, to reflect its true population proportion. Household surveys with one respondent per household. When you interview only one person per household, individuals from households with five people had one-fifth the chance of selection compared to individuals in single-person households. Those five-person households must be weighted down to correct this mathematical inequality. Telephone surveys with multiple lines. Households with two or three telephone lines have a higher probability of being reached by random-digit-dialing. These households have an inherent advantage in selection and must be weighted down to give all households equal influence on the final results. The Broader Role of Weights Beyond correcting selection probabilities, survey weights serve an additional critical function: they can compensate for non-response bias by effectively inflating the influence of respondents who share characteristics with non-respondents. If women were underrepresented among respondents, women's responses receive higher weights, so their opinions carry more statistical influence. This doesn't recover the lost information but can reduce distortion if the assumption holds that non-responding and responding women are similar. <extrainfo> Additional Details on Random Number Generation Pseudo-random number generators, despite their name, are entirely deterministic: given the same starting "seed," they produce identical sequences. This reproducibility is actually useful in research—it means your random selection process is verifiable and others can replicate it. True randomness in a philosophical sense is rare in modern statistics; pseudo-random sequences that pass rigorous statistical tests of randomness are typically sufficient for sampling purposes. </extrainfo>