Statistics - Inference and Hypothesis Testing

Inferential Statistics What is Statistical Inference? Statistical inference is the process of using sample data to draw conclusions about the larger population from which the data came. Rather than collecting information about every single individual in a population (which is usually impossible), we collect data from a sample and use that sample to make educated guesses about population characteristics. The key insight of inferential statistics is this: sample statistics (like the sample mean) vary from sample to sample, but they follow predictable patterns described by probability distributions. By understanding these patterns, we can quantify our uncertainty and make reliable statements about unknown population parameters. Estimators: Using Sample Statistics to Estimate Population Parameters An estimator is a statistic—a formula applied to sample data—that we use to estimate an unknown population parameter. For example, the sample mean $\bar{x}$ is an estimator for the population mean $\mu$. Similarly, the sample variance $s^2$ is an estimator for the population variance $\sigma^2$. The crucial point is that estimators are random variables. Different samples will yield different estimates, even from the same population. Because of this randomness, we need to choose estimators carefully based on their statistical properties. Pivotal Quantities: Tools for Building Inference Procedures A pivotal quantity is a special kind of statistic with a remarkable property: its probability distribution does not depend on the unknown parameters we're trying to estimate. This makes pivotal quantities incredibly useful for constructing hypothesis tests and confidence intervals. For example, when sampling from a normal distribution with unknown mean $\mu$ and standard deviation $\sigma$, the standardized sample mean—known as the $z$-score—has this property: $$z = \frac{\bar{x} - \mu}{\sigma/\sqrt{n}}$$ Even though we don't know $\mu$ or $\sigma$, we know that this $z$-score follows a standard normal distribution. Other important pivotal quantities include the $t$-statistic (used when $\sigma$ is unknown) and the chi-square statistic. Desirable Properties of Estimators Not all estimators are equally good. Statistics has developed several criteria to evaluate which estimators we should use: Unbiasedness means that if we repeated the sampling process infinitely many times and averaged all our estimates, we'd get exactly the true parameter value. Mathematically, an estimator $\hat{\theta}$ is unbiased if $E[\hat{\theta}] = \theta$. This prevents systematic over- or underestimation. Consistency means that as our sample size grows, the estimator converges to the true parameter. Larger samples give us estimates that are closer to reality. More precisely, the probability that our estimate is far from the true value approaches zero as $n \to \infty$. Efficiency compares estimators by their variability. Even if two estimators are both unbiased, one might produce estimates that cluster tightly around the true value while the other is more scattered. The efficient estimator has lower mean squared error—it balances both bias and variance. UMVUE (Uniformly Minimum Variance Unbiased Estimator) is the "gold standard"—the unbiased estimator with the lowest possible variance. When a UMVUE exists, it's the estimator we should use. Methods for Obtaining Estimators There are several systematic approaches to finding good estimators, each with different strengths: Method of moments equates sample moments (like the sample mean) to population moments and solves for the parameters. This is straightforward but not always the most efficient approach. Maximum likelihood finds the parameter values that make the observed data most probable. This method often produces estimators with excellent properties, including consistency and asymptotic efficiency. Least squares minimizes the sum of squared differences between observations and model predictions. This is especially useful in regression analysis. Estimating equations generalizes these approaches by setting certain functions of the parameters equal to zero. These methods are less likely to be tested in detail, but understanding that multiple systematic approaches exist helps you interpret results from statistical software. Hypothesis Testing: Making Decisions from Data Setting Up the Problem: Null and Alternative Hypotheses Hypothesis testing is a formal framework for making decisions about population parameters based on sample evidence. We begin by formulating two competing statements: The null hypothesis ($H0$) represents the status quo or "nothing interesting happening" scenario. It typically claims that no relationship exists, no effect is present, or a parameter equals some specific value. For instance, $H0$ might state that a new drug is no more effective than an existing treatment, or that there is no correlation between two variables. The alternative hypothesis ($H1$) contradicts $H0$ and represents the claim we're investigating. It might state that the new drug is more effective, or that a correlation exists. The structure matters: we design the test to control the risk of falsely rejecting $H0$ (a Type I error). We are conservative about rejecting $H0$, requiring strong evidence before we do so. Making the Decision: Test Statistics and Critical Regions Once we've specified our hypotheses, we compute a test statistic—a single number that summarizes the evidence in our sample. Ideally, this test statistic is a pivotal quantity, so its distribution is known regardless of the unknown parameters. We then compare this test statistic to a critical region (a set of values that would be very unlikely if $H0$ were true). If our computed test statistic falls in the critical region, we reject $H0$; otherwise, we fail to reject it. The critical region is chosen so that, if $H0$ is true, the probability of the test statistic landing in the critical region equals our chosen significance level $\alpha$. Understanding Error Types Since we're making decisions based on incomplete information (a sample, not the whole population), we inevitably risk making mistakes: A Type I error occurs when we reject $H0$ when it is actually true—a false positive. This is like convicting an innocent person. The significance level $\alpha$ is precisely the probability of committing a Type I error. A Type II error occurs when we fail to reject $H0$ when it is actually false—a false negative. This is like acquitting a guilty person. The probability of a Type II error is usually denoted $\beta$. There is a trade-off between these errors. Lowering $\alpha$ (the bar for rejecting $H0$) increases $\beta$. Increasing sample size is one way to reduce both types of error simultaneously. The Significance Level and $p$-Value: Quantifying Evidence The significance level (denoted $\alpha$, commonly 0.05) is a threshold we set before collecting data. It represents the maximum acceptable probability of a Type I error. We reject $H0$ only if our results would occur less than $\alpha$ of the time if $H0$ were true. The $p$-value is the probability, assuming $H0$ is true, of observing a result at least as extreme as what we actually observed. A small $p$-value suggests our data would be surprising if $H0$ were true, providing evidence against $H0$. If the $p$-value is less than $\alpha$, we reject $H0$. If it is greater than $\alpha$, we fail to reject $H0$. Important caveat: The $p$-value is not the probability that $H0$ is true, and it is not the probability that we made a mistake. Many people misinterpret $p$-values in exactly this way. The $p$-value conditions on $H0$ being true, whereas we actually want to know whether $H0$ is true given the data. This distinction matters deeply and relates to Bayesian thinking, which we'll discuss later. Statistical Power: The Ability to Detect True Effects Power is the probability that we correctly reject $H0$ when it is false—that is, the probability we successfully detect a real effect. Mathematically, power = $1 - \beta$, where $\beta$ is the probability of a Type II error. Good statistical tests have high power. Several factors influence power: Sample size: Larger samples increase power. This is why researchers plan sample sizes before collecting data. Effect size: Larger true effects are easier to detect, so power increases. Significance level $\alpha$: Increasing $\alpha$ increases power but also increases the risk of Type I error. Variability in the data: Less variable data allows easier detection of effects. When designing a study, researchers often specify a desired power (like 0.80 or 0.90) and then calculate the sample size needed to achieve that power. Confidence Intervals: Estimating Parameter Values and Their Uncertainty While hypothesis testing answers a yes-or-no question ("Is there an effect?"), confidence intervals provide a range of plausible values for an unknown parameter. They communicate both our best estimate and our uncertainty. A 95% confidence interval is constructed so that if we repeated the sampling and interval-building process many times, about 95% of the intervals we constructed would contain the true population value. The key phrase is "in repeated sampling"—this describes the long-run behavior of the procedure, not a probability about the unknown parameter after we've observed the data. Once we've calculated a specific interval from our data (say, from 2.3 to 4.7), we don't say "the probability that the true value is between 2.3 and 4.7 is 95%." Rather, we've used a procedure that works 95% of the time. The true parameter either is or is not in our interval—there's no probability involved anymore. The interval is typically symmetric around the point estimate (the sample statistic), with the width depending on: Sample size: Larger samples yield narrower intervals. Variability in the data: More variable data yields wider intervals. Confidence level: Requiring higher confidence (like 99% instead of 95%) yields wider intervals. Asymmetrical Confidence Intervals Not all intervals are symmetric. One-sided confidence intervals provide only a lower bound or only an upper bound rather than a two-sided range. One-sided intervals are narrower and are appropriate when we only care about one direction of deviation. Additionally, when the sampling distribution of an estimator is skewed or heavily-tailed, the optimal confidence interval may be asymmetrical—extending further in one direction than the other. This is particularly common when working with small samples or with sampling distributions that aren't normal. <extrainfo> Bayesian Alternative: Credible Intervals A credible interval takes a fundamentally different approach to the same question. Rather than asking "what range would contain the true value 95% of the time under repeated sampling?", Bayesian inference asks "given my prior beliefs and the observed data, what is the probability that the true value lies in this range?" A credible interval directly assigns a probability to the parameter being in the interval, based on both a prior probability distribution (representing your beliefs before seeing the data) and the likelihood of the observed data. This is conceptually more intuitive—it directly answers the question many people mistakenly think the frequentist confidence interval answers. However, credible intervals depend on the choice of prior distribution, which is subjective. This is both an advantage (you can incorporate prior knowledge) and a disadvantage (different analysts might disagree on the prior). </extrainfo> Common Statistical Tests and Procedures Several standard tests are routinely used across different fields: Student's $t$-test compares means between groups when sample sizes are small or population standard deviation is unknown. Analysis of variance (ANOVA) extends $t$-testing to compare means across more than two groups simultaneously. Chi-squared test evaluates whether observed categorical data matches expected frequencies, or whether two categorical variables are independent. Regression analysis quantifies the relationship between a continuous outcome and one or more predictors, typically using ordinary least squares or maximum likelihood methods. Correlation (Pearson, Spearman's rank) measures the strength and direction of association between variables. Mann–Whitney U test compares two groups when data are ordinal or normally distributed data assumptions are violated. Factor analysis discovers underlying structure in a set of variables by identifying latent factors. Time series analysis models data collected over time, accounting for temporal dependence. You'll encounter these tests throughout statistics courses and in practice. The specific test to use depends on your data type, sample size, and assumptions met by your data. <extrainfo> Bayesian Statistics: An Alternative Framework Bayes' Theorem Bayes' theorem provides a mathematical framework for updating beliefs about hypotheses based on new evidence. In its basic form: $$P(H|E) = \frac{P(E|H) \cdot P(H)}{P(E)}$$ where: $P(H)$ is the prior probability of hypothesis $H$ before seeing the evidence $P(E|H)$ is the likelihood—the probability of observing evidence $E$ if $H$ is true $P(H|E)$ is the posterior probability of $H$ after observing $E$ $P(E)$ is the overall probability of observing the evidence Bayesian inference uses this theorem repeatedly to update probabilities as new data arrives. The approach is philosophically appealing because it directly assigns probabilities to hypotheses—something frequentist inference deliberately avoids. However, Bayesian methods require specifying a prior distribution, which involves subjective judgment. Additionally, Bayesian calculations can be computationally intensive, though modern algorithms have made this much more feasible. </extrainfo>