Normal distribution - Advanced Theoretical Tools and Extensions

Fourier Transform, Generating Functions, and the Normal Distribution Introduction When working with probability distributions, we often need tools that can capture all the information about a distribution in a compact form. Generating functions—especially the characteristic function and moment-generating function—are powerful mathematical tools that allow us to: Extract moments (mean, variance, skewness, etc.) easily Understand how distributions combine under addition Prove fundamental theorems about convergence of distributions This section introduces these tools and explores how they apply to the normal distribution and beyond. Characteristic Function The characteristic function of a random variable $X$ is defined as: $$\varphiX(t) = E[e^{itX}]$$ where $i = \sqrt{-1}$ is the imaginary unit and $t$ is a real parameter. Why Use the Characteristic Function? The characteristic function is a fundamental tool because: It always exists for any distribution (unlike the moment-generating function) It uniquely determines the distribution — if two random variables have the same characteristic function, they have the same distribution It makes theoretical work easier, especially for proving limit theorems The Normal Case For a normal random variable $X \sim N(\mu, \sigma^2)$, the characteristic function has a particularly elegant form: $$\varphiX(t) = \exp\left(i\mu t - \frac{1}{2}\sigma^2 t^2\right)$$ This formula tells us that knowing just the mean $\mu$ and variance $\sigma^2$ completely determines the distribution through its characteristic function. Moment-Generating Function (MGF) The moment-generating function is closely related to the characteristic function: $$MX(t) = E[e^{tX}]$$ where $t$ is a real parameter (not imaginary). Why "Moment-Generating"? The name comes from a key property: if you expand $MX(t)$ as a power series, the coefficients are the moments of $X$: $$MX(t) = E[e^{tX}] = E\left[\sum{k=0}^{\infty} \frac{(tX)^k}{k!}\right] = \sum{k=0}^{\infty} \frac{E[X^k]}{k!} t^k$$ This means that $E[X^k] = \frac{d^k MX}{dt^k}\bigg|{t=0}$ — you can find any moment by taking derivatives! The Normal Case For $X \sim N(\mu, \sigma^2)$, the MGF is: $$MX(t) = \exp\left(\mu t + \frac{1}{2}\sigma^2 t^2\right)$$ Important: The MGF doesn't exist for all distributions (e.g., Cauchy distribution), which is why the characteristic function is more general. Cumulant-Generating Function (CGF) The cumulant-generating function is simply the logarithm of the moment-generating function: $$KX(t) = \log MX(t)$$ For a normal distribution: $$KX(t) = \mu t + \frac{1}{2}\sigma^2 t^2$$ Understanding Cumulants Cumulants are quantities extracted from the CGF, defined as: $$\kappaj = \frac{d^j KX}{dt^j}\bigg|{t=0}$$ For the normal distribution, something remarkable happens: only the first two cumulants are non-zero: First cumulant: $\kappa1 = \mu$ (the mean) Second cumulant: $\kappa2 = \sigma^2$ (the variance) All higher cumulants: $\kappa3 = \kappa4 = \cdots = 0$ Why This Matters This property is special to the normal distribution. Cumulants measure higher-order properties like skewness and kurtosis, which are zero for normal distributions. This reflects the fact that the normal distribution is completely determined by just its mean and variance—nothing else matters. The Central Limit Theorem (CLT) The Central Limit Theorem is one of the most important results in statistics. It explains why the normal distribution appears so frequently in practice. The Statement Suppose $X1, X2, \ldots, Xn$ are independent, identically distributed (i.i.d.) random variables with: Mean: $E[Xi] = 0$ Variance: $\text{Var}(Xi) = \sigma^2 < \infty$ Then as $n \to \infty$, the sum properly scaled converges in distribution to a normal distribution: $$\frac{1}{\sqrt{n}}\sum{i=1}^{n}Xi \xrightarrow{d} N(0, \sigma^2)$$ The notation $\xrightarrow{d}$ means "converges in distribution to." The General Version If $Xi$ have non-zero mean $\mu$, then: $$\frac{1}{\sqrt{n}}\sum{i=1}^{n}(Xi - \mu) \xrightarrow{d} N(0, \sigma^2)$$ Or equivalently, for the sample mean $\bar{X}n = \frac{1}{n}\sum{i=1}^{n}Xi$: $$\sqrt{n}(\bar{X}n - \mu) \xrightarrow{d} N(0, \sigma^2)$$ Why This Matters The CLT is remarkable because: It applies to almost any distribution as long as the variables are i.i.d. with finite variance — the original distribution doesn't need to be normal It justifies normal approximations in practice: sample means, test statistics, and many other quantities approximately follow normal distributions for large enough sample sizes It explains why normal distributions are ubiquitous in nature and statistics <extrainfo> Convergence Rate: The CLT tells us the distribution converges to normal, but not how fast. The Berry-Esseen theorem provides bounds on the convergence rate, showing that convergence is generally faster for more symmetric distributions. </extrainfo> Extensions to Multivariate Settings Multivariate Normal Distribution So far we've discussed univariate (single-variable) normal distributions. The theory extends naturally to random vectors. A random vector $\mathbf{X} = (X1, X2, \ldots, Xk)^\top \in \mathbb{R}^k$ follows a multivariate normal distribution if: Every linear combination of its components is univariate normal. Mathematically, for any vector $\mathbf{a} \in \mathbb{R}^k$, the quantity $\mathbf{a}^\top \mathbf{X} = a1 X1 + a2 X2 + \cdots + ak Xk$ must be normally distributed. Parameters of the Multivariate Normal A multivariate normal distribution is completely specified by two parameters: Mean vector $\boldsymbol{\mu} \in \mathbb{R}^k$: This is simply the vector of means of each component Covariance matrix $\boldsymbol{\Sigma} \in \mathbb{R}^{k \times k}$: This is a square matrix where entry $(i,j)$ is $\text{Cov}(Xi, Xj)$ We write: $\mathbf{X} \sim N(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ Requirements on the Covariance Matrix For a valid multivariate normal distribution, the covariance matrix must be: Symmetric: $\boldsymbol{\Sigma} = \boldsymbol{\Sigma}^\top$ Positive-definite: All eigenvalues are strictly positive, and for any non-zero vector $\mathbf{v}$, we have $\mathbf{v}^\top \boldsymbol{\Sigma} \mathbf{v} > 0$ These conditions ensure the distribution is well-defined and non-degenerate. Why This Generalization Is Important Multivariate normals are crucial for: Understanding joint distributions of related variables Regression and linear models Multivariate hypothesis testing Principal component analysis and other dimension reduction techniques Summary The normal distribution's mathematical elegance emerges through generating functions: The characteristic function $\varphiX(t) = \exp(i\mu t - \frac{1}{2}\sigma^2 t^2)$ captures all distribution information The moment-generating function provides a tool for extracting moments The cumulant-generating function reveals that normal distributions are uniquely determined by just two cumulants (mean and variance) The Central Limit Theorem explains why normal distributions are fundamental: sums of i.i.d. random variables converge to normal distributions The multivariate normal extends these concepts to multiple dimensions, governed by a mean vector and covariance matrix Understanding these tools prepares you to work confidently with normal distributions in theoretical and applied contexts.