Introduction to Linear Algebra

Fundamentals of Linear Algebra What is Linear Algebra? Linear algebra is the branch of mathematics that studies vectors, vector spaces, and the linear transformations between them. The central organizing principle is linearity—the property that operations preserve addition and scalar multiplication. Why does this matter? Linear algebra provides a unified framework for describing and solving countless real-world problems: systems of equations with hundreds of unknowns, transformations in computer graphics, stability analysis in engineering, and dimensionality reduction in data science. The power of linear algebra comes from its abstraction—once you understand the general principles, they apply across all these diverse domains. Basic Building Blocks: Vectors and Scalars A vector is an ordered list of numbers, typically written as a column. For example: $$\mathbf{v} = \begin{pmatrix} 2 \\ -1 \\ 3 \end{pmatrix}$$ A scalar is just a single number (like 5 or -2.3). Vectors can be manipulated in two fundamental ways: Vector addition combines two vectors component-wise: $$\begin{pmatrix} 1 \\ 2 \end{pmatrix} + \begin{pmatrix} 3 \\ 4 \end{pmatrix} = \begin{pmatrix} 4 \\ 6 \end{pmatrix}$$ Scalar multiplication scales every component by the same factor: $$3 \cdot \begin{pmatrix} 1 \\ 2 \end{pmatrix} = \begin{pmatrix} 3 \\ 6 \end{pmatrix}$$ The set of all $n$-component vectors forms what we call $\mathbb{R}^n$, the $n$-dimensional vector space. For instance, $\mathbb{R}^2$ is the familiar 2D plane, and $\mathbb{R}^3$ is 3D space. Linear Transformations A linear transformation is a function that maps vectors to vectors while respecting the structure of vector addition and scalar multiplication. More precisely, a transformation $T$ is linear if: $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$ $T(c\mathbf{u}) = cT(\mathbf{u})$ for any scalar $c$ The crucial insight is that every linear transformation can be represented as a matrix. If $A$ is a matrix and $\mathbf{x}$ is a vector, then the product $A\mathbf{x}$ gives the image of $\mathbf{x}$ under the transformation represented by $A$. This connection between abstract transformations and concrete matrix computations is what makes linear algebra so practical. Vectors and Vector Spaces Understanding Vector Spaces A vector space is a collection of vectors that is "closed" under vector addition and scalar multiplication. This means if you take any two vectors in the space and add them, the result is still in the space. Similarly, multiplying any vector by any scalar keeps you within the space. Every vector space must contain a zero vector (often written as $\mathbf{0}$), which acts as the additive identity: $\mathbf{v} + \mathbf{0} = \mathbf{v}$ for any vector $\mathbf{v}$. Think of a vector space as a mathematical container with very specific properties—it's structured enough to be useful, but general enough to encompass many different settings. Subspaces A subspace is a subset of a vector space that is itself a vector space. In other words, if you take a smaller collection of vectors from a larger vector space, and that smaller collection is closed under addition and scalar multiplication (and contains the zero vector), then you have a subspace. Key fact: Any linear combination of vectors in a subspace remains inside the subspace. This makes subspaces useful for understanding the structure within vector spaces. Basis and Dimension Imagine you want to describe every possible vector in a space, but you want to use as few "building blocks" as possible. A basis is a minimal set of vectors such that every vector in the space can be written as a linear combination of basis vectors. The dimension of a space is simply the number of vectors in any basis. For example: $\mathbb{R}^2$ has dimension 2 (the standard basis is the two unit vectors in the $x$ and $y$ directions) $\mathbb{R}^3$ has dimension 3 A line through the origin is a 1-dimensional subspace of $\mathbb{R}^3$ This concept is crucial because dimension tells you how many degrees of freedom exist in a space—how many independent numbers you need to specify a vector completely. Linear Independence and Span Two important concepts define the structure of vector spaces: Linear independence: A set of vectors is linearly independent if no non-trivial linear combination of them equals the zero vector. In plain language, no vector in the set is "redundant"—none can be written as a combination of the others. Span: The span of a set of vectors is the collection of all possible linear combinations of those vectors. It represents every point you can reach by scaling and adding those vectors together. These concepts work together: a basis is a set of linearly independent vectors whose span is the entire vector space. Matrices and Linear Transformations What is a Matrix? A matrix is a rectangular array of numbers organized in rows and columns. For example: $$A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}$$ is a $2 \times 3$ matrix (2 rows, 3 columns). Matrices are central to linear algebra because they provide a compact, computational way to represent linear transformations and systems of linear equations. Instead of writing out a complicated system by hand, we write it in matrix form: $A\mathbf{x} = \mathbf{b}$. Matrix Operations Matrix addition: Two matrices of the same dimensions can be added by adding corresponding entries: $$\begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} + \begin{pmatrix} 5 & 6 \\ 7 & 8 \end{pmatrix} = \begin{pmatrix} 6 & 8 \\ 10 & 12 \end{pmatrix}$$ Scalar multiplication: Multiply every entry by the scalar: $$2 \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} = \begin{pmatrix} 2 & 4 \\ 6 & 8 \end{pmatrix}$$ Matrix multiplication is more involved but extraordinarily important. The product $AB$ represents composing two linear transformations: first apply $B$, then apply $A$. Algebraically, if $C = AB$, then $C\mathbf{x} = A(B\mathbf{x})$. The key point: matrix multiplication is not commutative—$AB \neq BA$ in general. Special Matrices A square matrix has the same number of rows and columns. Square matrices are special because they can represent transformations on $\mathbb{R}^n$ back to itself. The identity matrix $I$ is the multiplicative neutral element. It leaves any vector unchanged: $I\mathbf{x} = \mathbf{x}$. The identity matrix has 1's on the diagonal and 0's elsewhere: $$I = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$$ <extrainfo> Matrix Representations of Transformations In applications, many geometric transformations can be expressed as matrices. For instance, a 2D rotation by angle $\theta$ is represented by: $$R\theta = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$$ Projections, reflections, and stretches have similar matrix representations. This is why linear algebra is so powerful in computer graphics and physics simulations. </extrainfo> Solving Linear Systems Linear Systems in Matrix Form A linear system of equations like: $$\begin{align} 2x + y &= 5 \\ x - y &= 1 \end{align}$$ can be written compactly in matrix form as: $$A\mathbf{x} = \mathbf{b}$$ where $A$ is the coefficient matrix, $\mathbf{x}$ is the vector of unknowns, and $\mathbf{b}$ is the vector of constants. This notation makes it clear that solving a linear system is about finding the vector(s) $\mathbf{x}$ that the matrix $A$ transforms into $\mathbf{b}$. Gaussian Elimination Gaussian elimination is the workhorse algorithm for solving linear systems. The idea is simple: use row operations to transform the coefficient matrix into an upper-triangular form (zeros below the diagonal). Once you have this triangular form, you can use back-substitution to solve for the unknowns, starting from the last equation and working backward. The three allowed row operations are: Swap two rows Multiply a row by a non-zero scalar Add a multiple of one row to another row These operations don't change the solution set, so the system is equivalent after each step. Once in triangular form, back-substitution is straightforward. Why this matters: Gaussian elimination is the most practical algorithm for solving large systems on a computer because it's systematic, reliable, and has well-understood computational complexity. Matrix Inverses For a square matrix $A$, the inverse $A^{-1}$ (if it exists) satisfies: $$A A^{-1} = A^{-1} A = I$$ If $A$ is invertible, the solution to $A\mathbf{x} = \mathbf{b}$ is simply: $$\mathbf{x} = A^{-1}\mathbf{b}$$ However, not all matrices have inverses. A matrix must be full rank (roughly speaking, its rows and columns must be linearly independent) to be invertible. A matrix that is not invertible is called singular. This is a critical point: you cannot always "divide by a matrix" the way you divide by scalars. Understanding when inverses exist, and when they don't, is essential. <extrainfo> LU Decomposition An alternative approach to solving linear systems is LU decomposition, which factors $A$ into a lower-triangular matrix $L$ and an upper-triangular matrix $U$: $$A = LU$$ Once you have this factorization, solving $A\mathbf{x} = \mathbf{b}$ becomes two simpler problems: Solve $L\mathbf{y} = \mathbf{b}$ (easy, forward substitution) Solve $U\mathbf{x} = \mathbf{y}$ (easy, back substitution) LU decomposition is particularly useful when you need to solve multiple systems with the same coefficient matrix $A$ but different right-hand sides $\mathbf{b}$. You compute the decomposition once, then reuse it. </extrainfo> The Internal Structure of Vector Spaces Rank: A Measure of Independence The rank of a matrix is the number of linearly independent rows it contains (equivalently, the number of linearly independent columns). Rank tells you how much "information" the matrix contains—a full-rank matrix has no redundancy, while a lower-rank matrix has redundant rows or columns. Geometrically, rank equals the dimension of the column space (the span of all column vectors). The column space is where the transformation $A\mathbf{x}$ can reach. The Null Space (Kernel) The null space of a matrix $A$, often written $\text{null}(A)$ or $\ker(A)$, consists of all vectors $\mathbf{x}$ that satisfy: $$A\mathbf{x} = \mathbf{0}$$ The null space is a subspace (it's closed under addition and scalar multiplication). Geometrically, it represents all the directions that the transformation $A$ maps to zero—the "information lost" by the transformation. The Rank-Nullity Theorem One of the most elegant results in linear algebra is the rank-nullity theorem: $$\text{rank}(A) + \text{nullity}(A) = n$$ where $n$ is the number of columns of $A$, and nullity is the dimension of the null space. What does this mean intuitively? The information carried by matrix $A$ splits into two parts: the rank (the "new" information about which vectors it can reach) and the nullity (the "lost" information about which directions it maps to zero). Together, they must account for all $n$ dimensions of the input space. Degrees of Freedom in Linear Systems When solving $A\mathbf{x} = \mathbf{b}$, the number of free variables (variables you can choose arbitrarily) equals the nullity of $A$. Here's why this matters: If nullity = 0, the system has at most one solution (either no solution if $\mathbf{b}$ is not in the column space, or exactly one solution if it is). If nullity > 0, the system has infinitely many solutions—you can choose the free variables however you want, then solve for the remaining variables. A higher nullity means more degrees of freedom in the solution set. Eigenvalues and Eigenvectors The Big Picture Eigenvalues and eigenvectors reveal the "natural directions" of a linear transformation—the directions that the transformation only stretches or compresses, without rotating. This might seem abstract, but it's incredibly powerful: eigenvalues help us understand system stability, compute powers of matrices efficiently, analyze vibrations, and much more. Definitions An eigenvector of a matrix $A$ is a non-zero vector $\mathbf{v}$ such that: $$A\mathbf{v} = \lambda \mathbf{v}$$ for some scalar $\lambda$. The scalar $\lambda$ is called the eigenvalue corresponding to that eigenvector. The equation says something remarkable: when $A$ acts on $\mathbf{v}$, the result is just a scaled version of $\mathbf{v}$. The direction doesn't change—only the magnitude (by a factor of $\lambda$). Geometric Interpretation Imagine a linear transformation represented by matrix $A$. Most vectors get rotated and scaled in complicated ways. But eigenvectors are special: they point in directions that the transformation only scales. If $\lambda > 1$, the eigenvector is stretched. If $0 < \lambda < 1$, it's compressed. If $\lambda < 0$, it's reversed (flipped and scaled). If $\lambda = 1$, the eigenvector is unchanged by the transformation. This geometric insight is powerful: it tells you which directions are "stable" or "unstable" under the transformation. Finding Eigenvalues To find the eigenvalues, you solve the characteristic equation: $$\det(A - \lambda I) = 0$$ Here, $\det$ denotes the determinant. This equation gives you a polynomial (the characteristic polynomial) whose roots are the eigenvalues. For example, if $A$ is a $2 \times 2$ matrix, the characteristic polynomial is typically quadratic, giving you (potentially) two eigenvalues. An $n \times n$ matrix has a degree-$n$ characteristic polynomial, giving you (potentially) up to $n$ eigenvalues. Why this works: The equation $A\mathbf{v} = \lambda\mathbf{v}$ can be rewritten as $(A - \lambda I)\mathbf{v} = \mathbf{0}$. For a non-trivial solution $\mathbf{v}$ to exist, the matrix $A - \lambda I$ must be singular (non-invertible), which happens exactly when its determinant is zero. Diagonalization If a matrix $A$ has $n$ linearly independent eigenvectors (where $n$ is the size of the matrix), then $A$ can be diagonalized: $$A = PDP^{-1}$$ where: $P$ is a matrix whose columns are the eigenvectors $D$ is a diagonal matrix whose diagonal entries are the eigenvalues Diagonalization is transformative (pun intended) because: Computing powers is easy: $A^k = PD^kP^{-1}$, and raising a diagonal matrix to a power is trivial. Understanding the transformation is easier: the diagonal matrix shows exactly how much each eigenvector is scaled. Many computations simplify dramatically. Important limitation: Not all matrices are diagonalizable. A matrix must have enough linearly independent eigenvectors. However, many matrices you'll encounter (especially symmetric ones) are diagonalizable. <extrainfo> Applications of Linear Algebra Eigenvalues and System Stability In differential equations, eigenvalues of the system's matrix determine whether the system is stable or unstable. Negative eigenvalues indicate decay toward equilibrium, while positive eigenvalues indicate exponential growth. This is critical in engineering and science—for instance, determining whether a mechanical system will oscillate wildly or settle down. Computer Graphics Matrices represent 3D rotations, scaling, shearing, and perspective projections. By composing these matrices, graphics systems efficiently transform entire scenes. Without linear algebra, modern computer graphics would be computationally impractical. Data Compression and Principal Component Analysis (PCA) When working with high-dimensional datasets (like thousands of variables per data point), Principal Component Analysis uses eigenvectors of the data's covariance matrix to find the directions of maximum variance. By projecting data onto the top eigenvectors, you can reduce dimensionality dramatically while retaining most information. This is essential for handling "big data." </extrainfo>