Introduction to Computer Vision

Computer Vision: Understanding Visual Intelligence in Machines What is Computer Vision? Computer vision is a field of artificial intelligence that enables computers to interpret and understand visual information from digital images and video. Rather than simply storing pixel data, computer vision systems extract meaningful knowledge from raw visual information to answer important questions: What objects are present in a scene? Where are they located? How are they moving? The fundamental goal is to transform low-level pixel data into high-level, actionable knowledge that machines can use for decision-making, automation, or interaction with humans. How Images Are Represented Before a computer can understand an image, we need a standard way to represent visual data digitally. Understanding image representation is essential for all downstream computer vision tasks. Grayscale and Color Images The simplest image format is grayscale, where each pixel (the smallest unit of an image) has a single intensity value representing brightness. These values typically range from 0 (black) to 255 (white). Color images extend this concept by storing three separate intensity values for each pixel, one each for red, green, and blue (RGB). By combining different intensities of these three colors, we can represent millions of distinct colors. Each color channel works independently, with values again typically ranging from 0 to 255. Resolution and Bit Depth Two key properties describe image data: Resolution refers to the dimensions of an image in pixels. An image with resolution 1920 × 1080 contains 1920 pixels horizontally and 1080 pixels vertically. Higher resolution means more detail but also more data to process. Bit depth specifies how many bits of information are used to store each pixel's value. An 8-bit image uses 8 bits per channel, allowing $2^8 = 256$ different intensity levels. A 32-bit image might use 8 bits each for red, green, blue, and transparency (alpha channel). Image Preprocessing: Preparation for Analysis Raw image data often contains noise, inconsistencies, or irrelevant information. Before extracting meaningful features, we apply preprocessing operations to enhance the image for analysis. Scaling and Cropping Scaling resizes an image to a different resolution. This is useful when you need consistent input sizes for processing or when you want to reduce computational load by working with smaller images. Scaling can preserve an image's aspect ratio (the relationship between width and height) or deliberately change it. Cropping extracts a rectangular sub-region from an image to focus computational effort on a region of interest. For example, if you're analyzing a photograph with multiple faces but only care about one, you could crop to just that face. Smoothing: Reducing Noise Images captured from cameras often contain random noise—slight variations in pixel values that don't represent real features. Smoothing filters (also called blur filters) reduce this noise by averaging nearby pixel values. The most common is the Gaussian blur, which weights nearby pixels more heavily than distant ones, creating a natural-looking smoothing effect. The trade-off is that smoothing also removes fine details. Too much smoothing destroys important information, while too little leaves noise that interferes with downstream analysis. Edge Detection: Finding Object Boundaries Objects in images are distinguished by their boundaries, where pixel intensity changes sharply. Edge detection identifies these sharp transitions. Common edge detection operators include: Sobel detector: Uses gradient operators to find intensity changes in both horizontal and vertical directions Canny detector: A multi-stage algorithm that detects edges while suppressing noise and avoiding false detections Edge detection is particularly valuable because object boundaries are often the most informative parts of an image for understanding what is present and where objects are located. Feature Extraction: Finding Meaningful Patterns Once an image is preprocessed, the next step is feature extraction—identifying distinctive patterns and characteristics that computers can use to understand image content. Features are compact representations of what's important in an image. Traditional Hand-Crafted Descriptors For decades, computer vision engineers manually designed feature descriptors based on intuition about what distinguishes different objects: Edge detectors (discussed above) capture object boundaries Corner detectors find distinctive point features where edges meet at angles Scale Invariant Feature Transform (SIFT) identifies distinctive keypoints that remain recognizable even if images are rotated or viewed at different scales Histogram of Oriented Gradients (HOG) describes the distribution of edge directions in local image regions These hand-crafted features work well when humans have good intuition about what makes objects distinct. However, they require manual engineering expertise and don't automatically adapt to new problem domains. Deep Learning and Automatic Feature Learning Modern computer vision relies on Convolutional Neural Networks (CNNs), which automatically learn features directly from data. Rather than designing features by hand, CNNs discover which features are useful by training on large datasets. Hierarchical Feature Learning A key insight of deep learning is that vision naturally decomposes into hierarchies: Early layers capture simple patterns: edges, corners, textures, and basic colors Middle layers combine these simple patterns into more complex features: shapes, parts of objects, distinctive patterns Deep layers recognize high-level concepts: complete objects, scenes, semantic relationships This hierarchical structure mirrors how the biological visual system works. Your brain doesn't process all visual information at once; instead, it builds up understanding from simple elements to complex concepts. Why Learned Features Excel Learned features have significant advantages: They automatically adapt to the specific data and task at hand They often achieve higher accuracy than fixed, hand-designed descriptors They can discover non-obvious patterns that humans wouldn't think to design They improve as more training data becomes available Core Vision Tasks Computer vision tackles several fundamental problems. Understanding these core tasks is essential because they represent the main applications of visual understanding. Image Classification The simplest task is image classification: assigning a single label to an entire image. Given a photo, classification answers: "Is this a cat or a dog?" or "Is this a benign or malignant tumor?" The output is typically one label covering the whole image. Object Detection Real-world scenes usually contain multiple objects of interest. Object detection goes beyond classification by locating multiple objects and labeling each one. Detection typically produces bounding boxes—rectangular regions around each detected object—along with class labels. For example, detecting pedestrians in a street scene requires identifying not just "there are people here" but "there is a person at location A, another at location B," etc. The image above illustrates a bounding box (shown as a green rectangle) around the stop sign, which is exactly what object detection produces. Image Segmentation For detailed scene understanding, we often need pixel-level analysis. Image segmentation partitions an image into meaningful regions: Semantic segmentation assigns a class label to each pixel. All pixels belonging to trees get one label, all pixels belonging to roads get another, etc. Note that semantic segmentation doesn't distinguish between multiple instances—all trees are labeled identically. Instance segmentation goes further by distinguishing separate instances of the same class. It identifies not just "this is a tree" but "this is tree #1, and this is tree #2." These images show examples of different feature patterns in various image regions, and how these patterns might be extracted at different granularities. Motion Analysis In video sequences, understanding how objects move is crucial. Motion analysis tracks how objects change position and deform over time. Techniques include: Optical flow: Estimates the apparent motion of brightness patterns across frames Object tracking: Follows the same object across multiple video frames <extrainfo> Motion analysis is particularly useful for video understanding and autonomous systems that need to predict future motion based on past patterns. </extrainfo> Learning Paradigms: How Systems Acquire Knowledge Computer vision systems learn from data in different ways. Understanding these paradigms helps explain why different approaches work for different problems. Supervised Learning In supervised learning, we train models using images that have been explicitly labeled with correct answers. For instance, training an image classifier requires thousands of images where each image is labeled "cat," "dog," etc. During training, the system learns to recognize patterns associated with each label. Supervised learning typically produces the highest accuracy but requires expensive manual labeling effort. Unsupervised Learning Unsupervised learning discovers structure in image data without explicit labels. Common approaches include clustering (grouping similar images together) and dimensionality reduction (finding compact representations that capture essential variation). Unsupervised learning is useful for exploration and understanding data, but it's less precise than supervised learning because there's no ground truth to train against. <extrainfo> Self-Supervised Learning A newer paradigm, self-supervised learning, creates training signals automatically from unlabeled data. Rather than requiring manual labels, the system creates proxy tasks from the images themselves. For example, a system might learn to predict what comes next in a video sequence, or to recognize that rotated versions of the same image are related. This approach dramatically reduces annotation burden while still enabling effective learning. </extrainfo> From Rules to Learning Historically, computer vision relied on rule-based algorithms: hand-crafted procedures that explicitly encoded how to solve vision problems. Modern computer vision has shifted to data-driven methods that learn patterns from large datasets. This transition occurred because: Learning from data automatically discovers patterns humans wouldn't think to code Data-driven systems improve as more data becomes available Learned systems generalize better to new situations than rigid rule-based approaches The Vision Pipeline: Putting It All Together A typical computer vision system follows a consistent workflow: Input and Preprocessing: Acquire raw images, then apply scaling, cropping, smoothing, and other operations to prepare them for analysis Feature Extraction: Extract meaningful patterns using either hand-crafted descriptors or deep learning feature learning Model Training: Train a model (classifier, detector, segmentation network, etc.) using labeled data to learn the task Evaluation and Inference: Test the model on new, unseen images to verify it works and deploy it for real applications This pipeline represents the high-level structure underlying most computer vision applications. While individual steps may be modified or omitted for specific problems, this framework captures the essential flow from raw pixels to actionable predictions. <extrainfo> Applications of Computer Vision Computer vision powers numerous real-world applications: Facial Recognition Facial recognition systems identify or verify individuals by analyzing facial features. These systems detect faces in images, extract facial features (distances between eyes, nose shape, etc.), and compare them to a database of known faces. Autonomous Driving Self-driving vehicles rely heavily on computer vision to: Detect lanes and track road position Identify pedestrians, cyclists, and other vehicles Recognize traffic signs and signals Estimate distances to obstacles Medical Imaging In healthcare, computer vision identifies abnormalities in radiology scans (X-rays, CT scans, MRI images), detecting tumors, fractures, and other pathologies. These systems can match or exceed radiologist accuracy while processing images far faster. Augmented Reality Augmented reality systems use computer vision to understand the real environment in live video, then overlay computer-generated graphics that align with the scene. Applications range from virtual furniture placement to gaming to industrial maintenance guidance. These applications demonstrate that computer vision has moved from academic research to practical systems affecting daily life. </extrainfo>