The covariance matrix is one of the most fundamental objects in statistics and machine learning. It captures how the features of your data vary together – not just how each one varies on its own, but how they move in tandem. It is the mathematical heart of Principal Component Analysis (PCA), the Mahalanobis distance, Gaussian models, and portfolio theory. If you understand the covariance matrix, you understand the shape of your data.
The Mahalanobis distance is a measure of the distance between a point and a distribution, or between two points, that accounts for the covariance structure of the data. Unlike Euclidean distance, it understands that features can have different scales and can be correlated.
Introduced by P. C. Mahalanobis in 1936, it is widely used in anomaly detection, classification, and any application where the shape of the data distribution matters.
The K-means algorithm is one of the simplest and most widely used clustering methods. It aims to find $K$ centroids that minimize the sum of squared distances from each data point to its nearest centroid.
K-means is iterative: assign points to the nearest centroid, update centroids to the mean of their points, repeat until convergence. Despite its simplicity, it is powerful for exploratory data analysis, image compression, and customer segmentation.
Pooling is an aggregation operation that transforms a set of input vectors (or a grid of numerical features) into a single representative vector – the so-called pooling vector. It captures the most essential information from the input while significantly reducing its dimensionality. Think of it as an "executive summary" of a data group – it keeps the core message but discards unnecessary fine-grained details.
For instance, imagine you have four students, each represented by a vector of 5 test scores. The pooling vector would be a single student with 5 scores that summarize the performance of the entire group. In computer vision, pooling shrinks image feature maps; in natural language processing, it turns a sequence of word embeddings into one compact semantic vector for the whole sentence.
The Gaussian distribution (also known as the Normal distribution) is the most fundamental probability distribution in statistics and machine learning. Its characteristic bell-shaped curve appears everywhere — from measurement errors in physics to heights and test scores in biology and education.
A single Gaussian is defined by two parameters: its mean ($\mu$), which sets the center of the curve, and its variance ($\sigma^2$), which determines the spread (width). However, real-world data is often multi-modal — it clusters around several distinct centers. This is where the Gaussian Mixture Model (GMM) comes in: it combines multiple Gaussian components, each with its own mean and variance, to model complex, multi-modal distributions.