Mahalanobis Distance: The Covariance-Aware Metric
The Mahalanobis distance is a measure of the distance between a point and a distribution, or between two points, that accounts for the covariance structure of the data. Unlike Euclidean distance, it understands that features can have different scales and can be correlated.
Introduced by P. C. Mahalanobis in 1936, it is widely used in anomaly detection, classification, and any application where the shape of the data distribution matters.
Part 1: Interactive 2D Mahalanobis Distance
The plot below shows synthetic data with correlated features. Use the slider to change the correlation – the Mahalanobis ellipse re-orients instantly. The black cross is the mean; the orange point is a test point whose distance we measure.
Figure 1: Mahalanobis distance. The dashed ellipse shows the 2σ contour; the orange point's distance is measured relative to this shape.
Part 2: When Euclidean Distance Fails – The Correlation Trap
Euclidean distance treats all directions equally. When features are correlated, this is wrong: a deviation along the correlation direction is normal, while a deviation perpendicular to it is surprising. The plot below shows how the two metrics diverge along the horizontal line y = 0.
Figure 2: Along y = 0, Euclidean distance grows linearly, but Mahalanobis distance grows faster in the direction where data is sparse.
Part 3: The Mahalanobis Distance – Definition
For a point \( \mathbf{x} \) and a distribution with mean \( \boldsymbol{\mu} \) and covariance matrix \( \mathbf{S} \), the Mahalanobis distance is defined as:
\[ d(\mathbf{x}, \boldsymbol{\mu}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^T \mathbf{S}^{-1} (\mathbf{x} - \boldsymbol{\mu})} \]Let's break this down:
- \( \mathbf{x} \) is the point we are measuring (a \(d\)-dimensional vector).
- \( \boldsymbol{\mu} \) is the mean of the distribution (the center).
- \( \mathbf{S} \) is the covariance matrix, capturing the variance of each feature and the correlations between features.
- \( \mathbf{S}^{-1} \) is its inverse – the key ingredient that "whitens" the space.
- \( (\mathbf{x} - \boldsymbol{\mu})^T \mathbf{S}^{-1} (\mathbf{x} - \boldsymbol{\mu}) \) is a quadratic form that measures the squared distance in the whitened space.
In other words, Mahalanobis distance first rotates the coordinate system so that the axes align with the principal directions of the data, then scales each axis by the inverse of its standard deviation, and finally computes the ordinary Euclidean distance in this new space. The result is a single number that tells you how many "standard deviations" away the point is – but in a multivariate sense.
If the covariance matrix is the identity (\( \mathbf{S} = \mathbf{I} \)), Mahalanobis distance reduces to Euclidean distance. So Mahalanobis is a generalization of Euclidean distance that respects the geometry of the data.
Connection to the Gaussian Distribution
If \( \mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{S}) \), the probability density is:
\[ p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2} |\mathbf{S}|^{1/2}} \exp\left(-\frac{1}{2} d^2(\mathbf{x}, \boldsymbol{\mu})\right) \]Taking the logarithm:
\[ \ln p(\mathbf{x}) = -\frac{1}{2} d^2(\mathbf{x}, \boldsymbol{\mu}) + \text{constant} \]So the squared Mahalanobis distance is proportional to the negative log-likelihood under a Gaussian model. This is why Mahalanobis distance is the natural metric for Gaussian data: points with large Mahalanobis distance are exactly the points with low probability.
Important: Mahalanobis distance is not a replacement for the Gaussian distribution. It is a geometric measure that the Gaussian model naturally gives rise to. You can compute it without assuming normality, but if you want to interpret it as a probability, you need the Gaussian assumption.
How to Estimate the Covariance Matrix?
The covariance matrix \( \mathbf{S} \) must be estimated from data. Common approaches:
- Pooled covariance: Use all data (or all classes) to estimate a single \( \mathbf{S} \).
- Class-specific covariance: Estimate \( \mathbf{S}_k \) for each class \( k \) – this leads to quadratic discriminant analysis (QDA).
- Regularized covariance: Add a small value \( \lambda \) to the diagonal to make \( \mathbf{S} \) invertible when data is scarce or high-dimensional.
- Diagonal covariance: Assume features are independent – then Mahalanobis reduces to a weighted Euclidean distance.
Applications
- Anomaly detection: Points with large Mahalanobis distance from the mean are outliers.
- Classification: Linear discriminant analysis (LDA) uses Mahalanobis distance with a pooled covariance.
- Speaker recognition: Acoustic feature vectors (MFCCs) are compared using Mahalanobis distance.
- Image processing: Color similarity in correlated RGB space.
- Finance: Detecting unusual market conditions in correlated asset returns.
- Acoustic feature vectors: When applied to two acoustic vectors, the Mahalanobis distance gives a single scalar that measures how "surprising" their difference is, given how the features vary and correlate across the dataset.
Common Pitfalls
- Requires \( \mathbf{S} \) to be invertible – fails when features outnumber samples or features are collinear.
- Sensitive to outliers when estimating \( \boldsymbol{\mu} \) and \( \mathbf{S} \).
- Assumes the data distribution is approximately Gaussian for probabilistic interpretation.
- Computing \( \mathbf{S}^{-1} \) is \( O(d^3) \) – expensive for very high dimensions.
- Not a replacement for Gaussian distributions – it is a metric, not a probability model.
The Takeaway
Mahalanobis distance is a powerful, covariance-aware metric that generalizes Euclidean distance. It is the natural choice when features are correlated or have different scales, and it forms the geometric foundation of Gaussian-based classification and anomaly detection. Understanding its relationship to the Gaussian distribution helps you apply it correctly – as a distance measure first, and as a probabilistic tool only when the Gaussian assumption holds.