The Gaussian distribution (also known as the Normal distribution) is the most fundamental probability distribution in statistics and machine learning. Its characteristic bell-shaped curve appears everywhere — from measurement errors in physics to heights and test scores in biology and education.
A single Gaussian is defined by two parameters: its mean ($\mu$), which sets the center of the curve, and its variance ($\sigma^2$), which determines the spread (width). However, real-world data is often multi-modal — it clusters around several distinct centers. This is where the Gaussian Mixture Model (GMM) comes in: it combines multiple Gaussian components, each with its own mean and variance, to model complex, multi-modal distributions.
Part 1: The Univariate Gaussian (Bell Curve)
The probability density function (PDF) of a univariate Gaussian is given by the formula:
\[ f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(x - \mu)^2}{2\sigma^2} \right) \]Play with the sliders below to change the Mean ($\mu$) and Standard Deviation ($\sigma$). The curve shifts horizontally with the mean and becomes taller/narrower or shorter/wider with the standard deviation.
Figure 1: The classic bell curve. 68-95-99.7 rule: ~68% of the data lies within ±1σ of μ.
Part 2: The Multivariate Gaussian (2D)
When we have $d$ variables, the univariate formula generalises to the multivariate Gaussian (also called the multivariate normal distribution). The PDF is:
\[ f(\mathbf{x} \mid \boldsymbol{\mu}, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \exp\left( -\frac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^T \Sigma^{-1} (\mathbf{x} - \boldsymbol{\mu}) \right) \]Here, $\boldsymbol{\mu}$ is the mean vector, $\Sigma$ is the covariance matrix (which we covered in a previous article), and $|\Sigma|$ is its determinant. The covariance matrix determines the shape and orientation of the "mountain" in $d$-dimensions.
The contour plot below shows a 2D Gaussian with a non-diagonal covariance — the ellipse tilts because the variables are correlated ($\text{Cov}(X,Y) = 0.7$).
Figure 2: A bivariate Gaussian. The peak is at (0,0). The elliptic contours indicate positive covariance — X and Y increase together.
Part 3: Gaussian Mixture Models (GMM) — Combining Gaussians
A single Gaussian is often insufficient for complex data. A GMM represents a distribution as a weighted sum of $K$ Gaussian components. The full model is given by:
\[ p(x) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(x \mid \mu_k, \sigma_k^2) \]where the mixing coefficients (weights) satisfy $\sum_{k=1}^{K} \pi_k = 1$ and $\pi_k \ge 0$. Each component $\mathcal{N}(x \mid \mu_k, \sigma_k^2)$ is a Gaussian with its own mean and variance.
Below, we have 3 components (blue, orange, green dashed lines). The red solid line is the resulting mixture density.
Try it: Move the sliders for Component 1 and 2 (weights, means, variances). Component 3 automatically takes the remaining weight. See how the mixture (red line) morphs to create bimodal or trimodal shapes!
Comp 1
Comp 2
Comp 3
Figure 3: A Gaussian Mixture Model with 3 components. The red curve is the mixture — it can model bimodal or even trimodal distributions.
Why are Gaussian Mixture Models so powerful?
GMMs are one of the most versatile tools in unsupervised learning and density estimation:
- Soft Clustering: Unlike K-Means (which assigns each point to exactly one cluster), GMM assigns probabilities (soft assignments) to each cluster. A point in the middle might be 60% Cluster A and 40% Cluster B.
- Density Estimation: They can approximate any continuous distribution arbitrarily well, given enough components. This makes them ideal for anomaly detection — if a new point has extremely low density under the GMM, it's an outlier.
- Generative Modelling: You can sample new synthetic data points from the learned mixture distribution, which is useful for data augmentation and simulation.
- Applications: Speaker identification (in speech processing), image segmentation (each region has a different color distribution), and financial risk modelling (different market regimes).
How GMMs learn — the EM Algorithm
Fitting a GMM to data is done via the Expectation-Maximization (EM) algorithm:
- E-Step: Estimate the probability that each data point belongs to each component (given current parameters).
- M-Step: Update the parameters (weights, means, variances) using the weighted average of the points assigned to each component.
We repeat these two steps until the parameters converge. This algorithm is guaranteed to find a local optimum, but it can get stuck in poor local optima — so it's common to run it multiple times with different random initializations.
Limitations and common pitfalls
- Number of components (K): You must choose K in advance. This is often done using criteria like AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion).
- Curse of Dimensionality: In high-dimensional spaces, GMMs require a massive number of parameters (the covariance matrix grows as $O(K \times d^2)$). This can lead to overfitting and numerical instability.
- Singularities: If a component collapses onto a single data point (variance $\to 0$), the likelihood becomes infinite. This is usually avoided by using regularization (adding a small value to the diagonal of the covariance matrix).
- Assumption of Gaussianity: GMM assumes each component is Gaussian. If the underlying sub-populations are heavily skewed or have heavy tails, other mixture models (e.g., Student-t mixtures) might be more appropriate.
The takeaway
The Gaussian distribution is the mathematical foundation of parametric statistics. It is simple, elegant, and fully described by its mean and covariance.
But real data is often messy. A Gaussian Mixture Model takes this simple building block and stacks them together to model the complexity of the real world. Whether you are identifying customer segments, detecting fraudulent transactions, or decoding speech, GMMs provide a probabilistic, interpretable way to capture the hidden structure in your data.
Remember: One bell curve is simple and pure; many bell curves together can model almost anything.