Modulation Spectra
So far we have looked at features that describe the spectrum at a single moment in time. But sound changes over time — speech has rhythm, music has tempo, and mechanical vibrations evolve. The modulation spectrum captures how the spectrum itself changes over time.
Think of it as a "spectrum of a spectrum." First we compute the regular spectrum (frequency vs. amplitude). Then we look at how the energy in each frequency band fluctuates over time, and we compute the spectrum of those fluctuations. The result is a two-dimensional representation: one axis is the acoustic frequency (the original spectrum), and the other is the modulation frequency (how fast the energy changes).
This is particularly useful because different types of signals have different modulation patterns. Speech, for example, has strong modulation in the 4–8 Hz range (the rate of syllables). Music has modulation at the tempo (1–3 Hz) and at note onsets (10–30 Hz). A steady tone has almost no modulation at all. By examining the modulation spectrum, we can distinguish between these cases without needing to recognize the actual content.
Computing the Modulation Spectrum
The modulation spectrum is computed in two steps:
- Time-frequency representation: We start with a spectrogram — a sequence of short-time spectra over time. This gives us energy P(t, f), where t is the frame index and f is the frequency bin.
- Modulation analysis: For each frequency bin f, we take the time-varying energy P(t, f) and apply a Fourier transform along the time axis. This reveals the modulation frequencies present in that band.
The result is M(ω, f), where ω is the modulation frequency (in Hz) and f is the acoustic frequency. This is the modulation spectrum, often visualized as a heatmap.
Interpreting the Heatmap
In the heatmap below, the vertical axis shows acoustic frequency (low to high), and the horizontal axis shows modulation frequency (how fast the energy changes). Brighter areas indicate stronger modulation activity.
- Low modulation frequencies (0–5 Hz): Slow changes — rhythm, tempo, syllable rate.
- Mid modulation frequencies (5–20 Hz): Note onsets, transients, fast speech.
- High modulation frequencies (> 20 Hz): Rapid fluctuations, often noise-like.
Modulation Spectrum Visualization
The interactive heatmap below shows a simulated modulation spectrum. The data is synthetic, but it mimics the typical pattern of a speech signal — strong modulation in the low frequencies (4–8 Hz) across the mid-range acoustic frequencies.
What We See in the Heatmap
The heatmap above is a simulation, but it illustrates the key features of a modulation spectrum:
- Bright region at 4–8 Hz modulation: This is typical for speech — the rhythm of syllables. The energy is concentrated in the mid-frequency acoustic range (500–2000 Hz), which is where most speech energy lies.
- Very low modulation (< 2 Hz): This corresponds to slow changes like musical tempo or the overall envelope of a phrase.
- High modulation (> 20 Hz): Often associated with noise-like signals, such as fricatives (/s/, /ʃ/) in speech or percussion in music.
Practical Applications
The modulation spectrum is used in several important tasks:
- Speech/music discrimination: Speech has a characteristic modulation peak at 4–8 Hz, while music has its main modulation at the tempo (1–3 Hz) and at note onsets (10–30 Hz).
- Emotion recognition in speech: Fast modulation (high frequency) is associated with excitement or anger; slow modulation with calmness or sadness.
- Music genre classification: Different genres have different modulation patterns — dance music has strong tempo-related modulation, classical music has smoother, slower modulation.
- Audio enhancement: By identifying and modifying the modulation spectrum, we can reduce noise or emphasize certain temporal features.
The modulation spectrum is a powerful tool because it captures the temporal dynamics of the signal — something that static spectral features miss entirely. It bridges the gap between frequency-domain analysis and time-domain behavior.