I've been experimenting with methods for visualising harmony, intonation (tuning), and overtones in music. Ordinary spectrograms aren't very well suited for that as the harmonic relations are not intuitively visible. Let's see what could be done about this. I'll try to sprinkle the text with Wikipedia links in order to immerse (nerd snipe?) the reader in the subject.
Equal temperament cents against time
We can examine how tuning evolves during a recording by choosing a reference pitch and plotting all frequencies relative to it modulo 100 cents. This is similar to what an electronic tuner does, but instead of just showing the fundamental frequency, we'll plot the whole spectrum. Information about the absolute frequencies is lost. This "zoomed-in" plot visualises how the distribution of frequencies fits the 12-tone equal temperament system (12-TET) common in Western music.
Here's the first 20 seconds of Debussy's Clair De Lune as found on YouTube, played with a well-tuned (video) and an out-of-tune piano (video). The second piano sounds out of tune because there are relative differences in tuning between the strings. The first piano looks to be a few cents sharp as a whole, but consistently so, so it's not perceptible.
The vertical axis is the same that electronic tuners use. All the notes of a chord will appear in the middle, as long as they are well tuned against the reference pitch of, say, A = 440 Hz. The top edge of the graph is half a semitone sharp (quarter tone = 50c), and the bottom is half a semitone flat.
Overtones barely appear in the picture because the first three conveniently hit other notes in tune. But from f5 onwards the harmonic series starts deviating from 12-TET and the harmonics start to look out-of-tune (f5 = −14c, f7 = −31c, ...). These can be cut out by filtering, or hoping that they're lower in volume and setting the color range accordingly.
Six months later...
You know how you sometimes accidentally delete a project and have to rewrite it from scratch much later, and it's never exactly the same? That's what happened at this point. It's a little scuffed, but here's some piano music that utilises quarter tones (by Wyschnegradsky). I used the same scale on purpose, so the quarter tones wrap around from the top and bottom.
More examples of these "intonation plots" in a tweet.
This suits well for piano music. However, not even all western classical music is tuned to equal temperament; for instance, solo strings may be played with Pythagorean intonation[1], whereas vocal ensembles[2] and string quartets may tune some intervals closer to just intonation. Unlike the piano, these wouldn't look too good in the equal-note plot.
Octave spiral
If we instead plot the spectrum modulo 1200 cents (1 octave) we get an interesting interval view. We could even ditch the time scale and wind the spectrogram into a spiral to make it prettier and preserve the overtones and absolute frequency information. Now each note is represented by an angle in this spiral; in 12-TET, they're separated by 30 degrees. At any point in the spiral, going 1 turn outwards doubles the frequency.
Here's a C major chord on a piano. Note how the harmonic content adds several high-frequency notes on top of the C, E, and G of the triad chord, and how multiple notes can contribute to the same overtone:
I had actually hoped I would get an Ultimate Chord Viewer where any note, regardless of frequency, would have all its harmonics neatly stacked on top of it. But it's not what has happened here: the harmonic series is not a stack of octaves (2^n), but instead of integer multiples (n). Some harmonics appear at seemingly unrelated angles. But it's still a pretty interesting visualisation, and perhaps makes more sense musically.
This plot is also better at illustrating different tuning systems. Let's look at a major third interval F-A in equal temperament and just intonation, with a few more harmonics.
The intuition from this plot is that equal temperament aims for equal distances (angles) between notes and just intonation tries to make more of the harmonics match instead.
Even though the Ultimate Chord Viewer was not achieved I now have ideas for Christmas lights...
Live visualization
Here's what a cappella music with some reverb looks like on the spiral spectrogram.
A shader toy
This little real-time GLSL demo on shadertoy draws a spiral FFT from microphone audio. But don't get your hopes up: The spectrograms in this blog post were made with a 65536-point FFT. Shadertoy's 512px microphone texture offers a lot less in terms of frequency range and bins. This greatly blurs the frequency resolution, especially towars the low end. Could it be improved with the right colormap? Or a custom FFT with the waveform texture as its input?
References
- Loosen, F. (1994): Tuning of diatonic scales by violinists, pianists, and nonmusicians. Perception & Psychophysics 56(2): 221–226.
- D'Amario, S., Howard, D.M., Daffern, H., Pennill, N. (2020): A Longitudinal Study of Intonation in an a cappella Singing Quintet. Journal of Voice 34(1): 159.e13–159.e27.
What's your opinion on "Constant Q transform"? Compared to DFT, it returns the frequencies on a logarithmic scale instead of linear, which could be useful for music... but if one can afford large windows, perhaps an FFT is all one needs.
ReplyDeleteThanks, I hadn't heard of that! Looks really interesting for the purpose.
DeleteThis was really cool! I like how the tidy ideals of equal temperament bump up against the messy realities of harmonics
ReplyDeleteHave you looked into Stabilized Auditory Images? Richard Lyon shows some wild visualizations in this talk, and discusses their open-source toolkit, which I've been meaning to dig into
Thanks for the recommendation, just watched it. He seems to have done a lot of interesting research. Correlation plots would be worth experimenting with for these musical spirals, too.
DeleteHe certainly has. I'm working my way through his book Human and Machine Hearing (there's a free PDF version), and it is *i n t e n s e.* I've made it through the complete run-down of the theory of hearing and its history, but I haven't gotten far in the proposed computational models.
DeleteCorrelations would be interesting! I'd love to see how piano music would contrast with choral/wind/string ensembles, where musicians adjust their intonation by ear chord-by-chord. I'd expect it to deviate from fixed temperament, and that a note's frequency would shift as its harmonic function changes. (I notice now that you cited a paper on choral intonation analysis. The "Rehearsal strategies..." section is interesting, that they had a bit of insight into the intentions of highly trained musicians in this regard.)
Harmonics should be on a (logarithmic) spiral, so adding that to the visualization may help. I tried to make something similar to your octave spiral viewer in the past (the idea is pretty straightforward), but I haven't figured out a good view to view those spiral harmonics, in particular because the lower order harmonics are spread around the circle.
ReplyDeleteAnother interesting thing to do is to do an "equal-tempered" pentatonic division and compare that to the 12-tone scales.
Thanks, nice!
ReplyDeleteThe octave spiral is essential. When I came upon it early in this century I thought it was my discovery but I have realized that its history reaches at least a century back. I have written about the history.
In collaboration with programmer Marco Gianotta I have made an interactive version available at the following link:
https://suonoterapia.org/overtones/
Maybe you will find the another of Marco's programs interesting as well:
https://suonoterapia.org/sound-of-color/
Cheers
Beautiful! Thanks for the valuable comment!
DeleteHi. Do you have any info on how to get the newer baudline version? I tried contacting sigblips but have no response.
ReplyDeleteThanks in advance!
Hi, I don't, unfortunately.
Delete