Squelch it out

Many air traffic and maritime radio channels, like 2182 kHz and VHF 16, are being monitored and continuously recorded by numerous vessels and ground stations. Even though gigabytes are now cheap, long recordings would have to be compressed to make them easier to move around.

There are some reasons not to use lossy compression schemes like MP3 or Vorbis. Firstly, radio recordings are used in accident investigations, and additional artifacts at critical moments could hinder investigation. Secondly, lossy compression performs poorly on noisy signals, because of the amount of entropy present. Let's examine some properties of such recordings that may aid in their lossless compression instead.

A common property of radio conversations is that the transmissions are short, and they're followed by sometimes very long pauses. Receivers often silence themselves in the absence of a carrier (known as 'squelching'), so there's nothing interesting to record during that time.

This can be exploited by replacing such quiet passages with absolute digital silence, that is a stream of zero-valued samples. Now, using lossless FLAC, passages of digital silence compress into a very small space because of run-length encoding; the encoder simply has to save the number of zero samples encountered.

Update 11/2015: It seems many versions of rtl_fm now do this automatically, i.e. they don't output anything when squelched unless instructed to do so.

Real-life test

Let's see how the method performs on an actual recording. A sample WAV file will be recorded from the radio, from a conversation on VHF FM. The SDR does the actual squelching, and then the PCM waveform will be piped through squelch, a silencer tool I wrote.

rtl_fm      -f 156.3M -p 96 -N -s 16k -g 49.2 -l 200 |\
  sox       -r 16k -t raw -e signed -b 16 -c 1 -V1 -\
               -r 11025 -t raw - sinc 200-4500 |\
  squelch   -l 5 -d 4096 -t 64 |\
  sox       -t raw -e signed -c 1 -b 16 -r 11025 - recording.wav

The waveform is sinc filtered so as to remove the DC offset resulting from FM demodulation without a PLL. This makes silence detection simpler. It also removes any noise outside the actual bandwidth of VHF voice transceivers. (The output of the first SoX at quiet passages is not digital silence; the resample to the common rate of 11,025 Hz introduces some LSB noise. This is what we're squelching in this example.)

30 minutes of audio was recorded. A carrier was on less than half of the time. FLAC compresses the file at a ratio of 0.174; this results in an average bitrate of 30 kbps, which is pretty good for lossless 16-bit audio. At this rate, a CD can fit almost 50 hours of recording.

Update 1/2019: A similar effect can also be achieved using the command-line tool csdr. It supports different sample formats and even a dynamically controllable squelch level.

Reader challenge explained

As I promised, here's a brief explanation of the reader challenge I posted earlier. (Deadlines in the summer? Baaad idea.) I will leave decoding as homework though.

Spoiler alert!

I. Almost trivial

This is a simple substitution cipher using my haxor handle (repeated characters removed) as the key:

WINDYTABCEFGHJKLMOPQRSUVXZ
ABCDEFGHIJKLMNOPQRSTUVWXYZ

Reversal is almost trivial due to the distinct pattern it leaves in the letter frequency spectrum. "Almost", because there is an added challenge in the language used.

II. Shouldn't be very hard either

This message was transformed into audio with the Hellschreiber teleprinter. There's no further encryption. Some people noticed that there's a hint in the audio metadata; this was unintentional.

III. Might take some effort

A visual encryption scheme. A message is encoded in the difference of these two random dot patterns. I tried to add noise into the encoded message to make the resulting similarities in the patterns less obvious to the eye.

IV. NSA-grade

Okay, it's not really NSA-grade. But it's a Playfair cipher, using the following key:

$square =
  "WINDY".
  "ABCEF".
  "GHKLM".
  "OPQRS".
  "TUVXZ";

Whitespaces and punctuation was removed, and full stops were converted to "STOP".

I was surprised at how many people actually reversed this!