Jul 13, 2013

Squelch it out

Many air traffic and maritime radio channels, like 2182 kHz and VHF 16, are being monitored and continuously recorded by numerous vessels and ground stations. Even though gigabytes are now cheap, long recordings would have to be compressed to make them easier to move around.

There are some reasons not to use lossy compression schemes like MP3 or Vorbis. Firstly, radio recordings are used in accident investigations, and additional artifacts at critical moments could hinder investigation. Secondly, lossy compression performs poorly on noisy signals, because of the amount of entropy present. Let's examine some properties of such recordings that may aid in their lossless compression instead.

A common property of radio conversations is that the transmissions are short, and they're followed by sometimes very long pauses. Receivers often silence themselves in the absence of a carrier (known as 'squelching'), so there's nothing interesting to record during that time.

This can be exploited by replacing such quiet passages with absolute digital silence, that is a stream of zero-valued samples. Now, using lossless FLAC, passages of digital silence compress into a very small space because of run-length encoding; the encoder simply has to save the number of zero samples encountered.

Real-life test

Let's see how the method performs on an actual recording. A sample WAV file will be recorded from the radio, from a conversation on VHF FM. The SDR does the actual squelching, and then the PCM waveform will be piped through squelch, a silencer tool I wrote.

rtl_fm      -f 156.3M -p 96 -N -s 16k -g 49.2 -l 200 |\
  sox       -r 16k -t raw -e signed -b 16 -c 1 -V1 -\
               -r 11025 -t raw - sinc 200-4500 |\
  squelch   -l 5 -d 4096 -t 64 |\
  sox       -t raw -e signed -c 1 -b 16 -r 11025 - recording.wav

The waveform is sinc filtered so as to remove the DC offset resulting from FM demodulation without a PLL. This makes silence detection simpler. It also removes any noise outside the actual bandwidth of VHF voice transceivers. (The output of the first SoX at quiet passages is not digital silence; the resample to the common rate of 11,025 Hz introduces some LSB noise. This is what we're squelching in this example.)

30 minutes of audio was recorded. A carrier was on less than half of the time. FLAC compresses the file at a ratio of 0.174; this results in an average bitrate of 30 kbps, which is pretty good for lossless 16-bit audio. At this rate, a CD can fit almost 50 hours of recording.

7 comments:

  1. Note that significant countries such as the US have ceased monitoring 500 kHz and 2182 kHz distress frequencies. Marine channel 16 is still used frequently, however...

    ReplyDelete
  2. I can't get your program to work but I don't understand what is different from just using sox with the "silence" option like this:

    $ rtl_fm -M -f 137M -s 12k -l 280 | sox -t raw -r 12k -e signed-integer -b 16 -c 1 - output.ogg silence -l 1 0.1 0.1% -1 2.0 0.1%

    If you were running some GUI SDR application and pulseaudio, you could do the same thing like this after you routed the audio streams accordingly:

    $ sox -d -r 11025 -c 1 output.ogg silence -l 1 0.1 0.1% -1 2.0 0.1%

    ReplyDelete
    Replies
    1. By using SoX's silence, you lose the track of time.

      Delete
    2. I don't understand what you mean by that.

      Delete
    3. "silence" cuts the audio out. My "squelch" replaces it with zeroes. Number of samples is preserved.

      Delete
  3. Okay, I get it now. That is why it was not working in the manner I was thinking of. However, you did solve a long standing problem for me. I finally realized the the 'sinc' band pass filter along with 'silence' in sox can be used as an extremely effective way to ignore analog noise on the line-in of many motherboards. I've been testing this for a few days with a line-in source and it works great. Prior to using the 'sinc' filter I would have many problems with finding a good silence threshold percentage as well as the Alsa/Pulseaudio capture levels and source amplitude.

    $ rec -c 1 -r 22050 sinctest.mp3 sinc 200-4500 silence -l 1 0 1% -1 2.0 1%

    This filters out the low frequency analog noise which would normally create problems for the silence detection filter.

    ReplyDelete
  4. I use something like this to record a P25 radio system in my area. It logs each call into a folder with the year, month and date, the filename is the UNIX timestamp, radio ID, and talkgroup ID, saved in MP3. I then wrote a front end in PHP that parses this, and a has searchable database via 'Everything Search'. It's all done in Windows, the back end is written in batch. But your post reminded me that I can use rtl_fm instead of something clunky like SDR# in Windows. In case you're wondering, I have the db here http://sdrscan.com/

    ReplyDelete

To prevent spam, comments will only appear after moderation.