Jul 13, 2013

Squelch it out

Many air traffic and maritime radio channels, like 2182 kHz and VHF 16, are being monitored and continuously recorded by numerous vessels and ground stations. Even though gigabytes are now cheap, long recordings would have to be compressed to make them easier to move around.

There are some reasons not to use lossy compression schemes like MP3 or Vorbis. Firstly, radio recordings are used in accident investigations, and additional artifacts at critical moments could hinder investigation. Secondly, lossy compression performs poorly on noisy signals, because of the amount of entropy present. Let's examine some properties of such recordings that may aid in their lossless compression instead.

A common property of radio conversations is that the transmissions are short, and they're followed by sometimes very long pauses. Receivers often silence themselves in the absence of a carrier (known as 'squelching'), so there's nothing interesting to record during that time.

This can be exploited by replacing such quiet passages with absolute digital silence, that is a stream of zero-valued samples. Now, using lossless FLAC, passages of digital silence compress into a very small space because of run-length encoding; the encoder simply has to save the number of zero samples encountered.

Real-life test

Let's see how the method performs on an actual recording. A sample WAV file will be recorded from the radio, from a conversation on VHF FM. The SDR does the actual squelching, and then the PCM waveform will be piped through squelch, a silencer tool I wrote.

rtl_fm      -f 156.3M -p 96 -N -s 16k -g 49.2 -l 200 |\
  sox       -r 16k -t raw -e signed -b 16 -c 1 -V1 -\
               -r 11025 -t raw - sinc 200-4500 |\
  squelch   -l 5 -d 4096 -t 64 |\
  sox       -t raw -e signed -c 1 -b 16 -r 11025 - recording.wav

The waveform is sinc filtered so as to remove the DC offset resulting from FM demodulation without a PLL. This makes silence detection simpler. It also removes any noise outside the actual bandwidth of VHF voice transceivers. (The output of the first SoX at quiet passages is not digital silence; the resample to the common rate of 11,025 Hz introduces some LSB noise. This is what we're squelching in this example.)

30 minutes of audio was recorded. A carrier was on less than half of the time. FLAC compresses the file at a ratio of 0.174; this results in an average bitrate of 30 kbps, which is pretty good for lossless 16-bit audio. At this rate, a CD can fit almost 50 hours of recording.

12 comments:

  1. Note that significant countries such as the US have ceased monitoring 500 kHz and 2182 kHz distress frequencies. Marine channel 16 is still used frequently, however...

    ReplyDelete
  2. I can't get your program to work but I don't understand what is different from just using sox with the "silence" option like this:

    $ rtl_fm -M -f 137M -s 12k -l 280 | sox -t raw -r 12k -e signed-integer -b 16 -c 1 - output.ogg silence -l 1 0.1 0.1% -1 2.0 0.1%

    If you were running some GUI SDR application and pulseaudio, you could do the same thing like this after you routed the audio streams accordingly:

    $ sox -d -r 11025 -c 1 output.ogg silence -l 1 0.1 0.1% -1 2.0 0.1%

    ReplyDelete
    Replies
    1. By using SoX's silence, you lose the track of time.

      Delete
    2. I don't understand what you mean by that.

      Delete
    3. "silence" cuts the audio out. My "squelch" replaces it with zeroes. Number of samples is preserved.

      Delete
  3. Okay, I get it now. That is why it was not working in the manner I was thinking of. However, you did solve a long standing problem for me. I finally realized the the 'sinc' band pass filter along with 'silence' in sox can be used as an extremely effective way to ignore analog noise on the line-in of many motherboards. I've been testing this for a few days with a line-in source and it works great. Prior to using the 'sinc' filter I would have many problems with finding a good silence threshold percentage as well as the Alsa/Pulseaudio capture levels and source amplitude.

    $ rec -c 1 -r 22050 sinctest.mp3 sinc 200-4500 silence -l 1 0 1% -1 2.0 1%

    This filters out the low frequency analog noise which would normally create problems for the silence detection filter.

    ReplyDelete
  4. I use something like this to record a P25 radio system in my area. It logs each call into a folder with the year, month and date, the filename is the UNIX timestamp, radio ID, and talkgroup ID, saved in MP3. I then wrote a front end in PHP that parses this, and a has searchable database via 'Everything Search'. It's all done in Windows, the back end is written in batch. But your post reminded me that I can use rtl_fm instead of something clunky like SDR# in Windows. In case you're wondering, I have the db here http://sdrscan.com/

    ReplyDelete
  5. I am also messing around with audio streams using rtl_fm on the RPi, I use the following commands:

    rtl_fm -f 145.575M -s 16k -l 0 | lame --scale 2 --lowpass 4.0 -r -s 16.0 -m m - - | ezstream -c /etc/ezstream.xml

    rtl_fm is piped into lame, encoded into mp3 and sent to an icecast server. Squelch is broken, if you turn it up the stream stops, probably due to the data stream from rtl_fm stopping. Does Sox handle this differently? I don't need efficient archival of data, just a low bandwidth stream with squelch to kill the noise. Going to try using sox to see if this makes any difference.

    ReplyDelete
    Replies
    1. Read the comments above. SoX does something completely different, it doesn't output any samples when squelched.

      Delete
    2. I figured that out already. :) What I want to know is how to keep lame and ezstream generating data even when rtl_fm is squelched. That is something different than described on this page but your knowledge seems to be quite extensive so maybe you have a solution?

      Delete
    3. my rtl_fm does output a stream of zeroes when squelched

      Delete
    4. Well, my copy doesn't, piped some data into a file, when squelched it does not generate any data at all. Maybe you are running an older version or squelch or somehow it compiled differently.
      Anyway, have to figure out how to pad stdout with zeroes when idle or get a rtl_fm version that correctly outputs zeroes when squelched.
      Great weblog by the way, lots of interesting articles!

      Delete

To prevent spam, comments will only appear after moderation.