Jul 13, 2013

Squelch it out

Many air traffic and maritime radio channels, like 2182 kHz and VHF 16, are being monitored and continuously recorded by numerous vessels and ground stations. Even though gigabytes are now cheap, long recordings would have to be compressed to make them easier to move around.

There are some reasons not to use lossy compression schemes like MP3 or Vorbis. Firstly, radio recordings are used in accident investigations, and additional artifacts at critical moments could hinder investigation. Secondly, lossy compression performs poorly on noisy signals, because of the amount of entropy present. Let's examine some properties of such recordings that may aid in their lossless compression instead.

A common property of radio conversations is that the transmissions are short, and they're followed by sometimes very long pauses. Receivers often silence themselves in the absence of a carrier (known as 'squelching'), so there's nothing interesting to record during that time.

This can be exploited by replacing such quiet passages with absolute digital silence, that is a stream of zero-valued samples. Now, using lossless FLAC, passages of digital silence compress into a very small space because of run-length encoding; the encoder simply has to save the number of zero samples encountered.

Real-life test

Let's see how the method performs on an actual recording. A sample WAV file will be recorded from the radio, from a conversation on VHF FM. The SDR does the actual squelching, and then the PCM waveform will be piped through squelch, a silencer tool I wrote.

rtl_fm      -f 156.3M -p 96 -N -s 16k -g 49.2 -l 200 |\
  sox       -r 16k -t raw -e signed -b 16 -c 1 -V1 -\
               -r 11025 -t raw - sinc 200-4500 |\
  squelch   -l 5 -d 4096 -t 64 |\
  sox       -t raw -e signed -c 1 -b 16 -r 11025 - recording.wav

The waveform is sinc filtered so as to remove the DC offset resulting from FM demodulation without a PLL. This makes silence detection simpler. It also removes any noise outside the actual bandwidth of VHF voice transceivers. (The output of the first SoX at quiet passages is not digital silence; the resample to the common rate of 11,025 Hz introduces some LSB noise. This is what we're squelching in this example.)

30 minutes of audio was recorded. A carrier was on less than half of the time. FLAC compresses the file at a ratio of 0.174; this results in an average bitrate of 30 kbps, which is pretty good for lossless 16-bit audio. At this rate, a CD can fit almost 50 hours of recording.

24 comments:

  1. Note that significant countries such as the US have ceased monitoring 500 kHz and 2182 kHz distress frequencies. Marine channel 16 is still used frequently, however...

    ReplyDelete
  2. I can't get your program to work but I don't understand what is different from just using sox with the "silence" option like this:

    $ rtl_fm -M -f 137M -s 12k -l 280 | sox -t raw -r 12k -e signed-integer -b 16 -c 1 - output.ogg silence -l 1 0.1 0.1% -1 2.0 0.1%

    If you were running some GUI SDR application and pulseaudio, you could do the same thing like this after you routed the audio streams accordingly:

    $ sox -d -r 11025 -c 1 output.ogg silence -l 1 0.1 0.1% -1 2.0 0.1%

    ReplyDelete
    Replies
    1. By using SoX's silence, you lose the track of time.

      Delete
    2. I don't understand what you mean by that.

      Delete
    3. "silence" cuts the audio out. My "squelch" replaces it with zeroes. Number of samples is preserved.

      Delete
  3. Okay, I get it now. That is why it was not working in the manner I was thinking of. However, you did solve a long standing problem for me. I finally realized the the 'sinc' band pass filter along with 'silence' in sox can be used as an extremely effective way to ignore analog noise on the line-in of many motherboards. I've been testing this for a few days with a line-in source and it works great. Prior to using the 'sinc' filter I would have many problems with finding a good silence threshold percentage as well as the Alsa/Pulseaudio capture levels and source amplitude.

    $ rec -c 1 -r 22050 sinctest.mp3 sinc 200-4500 silence -l 1 0 1% -1 2.0 1%

    This filters out the low frequency analog noise which would normally create problems for the silence detection filter.

    ReplyDelete
  4. I use something like this to record a P25 radio system in my area. It logs each call into a folder with the year, month and date, the filename is the UNIX timestamp, radio ID, and talkgroup ID, saved in MP3. I then wrote a front end in PHP that parses this, and a has searchable database via 'Everything Search'. It's all done in Windows, the back end is written in batch. But your post reminded me that I can use rtl_fm instead of something clunky like SDR# in Windows. In case you're wondering, I have the db here http://sdrscan.com/

    ReplyDelete
  5. I am also messing around with audio streams using rtl_fm on the RPi, I use the following commands:

    rtl_fm -f 145.575M -s 16k -l 0 | lame --scale 2 --lowpass 4.0 -r -s 16.0 -m m - - | ezstream -c /etc/ezstream.xml

    rtl_fm is piped into lame, encoded into mp3 and sent to an icecast server. Squelch is broken, if you turn it up the stream stops, probably due to the data stream from rtl_fm stopping. Does Sox handle this differently? I don't need efficient archival of data, just a low bandwidth stream with squelch to kill the noise. Going to try using sox to see if this makes any difference.

    ReplyDelete
    Replies
    1. Read the comments above. SoX does something completely different, it doesn't output any samples when squelched.

      Delete
    2. I figured that out already. :) What I want to know is how to keep lame and ezstream generating data even when rtl_fm is squelched. That is something different than described on this page but your knowledge seems to be quite extensive so maybe you have a solution?

      Delete
    3. my rtl_fm does output a stream of zeroes when squelched

      Delete
    4. Well, my copy doesn't, piped some data into a file, when squelched it does not generate any data at all. Maybe you are running an older version or squelch or somehow it compiled differently.
      Anyway, have to figure out how to pad stdout with zeroes when idle or get a rtl_fm version that correctly outputs zeroes when squelched.
      Great weblog by the way, lots of interesting articles!

      Delete
  6. Howcome the value of -l 1024 is equivalent to -36dB's ?
    Is there a formula for this?

    This is why I think that values should be measurable, -l parameter should have been supplied as dB's instead...

    ReplyDelete
    Replies
    1. You're right, 1024 / (2^16) --> -30 dB. Pull requests are welcome!

      Delete
    2. I get it now!
      Level_dB=10*log10(1/Level)

      Delete
  7. Hi,

    How do you do for keep the timeline in conversation ? Because, actually in the soundfile, the timeline is lost

    Example, if at:
    00:00:00 sound during 1 minute
    00:01:00 silence during 59 minutes,
    01:00:00 sound during 1 minute

    The sound wave file content approximativelly only 2 minutes sound. Normally we must have 1h01.

    I think, that not depend of your script but the "-l" options of the rtl_fm command

    I use this command line "rtl_fm -M nfm -f $FREQ1 -f $FREQ2 -p $PPM -s 16k -g 49.2 -l $SQUETCH | sox -r 16k -t raw -e signed -b 16 -c 1 -V1 - -r 11025 -t raw - sinc 200-4500 | /usr/local/bin/squelch -l 5 -d 4096 -t 64 | sox -t raw -e signed -c 1 -b 16 -r 11025 - ~/tmp/"$PREFIX"_$DATE.flac"

    Thanks

    ReplyDelete
    Replies
    1. Timeline is kept, no samples are thrown away, just zeroed out. Please read the comments above.

      Delete
    2. Hi,

      Thank for your answer.

      As I said in my previous post, i think the problem it's not it in your script but in the rtl_fm

      You can see the difference between this two commands

      rtl_fm -M nfm -f 99.9999M -p 57 -s 16k -g 49.2 -l 1
      and
      rtl_fm -M nfm -f 99.9999M -p 57 -s 16k -g 49.2 -l 9999

      In the second command, you not receive data due to the rtl_fm squelch hight level. During the squelch selection, no data added to other pipe (sox or soundfile)

      Sorry for my bad english, i'm french. I hope you understand what I am trying to say ? The exemple should help understand :)

      Thanks

      Delete
    3. Ah, could be that I was using another version of the tool.

      Delete
    4. You have a data with this command rtl_fm -M nfm -f 99.9999M -s 16k -g 49.2 -l 9999 ?

      If yes, effectively, we have a different tool version :)

      For information, I use the Archlinux rtl-sdr git version.

      Delete
  8. Hi,
    I found a other alternative for get a timestamped audio, i add a this sox filter at the end of treatment "silence 1 0.50 0.1% 1 2.0 0.1% : newfile : restart". It create a new file after silence. The timestamped audio is in the date creation of filename

    Here the commande line sample

    rtl_fm -M nfm -f $FREQ -p $PPM -s 16k -g $GAIN -l $SQUETCH | sox -r 16k -t raw -e signed -b 16 -c 1 -V1 - $FILENAME sinc 400-4500 silence 1 0.50 0.1% 1 2.0 0.1% : newfile : restart

    ReplyDelete
    Replies
    1. Forgive me if this sounds ignorant, I have only played with Linux a few hours per month for the past few years, I am trying to wrap my head around this. It seems both solutions have their own purpose and can be used and abused in many ways. b_adele please correct me if I am wrong but your solution seems to focus on individual files (something I personally could use). When a user keys a mic on a radio and speaks sox would record the audio and when complete would save a single audio file with a timestamp and immediately restart to await the next radio transmission. Is this correct?

      And Oona your solution looks excellent for streaming as well as long unattended recordings. I think I understand that timecode can be contained but this method truncates all transmissions (excluding silence breaks) into a single audio file. Would it be possible to split this into multiple single files and rename them based on that timecode? How would the real time of day be calculated, if possible? Is that how these two differ?

      Delete
  9. Hi @Joshua,

    I will try explain the problematic. With the rtl_fm tools you can record the sound for a selected frequencies. But if you use the rtl_fm squelch feature (- l) when the squelch is done you have no data out from the rtl_fm. For example. if you record during 24H and the user talk only twice during 2 minutes. The file recording time is only 2 minutes and not 24H. With this, how to know when the user has spoken ?

    For this reason, i'm working in the tools for recording and logging the date of record.

    The tools is available here https://github.com/badele/SDRHunter/tree/master/SDRHunter/tools

    Actually, this tools record the pulseaudio Sox Streams, one sox stream must running in the computer session. Later, this tools will work with GQRX or other tools

    Note: these tools is In the alpha state

    ReplyDelete

The comments section is pre-moderated.

You might want to check out the FAQ first.