Jun 16, 2014

Headerless train announcements

The Finnish state railway company just changed their automatic announcement voice, discarding old recordings from trains. It's a good time for some data dumpster diving for the old ones, don't you think?

A 67-megabyte ISO 9660 image is produced that once belonged to an older-type onboard announcement device. It contains a file system of 58 directories with five-digit names, and one called "yleis" (Finnish for "general").

Each directory contains files with three-digit file names. For each number, there's 001.inf, 001.txt and 001.snd. The .inf and .txt files seem to contain parts of announcements as ISO 8859 encoded strings, such as "InterCity train" and "to Helsinki". The .snd files obviously contain the corresponding audio announcements. There's a total of 1950 sound files.

Directory structure

The file system seems to be structurally pointless; there's nothing apparent that differentiates all files in /00104 from files in /00105. Announcements in different languages are numerically separated, though (/001xx = Finnish, /002xx = Swedish, /003xx = English). Track numbers and time readouts are stored sequentially, but there are out-of-place announcements and test files in between. The logic connecting numbers to their meanings is probably programmed into the device for every train route.

Everything can be spliced together from almost single words. But many common announcements are also recorded as whole sentences, probably to make them sound more natural.

Audio format

The audio files are headerless; there is no explicit information about the format, sample rate or sample size anywhere.

The byte histogram (left) and Poincaré plot (right) from baudline suggest a 4-bit sample size; this, along with the fact that all files start with 0x80, is indicative of an adaptive differential PCM encoding scheme.

Unfortunately there are as many variations to ADPCM as there are manufacturers of encoder chips. None of the decoders known by SoX produce clean results. But with the right settings for the OKI-ADPCM decoder we can already hear some garbled speech under heavy Brownian noise.

For unknown reasons, the output signal from SoX is spectrum-inverted. Luckily it's trivial to fix (see my previous post on frequency inversion). The pitch sounds roughly natural when a 19,000 Hz sampling rate is assumed. A test tone found in one file comes out as a 1000 Hz sine when the sampling rate is further refined to 18,930 Hz.

This is what we get after frequenqy inversion, spectral equalization, and low-pass filtering:

There's still a high noise floor due to the mismatch between OKI-ADPCM and the unknown algorithm used by the announcement device, but it's starting to sound alright!

Peculiarities

There seems to be an announcement for every thinkable situation, such as:

  • "Ladies and Gentlemen, as due to heavy snowfall, we are running slightly late. Please accept our apologies."
  • "Ladies and Gentlemen, an animal has been run over by the train. We have to wait a while before continuing the journey."
  • "Ladies and Gentlemen, the arrival track of the train having been changed, the platform is on your left hand side."
  • "Ladies and Gentlemen, we regret to inform you that today the restaurant-car is exceptionally closed."

Also, there is an English recording of most announcements, even though only Finnish and Swedish are usually heard on commuter trains.

One file contains a long instrumental country song.

In an eerily out-of-place sound file, a small child reads out a list of numbers.

Final words

This is something I've wanted to do with this almost melodically intonated announcement about ticket selling compartments.

16 comments:

  1. Cool Thanks for sharing! Is it possible to decode all the files to mp3/wav? It would be cool to hack a simple site where you can interactively construct any phrase from the existing words.

    ReplyDelete
    Replies
    1. Yes, I've decoded them in a similar manner.

      Delete
  2. Wow, this is great. Please please post more audio files. Would love to hear the ones in Finnish, and some of the odder ones in English, and the country song. These would all make amazing samples, and it's a cool archive.

    ReplyDelete
  3. Boris, i think that's exactly what Oona is trying to accomplish, but the encoding format is still not quite clear. I too would really like to have the whole sound library as usable and clear clips, but we'll have to wait. VR, the railway company, would do a culturally significant deed if they did release the sounds in an open and compatible manner. They are, after all, built on state money (mine included) so the could very well release some left overs to the people.

    ReplyDelete
  4. If you post a few of the original files, maybe somebody else can figure out the format.

    ReplyDelete
  5. Also the old Finnish metro announcements are available!
    http://sooda.dy.fi/2013/7/22/metrolaituripuhekoneen-datat/
    https://github.com/sooda/metro_station_display/tree/master/gspbox

    ReplyDelete
  6. Seems i have different bunch of announcements (for DKK), 5000+ sound files. Folder says "LÄHILIIKENNE". Files are dated to around 1999-2001, but format is same.

    ReplyDelete
  7. Greetings from the original sound driver coder. The onboard unit is (or was?) a Transputer based proprietary hardware designed and manufactured by Rauma Oceanics Oy (yes, the same company that designed and manufactured the submarine shown in Titanic). The sound chip was Crystal Semiconductor CS4231. The announcements were produced using Crystal's development ISA card mounted on a PC running NeXTstep 3.1. I wrote both the Transputer driver and the NeXTstep driver for the chip. As you guessed right, there's a separate routing info which connects GPS points to the announcements + train schedules. This way the system was able to drop out connecting train announcements, if the train was late for a GPS point. The routing info was stored on a PCMCIA memory card. This allowed using the same CD for all trains (330 at the time... ~1993) and schedule changes etc. were easy to update using the PCMCIA. Ole hyvä :-)

    ReplyDelete
    Replies
    1. Thank you! Comments like this are the reason I keep writing this blog :)

      Delete
  8. Kimmo, DKK=Digitaalinen KuulutusKone (=Digital Announcement Machine). KTL=KuulutustenTekoLaite (Announcement Producing Machine). I forgot to mention that the chip had a bug in recording ADPCM. You may need to add 127 (or 128, cannot remember anymore) to the values in order to produce clear sound. This may have been fixed in PC driver automatically - I don't know.

    ReplyDelete
  9. Since you now know which chip is used to generate the audio, this is probably out-of-scope. But I would anyhow have guessed the Stichting Mathematisch Centrum Intel/IMA adpcm codec. And reading the datasheet actually gives at hand that it probably is... Good luck.

    ReplyDelete
  10. Please, we all are waiting for the ISO... The music btw is more like jazz, not country ;)

    ReplyDelete
  11. We too would like to have the raw data! :D 8)

    ReplyDelete
  12. Can the spectrum inversion, be just caused by reading the MSB/LSB incorrectly(reversed)?

    ReplyDelete
    Replies
    1. It's a digital baseband signal, so there are no sidebands anywhere.

      Delete

To prevent spam, comments will only appear after moderation.