This is a recipe to train word n-gram language models using the newswire text provided for the Continuous Speech Recognition (CSR) project (222M words from WSJ, SJM and AP). It prepares the dictionaries needed to use the LMs with the HTK and Sphinx speech recognizers. You can also download the trained ARPA format language models.

Requirements:

• Linux computer with Perl installed
• LDC's CSR-III text corpus
• SRI Language Modeling Toolkit.

By default, the scripts use: interpolated, modified Kneser-Ney smoothing, bigram cutoff 1, trigram cutoff 3, with an unknown word. I used the top 5K, 20K, and 64K words occurring in the training text as vocabularies. Both verbalized punctuation (VP) and non-verbalized punctuation (NVP) LMs are built.

Here are some OOV% and perplexity results measured on three different held-out evaluation test sets:

• csr - held-out portion of CSR LM-1 training data (~6M words)
• setasd - text from the CSR LM-1 setaside dir (~15M words)
• giga - held-out portion of Gigaword training data from my Gigaword training recipe (~25M words)

Note that the Gigaword test sets are from different newswire sources and cover a different historical period than CSR.