Frequency analysis

Frequency analysis is a general procedure that can be used during the cracking of several different kinds of ciphers. In frequency analysis, we typically go through the cipher text and try to find some similarities with the language in which the message is written.

The frequency of letters in the ciphertext

One of the basic tools of frequency analysis is the frequency of characters in a given language. If we analyse enough Czech text, we can calculate various statistical information, including how many times which letters occurred in these texts.

Since I bought about five thousand Czech-language books the other day, I did such an analysis on them. There were a total of 1,404,443,151 letters in those five thousand books. The following table shows the distribution of letters (after removing diacritics):

The following table then shows the relative frequency of letters:

Thus, the table says that approximately 9.58% of the letters in all the books were the letter "a". From this we can deduce that approximately 9.58% of the letters in a normal Czech text would be "a".

Each such statistic will show slightly different values, but that's okay - it's not so much about exact values, you can't find those out anyway, just approximate values.

The deviation of the frequency of letters from the frequency of letters in the language

If we know the expected relative frequencies of each letter in a given language, we can calculate the deviation of some text from the expected values. As an example, consider two texts, "fhes dudy fqhup ijhecuc fhejepu eriqxkzu ahkpdysu" and "proc neni parez tree because it contains crucifixes". At first glance we see that the first text is some gibberish, while the second sentence is the Czech text. Let's calculate the deviation - it should work out that the deviation of the first text will be substantially higher than that of the second.

First, let's calculate the occurrences of letters in each text:

Now we calculate the deviations. We calculate these by subtracting the relative frequency of a letter in the language from the relative frequency in the first text and squaring. That is, for the letter "a" and the first text, we would get the deviation as

$$ (0.0232558 - 0.0958927)^2=0.0052761 $$

and similarly for all the other letters. If you hover the mouse over the cell with the result, the expression that was used to arrive at the result is also displayed.

Finally, we summed all the deviations. Correctly, we should still divide this value by the number of letters and then subtract to really get the deviation. But these operations do not change the fact that the second text has a significantly lower deviation, so it is much more likely to be a Czech text.

The most common words of the language

Sometimes a list of the most common words of a language can be useful.