Method
The autokey cipher, as used by members of the American Cryptogram Association, starts with a relatively-short keyword, the primer, and appends the message to it. For example, if the keyword is QUEENLY and the message is attack at dawn, then the key would be QUEENLYATTACKATDAWN.[2]
Plaintext: attackatdawn
Key: QUEENLYATTACKATDAWN
Ciphertext: QNXEPVYTWTWP
The ciphertext message would thus be "QNXEPVYTWTWP".
To decrypt the message, the recipient would start by writing down the agreed-upon keyword.
QNXEPVYTWTWP
QUEENLY
The first letter of the key, Q, would then be taken, and that row would be found in a tabula recta. That column for the first letter of the ciphertext would be looked across, also Q in this case, and the letter to the top would be retrieved, A. Now, that letter would be added to the end of the key:
QNXEPVYTWTWP
QUEENLYA
a
Then, since the next letter in the key is U and the next letter in the ciphertext is N, the U row is looked across to find the N to retrieve T:
QNXEPVYTWTWP
QUEENLYAT
at
That continues until the entire key is reconstructed, when the primer can be removed from the start.
QNXEPVYTWTWP
QUEENLYATTACKATDAWN
attackatdawn
With Vigenère's autokey cipher, a single mistake in encryption renders the rest of the message unintelligible.[3]
Cryptanalysis
Autokey ciphers are somewhat more secure than polyalphabetic ciphers that use fixed keys since the key does not repeat within a single message. Therefore, methods like the Kasiski examination or index of coincidence analysis will not work on the ciphertext, unlike for similar ciphers that use a single repeated key.[3]
A crucial weakness of the system, however, is that the plaintext is part of the key. That means that the key will likely contain common words at various points. The key can be attacked by using a dictionary of common words, bigrams, trigrams etc. and by attempting the decryption of the message by moving that word through the key until potentially-readable text appears.
Consider an example message meet at the fountain encrypted with the primer keyword KILT:[4] To start, the autokey would be constructed by placing the primer at the front of the message:
plaintext: meetatthefountain
primer: KILT
autokey: KILTMEETATTHEFOUN
The message is then encrypted by using the key and the substitution alphabets, here a tabula recta:
plaintext: meetatthefountain
key: KILTMEETATTHEFOUN
ciphertext: WMPMMXXAEYHBRYOCA
The attacker receives only the ciphertext and can attack the text by selecting a word that is likely to appear in the plaintext. In this example, the attacker selects the word the as a potential part of the original message and then attempts to decode it by placing THE at every possible location in the key:
cipher: WMP MMX XAE YHB RYO CA
key: THE THE THE THE THE ..
plain: dfl tft eta fax yrk ..
cipher: W MPM MXX AEY HBR YOC A
key: . THE THE THE THE THE .
plain: . tii tqt hxu oun fhy .
cipher: WM PMM XXA EYH BRY OCA
key: .. THE THE THE THE THE
plain: .. wfi eqw lrd iku vvw
In each case, the resulting plaintext appears almost random because the key is not aligned for most of the ciphertext. However, examining the results can suggest locations of the key being properly aligned. In those cases, the resulting decrypted text is potentially part of a word. In this example, it is highly unlikely that dfl is the start of the original plaintext and so it is highly unlikely either that the first three letters of the key are THE. Examining the results, a number of fragments that are possibly words can be seen and others can be eliminated. Then, the plaintext fragments can be sorted in their order of likelihood:
unlikely ←—————————→ promising
eqw dfl tqt ... ... eta oun fax
A correct plaintext fragment is also going to appear in the key, shifted right by the length of the keyword. Similarly, the guessed key fragment (THE) also appears in the plaintext shifted left. Thus, by guessing keyword lengths (probably between 3 and 12), more plaintext and key can be revealed.
Trying that with oun, possibly after wasting some time with the others, results in the following:
shift by 4:
ciphertext: WMPMMXXAEYHBRYOCA
key: K.LTM.ETA.THE.OUN
plaintext: m.eta.the.oun.ain
shift by 5:
ciphertext: WMPMMXXAEYHBRYOCA
key: SWT..EQW..THE..OU
plaintext: eqw..the..oun..og
shift by 6:
ciphertext: WMPMMXXAEYHBRYOCA
key: D...TQT...THE...O
plaintext: t...the...oun...m
A shift of 4 can be seen to look good (both of the others have unlikely Qs either in the plaintext or in the keyword).
A lot can be worked with now. The keyword is probably 4 characters long (K.LT), and some of the message is visible:
m.eta.the.oun.ain
Because the plaintext guesses have an effect on the key 4 characters to the left, feedback on correct and incorrect guesses is given. The gaps can quickly be filled in, giving both the plaintext and the keyword:
meetatthefountain
KILT
The ease of cryptanalysis is caused by the feedback from the relationship between plaintext and key. A three-character guess reveals six more characters (three on each side), which then reveal further characters, creating a cascade effect. That allows incorrect guesses to be ruled out quickly.