The Life and Death of Passwords: The History of Encryption with Simon Singh
Our documentary, “The Life and Death of Passwords,” explores with industry experts the history of passwords, why passwords have become less effective over time, and how trust is established in a passwordless future. With this interview series, we take a deeper dive into their insights and share bonus footage.
Today: Simon Sing – a science author and journalist who explores mathematics, cryptography, and cosmology – digs into the pre-computing history of passwords, the evolution of cryptography, and how they enabled today’s information revolution.
Why we encrypt
Chrysta: As a little bit of context for the audience, can you define a code and a cipher and what’s different between those two ideas or those two methods?
Simon: When we talk about secret communication, we use words like “encryption” and “encipher” and “encode,” and all of these things are kind of used interchangeably.
There are some technical definitions. So for example, a code is where one word is always replaced with a certain symbol, for example, and that’s always the case. One word, one symbol. And encipherment tends to mean that the word is jumbled up and it can be jumbled up in different ways on different occasions.
What human need drove the creation of codes and ciphers and secret communication?
As soon as people start writing things down, as soon as we have the invention of writing, we have the need to make sure that what we’re writing down can be secret, can be protected. So we might be writing down military plans, we might be writing down recipes for a pottery glaze, we might be writing down love letters. Whatever it is, we might not want prying eyes to know what we’re documenting.
In terms of the situations where these techniques would be used, was it always sort of a martial life or death situation, or was encryption used in more mundane scenarios as well?
When we think of encryption, we often think of military generals. Julius Caesar wrote a book, a little book on encryption and was quite familiar with how to use it. And we think of the second world war, we think of modern warfare and the information age and the information war. But also school kids like to keep their diaries encoded sometimes for very good reasons. So there are more mundane uses of encryption as well.
Early encryption: The Caesar and Vigènere Ciphers
Chrysta: One of the things that’s really interesting in your book is that you show this leapfrogging advancement, where code makers innovated new techniques, code breakers would develop their own techniques to try and reverse that method, and that kind of back and forth built more complex iterations on these basic techniques. What would you say some of the biggest or the earliest innovations in the development of codes and ciphers were?
Simon: The one we might all have used in school is to replace every letter with a symbol, okay? So “A” might be a diamond. “B” might be a cross, “C” might be a circle, okay? Every letter with a different symbol. And people used this for centuries. It was a very strong form of encryption.
And then people realized something called frequency analysis. Now, frequency analysis was first documented in Baghdad by a mathematician philosopher called Al-Kindi. And he wrote the first-ever document on code breaking. And what Al-Kindi said was that if a letter is very common — so in English “E” is very common, and let’s say we replace E with a triangle — well then, triangles are going to be very common. So the code breaker latches on to the frequency of triangles says, “Hey, triangles must be ‘E’s,” and then the next most common symbol will be “T” the next most common symbol will be “A” and so on. So by applying this frequency analysis, you can break what we call the simple substitution cipher.
So then you have this battle that the code maker wants to get ahead of the code breaker. So the code maker comes up with something a bit cleverer. So for example, what we can now do is if we have a letter like “E,” which accounts for 13% of all letters in English, we replace “E” with 13 different symbols, okay? So now all of these symbols that represent E appear only 1% of the time in the cipher text, as we call it. A rarer letter like “Q” might only be replaced by one symbol, a diamond, and that might appear only 1% of the time in the text. And so our distribution of frequencies is flat and therefore the code breaker has a much tougher problem.
Now, there are ways around that, and eventually code breakers figured out how to crack this more complicated cipher. And so the code makers had to do it all over again, they had to come up with an even better encryption system. So you have this constant battle between code makers cracking codes and coming up with even better code.
"So you have this constant battle between code makers, cracking codes and coming up with even better code." - Simon Singh
One of the earliest algorithms, or one that a lot of our viewers may be familiar with in some form is a Caesar cipher. Can you describe for us briefly what a Caesar cipher is and how it works?
A Caesar cipher is a type of simple substitution cipher, and we don’t just replace the letter A with any old symbol or any old letter, we replace A by shifting it down the alphabet. Now a classic Caesar cipher shifts by three places so A becomes B, C, D, A becomes D. B becomes C, D, E, B becomes E. And that’s all you do; you just shift every letter down three places.
Overall, this is not a great way to send secret messages because if I’m a code breaker, I’ve only got 25, 26 different shifts that I need to check.
And continuing from that, one of the next major leaps that we saw in an evolution of that approach was the Vigenère. Can you briefly describe how that cipher differed or what the advantage was that it provided?
Overall, this is not a great way to send secret messages. Because, if I’m a codebreaker, I’ve only got 25, 26 different shifts that I need to check. But you can make the Caesar cipher absolutely, brutally strong. You can transform it into something called the Vigenère cipher, invented by a French chap called Vigenère.
The way the Vigenère cipher works is you pick a code word. I’m going to pick the code word B-A-D. So you have your message, “Hello Fred” and above it you just write the word B-A-D over and over again. Now you can start encrypting.
To encrypt the H, well there’s a B above it and B is the second letter of the alphabet. So we shift H two places, the H becomes J. E has an A above it, so we only move the E by one place: E becomes F. The L? There’s a D above that, and that’s the fourth letter of the alphabet. L becomes P.
Now this is where it gets interesting. We’ve got another L. But this time, there’s not a D above the L, there’s a B and the B tells us to shift only two places. So the L becomes an N.
And that’s why the Vigenère cipher is so secure. You have the same letter being encrypted in different ways. It became known as le chiffrage indéchiffrable, the cipher that’s indecipherable. For decades, perhaps even centuries, it was unbroken.
Can you tell us a bit about who cracked the cipher and why?
Vigenère invents the Vigenère cipher in the 16th century. In fact, a couple of people had got there a little bit before him, but anyway, it was invented in the 16th century. In the Victorian era, the Vigenère cipher is eventually broken, and it was broken by Friedrich Kasiski, or at least that’s what we thought. It turns out that it was actually broken a decade earlier by a chap called Charles Babbage who’s famous today for being the kind of pioneer of mechanical computing, as well as many other things.
Mechanical encryption: Cracking the Enigma Cipher
Let’s talk about the pressures or the factors that drove the development of mechanical or proto computing approaches like Enigma. How did Enigma work and what made the Enigma codes harder to crack?
In the 20th century, we start having radio being used, particularly in warfare, particularly in the battlefield. The problem with radio is that anybody can tune into your radio frequency and hear what you are saying. So you then need to have a form of encryption that is rapid and secure, and that’s why you have the development of mechanical encryption devices, most famously of all the Enigma cipher machine.
Now, the reason why the Enigma cipher was considered so unbreakable is because it had so many what we call keys. The keys are the number of ways you can set up an encryption system.
So if we go back to the Caesar cipher, Caesar cipher is all about shifting and there are only really 25 ways you can shift the alphabet to get a different type of Caesar cipher.
With the Enigma machine, there are so many different ways to set up that machine. You can pick different rotors, different wheels to go into the machine. You can put them in different places, in different orders. You can orientate them in different ways. Each wheel kicks over the neighboring wheel when it does a full revolution, you can change that kick over point. There’s a plug board at the beginning, which allows you to swap letters around.
So if I type an A, it’s as if I typed in D. And when you have all of these possibilities, there are so many possible keys. I can’t remember the number of keys, frankly, but there are so many possible keys that it’s impractical for a code breaker to go away and check every possible permutation.
Ultimately, how did the Allies ultimately actually break Enigma?
The first people to break the Enigma — and this is a story that’s becoming better known now — were the Poles. Poland had the first people to realize that in the 20th century, you didn’t need linguists anymore, you needed mathematicians. You need people with a scientific mind to break a very scientific piece of equipment.
The Poles knew they were going to be invaded sooner or later. So they said, “We must try to do even the impossible. We must try and break the Enigma.” And they made some important breakthroughs and they managed to smuggle out those breakthroughs to London before the war started so that gave the British a huge head start.
People may have heard of Bletchley Park; this was the British code break center north of London. It was secluded, had good communication links. And during the course of the war, more and more code breakers gathered there at Bletchley Park to think of more and more ways to break the Enigma.
They broke the Enigma in several different ways, but let me just explain one of them. One of the quirks of the Enigma machine is that if you type in an A, it never encrypts A as A, it never encrypts a letter as itself. Now that seems completely fine because I don’t want to encrypt A as A, I’ve got 25 other letters that I can encrypt A into, okay? But nevertheless, it was an old quirk of the Enigma machine. No letter could be encrypted as itself.
One day, one of the codebreakers at Bletchley, a woman called Mavis Batey, was looking through some encrypted messages that had been intercepted. And she looked at this message, it was full of just gibberish. It seemed to be a random list of letters. But it wasn’t random because in the whole message, there wasn’t a single W. I don’t know how many characters there were, there may be 200, 300 characters. There should have been a dozen or so Ws, but there were no Ws at all. And Mavis thought to herself, “Well, look, the only way you can have an encrypted message which is devoid of Ws is if your input is entirely made of Ws. If you type W 300 times, your output will be everything except W.” And she’d crack the message, the message was just W 300 times.
Once she knew the input — all these Ws — and once she knew the output — the encrypted message she had in her hand — she could then work out the settings of the Enigma machine for that day. She and her colleagues could decipher all the other messages sent in that communications network that day. They gathered information about an impending attack on the British fleet in the Mediterranean, the British fleet was ready, and it led to the first allied victory in the Mediterranean.
You might wonder, why would somebody send a message of 300 Ws? Say I’ve got an unbreakable code and I’m sending my messages and you’ve got no hope of reading my messages, because I’ve got an unbreakable code. What you can do is what’s called traffic analysis, that means you can count the number of messages I’m sending each day. And if I send one on Monday, one on Tuesday, one on Wednesday and five on Thursday, then you know that maybe on Friday, something big is going to happen. So what I do to combat that is I send five messages every single day, and then my traffic is flat, and then you can’t gain any information. So the 300 Ws were probably sent as somebody’s effort to fulfill a quota for that day.
Modern encryption: Encrypting and decrypting without keys
The conversation so far has really brought us up to about World War II and the use of the Enigma machine. We now have most of a century of history between then and the present day. What would you say are sort of those landmark moments or evolution of cryptography between the end of the war, the postwar era and the present day?
Yeah, so one of the big changes in our lifetimes is that encryption used to be about governments and the military, and now encryption is about our everyday lives. It’s about encrypting our medical records, it’s about encrypting video streams, it’s about encryption of our bank details. And every day the devices we’re using are using encryption.
So it’s very, very much part of our day to day lives. It’s not just back in kind of Bletchley Park specialist government code breaking rooms or code making rooms. And I think the biggest change in our lifetimes again...or I would say one of the biggest changes in recent decades, is that if you imagine with Enigma, and this is Enigma, or in fact, any kind of code, I scramble up a message and I send it to you.
Now, you can only unscramble that message if I sent you the scrambling recipe, because if you know the scrambling recipe, you are going to know the unscrambling recipe. So before I sent you that message, I would’ve had to have given you that scrambling and unscrambling recipe, okay?
So with Enigma, for example, every month there was a sheet of paper and the sheet of paper had on every single day, had a list of instructions of how to scramble up using the Enigma for that day. Remember the Enigma has lots of settings and it would tell you the settings for that day. So the person sending the message would have actually the paper and the person receiving the message would have that piece of paper and dispatch riders on motorbikes would be riding over the Sahara delivering these bits of paper. New boats will be coming back into port to collect these bits of paper. And so nobody can have secure encryption unless they have access to this complicated and secure distribution network, okay?
Now that doesn’t work today. If I want to buy something online, I can’t get a dispatch rider to drive over to the online company and give them my scrambling recipe so that I can encrypt my credit card details, that doesn’t work. So the biggest revolution in the last few decades has been a form of encryption - in fact a couple forms of encryption - that mean that I can send you a secret message and I can scramble it up and even though we’ve never met, even though we’ve never exchanged any other information, you can still unscramble my message.
It sounds impossible. For centuries, for millennia, people thought it would be impossible, but it turns out you can do it. And that’s what’s enabled the revolution of online e-commerce and many of the other things that we live with today, there are only possible because of something called public key cryptography. Bottom line is I’ve never met you before in my life, I can send you a secret message and you can still read it. Okay, that’s the bottom line.
"Bottom line is, I've never met you before in my life, I can still send you a secret message and you can still read it." - Simon Singh
How does it work? Well, public key encryption involves some clever mathematics. It’s not the most complicated mathematics in the world and if you wanted to go in and Google it, you could probably try and figure it out and see how it works.
But I’ll give you an analogy, which is probably easier to explain in this kind of situation. I have a precious object that I want to send to you and I put the object in a box and I close the lid and I padlock the box and I send the box to you. Now, you can’t open it because I haven’t given you the key. Yeah, that’s our problem. That’s our key distribution problem.
But what you can do is you can put your padlock on the box and send it back to me. Now that just seems to make things worse. I’ve now got a box that’s doubly padlocked, but what I can now do is I can take my padlock off and send the box with your padlock on back to you. And now you can undo your padlock because it’s your padlock and you’ve had the key all the time, you can open the box and you can access the precious object, okay?
So at no time did we distribute a key, at no time was the box insecure or unsecured. And at the end of the day, you could access the precious object. So it is possible to send something in secret to somebody you’ve never met before, and the mathematical implementation of that kind of concept is what’s enabled the information revolution that we have today.
Want to learn more about the history of passwords and encryption?
Next in our extended interview series: Wolfgang Goerlich, advisory CISO at Cisco Security, reflects on the history of passwords, the limitations of human memory and what we can learn from the eternal nature of security.