Modern cryptography is built upon four main ‘primitive’ capabilities:
-
Random number generation
-
One-way hashing
-
Symmetric encryption/decryption
-
Asymmetric encryption/decryption (public-key cryptography)
Familiarity with these four primitives is enough to understand most security protocols and techniques that use cryptography.
Warning
|
Don’t make the mistake of thinking you can write your own cryptographic code. Use libraries that have been written by cryptography professionals. |
Warning
|
This is very important, so I’m going to repeat it. Don’t make the mistake of thinking you can write your own cryptographic code! Use thoroughly tested libraries developed by professional cryptographers. Cryptographers have meticulously coded the libraries for security against many attacks that are not obvious (for example, timing attacks). |
Warning
|
One final time. Don’t write your own cryptography! It is important. |
Random number generation
Computers are deterministic: they predicably follow instructions. Two computers running the same code at the same time with the same inputs will produce the same output.
In fact, before the introduction of special random number generation CPU instructions, such as RDRAND
and RDSEED
[1], it was theoretically impossible for a computer program to generate random numbers.
[2]
Node.js and JavaScript have a Math.random()
and other programming languages have similar functions. However, these use pseudorandom number generators.
An old example of pseudorandom number generators is the rand48()
function in the C programming language standard library. The default implementation uses an update rule, such as the following:
r' = (25214903917 * r + 11) % 281474976710656
Starting with r = 1, the next ten ‘random’ numbers are as follows:
Iteration |
r |
r / 281474976710656 |
---|---|---|
1 |
25214903928 |
0.0000895813340946 |
2 |
206026503483683 |
0.7319531771219197 |
3 |
245470556921330 |
0.872086605317186 |
4 |
105707381795861 |
0.3755480612563424 |
5 |
223576932655868 |
0.7943048269107607 |
6 |
102497929776471 |
0.3641457971656017 |
7 |
87262199322646 |
0.3100176091757803 |
8 |
266094224901481 |
0.9453565926573084 |
9 |
44061996164032 |
0.1565396564872117 |
10 |
147838658590923 |
0.5252284246314893 |
To an untrained eye, these numbers appear to be ‘random’: the third column looks like a list of ‘random’ numbers between 0 and 1. They’re good enough for games and for generating random visualizations.
Node.js (and Chrome) use a more complex function, XorShift128+
, defined in the Node.js C++ source code as follows:
static inline void XorShift128(uint64_t* state0, uint64_t* state1) {
uint64_t s1 = *state0;
uint64_t s0 = *state1;
*state0 = s0;
s1 ^= s1 << 23;
s1 ^= s1 >> 17;
s1 ^= s0;
s1 ^= s0 >> 26;
*state1 = s1;
}
It isn’t necessary to understand what this code does. My point is simply that the function is far too simple to use for security purposes. Future numbers can be easily guessed based on older numbers. So, if a website uses Math.random()
to generate login codes for SMS messages (i.e., in a multi-factor authentication system), an adversary will be able to predict the login codes sent to users, without needing access to their phones.
While computers cannot generate true random numbers, cryptographically secure pseudorandom numbers (CSPRNG) are considered a safe alternative. CSPRNGs start with a small amount of initial random data (e.g., mouse clicks, keyboard data, network packets). They then use extremely complex update rules to generate an endless stream of numbers that appear random. The rules are chosen so that there is no known technique to predict future values based on their outputs.
Node.js and modern web browsers expose CSPRNGs for situations where secure random numbers are required.
The following code generates 16 cryptographically secure random bytes in Node.js:
const crypto = require('crypto');
let bytes = crypto.randomBytes(16);
console.log(bytes);
A crypto
object has also been available in modern web browsers since 2014. The following code generates 16 random numbers in a browser:
[3]
let bytes = window.crypto.getRandomValues(new Uint8Array(16));
console.log(bytes);
Warning
|
Math.random() is NOT secure. Attackers can predict its output and it is not truly random.
|
Tip
|
If random numbers are essential to your application’s security, use require('crypto') in Node.js.
|
One-way hashing
A one-way hash function transforms any amount of data into a single fixed-length summary. For example, the following table shows the result of using the SHA-256 hash function on different data:
input |
sha256(input) |
---|---|
|
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 |
|
ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb |
|
3e23e8160039594a33894f6564e1b1348bbd7a0088d42c4acb73eeaed59c009d |
|
fb8e20fc2e4c3f248c60c39bd652f3c1347298bb977b8b4d5903b85055620603 |
|
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 |
|
09ca7e4eaa6e8ae9c7d261167129184883644d07dfba7cbfbc4c8a2e08360d5b |
|
160b4e433e384e05e537dc59b467f7cb2403f0214db15c5db58862a3f1156d2e |
|
6d1cf22d7cc09b085dfc25ee1a1f3ae0265804c607bc2074ad253bcc82fd81ee |
The full text of the Project Gutenberg edition of the book Gulliver’s Travels (1726) by Jonathan Swift |
d00c4f379e789c146ce22749b8bcba6559243e1eb9a457f7d917a26b6ff7bf9b |
The table above illustrates some desirable properties of a one-way hash:
-
The output is always the same length, irrespective of the input
-
The same input will always produce the same output
-
Slight differences in the input result in entirely different outputs
A good one-way hash algorithm has additional properties:
-
It is fast to calculate the hash output, given an input
-
It is extremely difficult to find an input if you only know the output
-
It is improbable and difficult to find two inputs that produce the same output (this is called a collision)
A one-way hash gets its name from the fact that it works in only one direction: it can go from input to output, but reversing from output to a matching input is nearly impossible.
Most modern one-way hash functions work internally by repeatedly applying simple functions that substitute and permute data. Each operation introduces a little bit of confusion and uncertainty. The result of repeated application of the simple operations is so complex that the only way to go from output to input is to randomly guess-and-check possible inputs.
Two common applications of one-way hash functions are as follows:
-
A one-way hash provides evidence of a file’s integrity. Any change to the input file will result in a completely different hash.
-
A one-way hash demonstrates that you know or have an input value, without revealing the actual input value.
Symmetric encryption/decryption
Symmetric encryption/decryption is a method for protecting data using a single key or password. The name symmetric comes from the idea that encryption and decryption use the same key. This approach may also be known as secret-key cryptography because the key is a password to keep secret.
The following table demonstrates the result of using the AES-256-CBC symmetric encryption algorithm on various inputs:
input |
key |
output = encrypt(input, key) |
decrypt(output, key) |
---|---|---|---|
hello |
aip |
cb7d8qkuZDvY09bJUaY4Q |
hello |
hello |
asdf |
RzmNro/N4SrUZ9r1CCMNvw |
hello |
hello, world |
aip |
MTLh2I7fY15v6Ly66Tex+Q |
hello, world |
hello, world |
asdf |
g8iyyq1UopvWER0uZs0NXg |
hello, world |
Note that changing either the input or the password results in a completely different output. Without the password, the output is meaningless. The password will decrypt the output back into the original text.
If the password is known, then symmetric key encryption is fast during both encryption and decryption. However, if the password is unknown, then it is extremely difficult (i.e., impossible) to find the input from the output. [4]
Asymmetric encryption/decryption
Asymmetric encryption/decryption was invented in the 1970s to protect data using a pair of matching keys. This approach is also called public-key cryptography because one key is usually made public and the other is kept secret. In an asymmetric cryptosystem, the private key decrypts messages encrypted by the public key (and vice-versa).
RSA and elliptic curve cryptography are the two main approaches to public-key cryptography. Both depend on mathematical operations that are easy to compute in one direction, but harder to reverse.
RSA depends on factorization being difficult. For example, using pen-and-paper, most people would be able to multiply 89681 and 96079 to get 8616460799 within a minute or two. However, it might take years to reverse the operation to find the factors of 8616460799, without a computer. [5] Computers are faster than people, so RSA depends on massive numbers with over 600 digits (2048 bits).
Elliptic curve cryptography also uses an operation that is difficult to reverse. Elliptic curves are mathematical structures with a group operator. The mathematical symbol ‘+’ is often used to represent this operator, even though it is not ordinary addition. For example, in an elliptic curve, it is easy to combine a value with itself eight times: P = G + G + G + G + G + G + G + G. However, it is difficult to calculate that the group operator was applied eight times when given only G and P. In practice, systems such as Curve25519 apply the group operator many more times. For example, rather than using the one-digit number ‘eight’, elliptic curve systems use 77 digit numbers (256 bit) or greater.
The first step in using an asymmetric cryptosystem, is to generate a public and private key. For example, on my computer, I used the openssl
command [6] to randomly and securely generate the following RSA private key and its matching public key.
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEArehpGoTexi70UPL7w2AsOTWj+HJrewxVkM3DMKU4CDzVpV4d
CHRKciIA6rOYgrKnT/yH2z7UasHf9KBZENVNdFqMCBTCNM4Pt4NDKZT4DNSw+2qS
qxEtYb7gtOZfTJxv/MPhrZ1Fi672N8AfECqfZabf/HoVim4wtO1l+XMHdD/dpmQ7
wQjF/BpIGLYQbI+0sgZWydOArwP7tDgeZJG7nz+pgtgDhfkovLN9YYrgSr7IVydM
71waCuC71ASQ6tTeUmlznP7mMJChND7trdNPCSXuWt62JAPTSWqyjU3FW8pBNtOO
hYaUe8RK8j1DuJxz94m6MEobnyUGiH5SCCnTvQIDAQABAoIBACR+pjfLdFiQl/K4
2v6IGx+yUwObN1TuJLKri2+U7GpGIet/EYapqMnEuv6Fy9Z5mUTe0L/AsqDoqI/U
anxu1r85FTPI72xXZdLz988tFNTUeYN5POgrRaPCg7NSuOMB3Tpk/OILJAIJKGBQ
r/QbjbGuUEjScdzH/O6q9wBfFExflff//QusUi7SsIFTniA609Ah5BOuI7F/dTGl
QyN2Bu/oXF6MnH9hGSaLCUvKg4ZXyKyoP1J3zIM2dj7tKLl0mQV1ahg4B+mvcQgU
zRo6I/jaoGfhA2NLchH12T1SqJ95SDcCn0kc8C7zYESI3dG1ViM6K6sjrd4Ac2ua
V3+nev0CgYEA30TfYYEj0QUTrFFMx7Tv4hvsggyP0bxJmJY9Zqzt01dPFAmUmV0/
R4fHTGXxIc2XnPkYi6WqM7g4pUXNUBTE9do9L8YIy22TvSA1o21bZ0yWX7VvK721
QB/V1Jb7cWUpFKseuvKHshoMgiwKB7x7GbGOwbx1vvMpXint13o58XcCgYEAx2cM
U0QrNZJmdxJUsDu2AoS4h/bL7opuKwfk/72TOMBGRzP2nKjMe+WRMJ0ea29ygBv6
nWx7gnU5NsAv7QFfByA2jF38YALwewdwCVzLZ4qNVlkEVkF5Ah0Xy4fxFr008E4T
+fFddQacKCeWMvJDDAvcgErj/2vG6oEd2eCfEWsCgYBXVtTfiqodKRRCE2eqs+An
Hm9NjGZyUGql0xff44QBaaUYnIrR18VaUQYon7RNWeSWVmdAsaS8KLOYC48+ZXGL
Dz1iQ+DK22mw0TnKXYwlA7PLauk7PjH6DLoUOJ/SAxWn7SzPSvLEPCZqgZnG3vd0
3J2Qsg2Jjgu/tz1ATqL+DwKBgQCdlDryRonbETH2YT8Z8mYYwWfO0uNARJdhXCDF
VbxVeeVP+amnDeJi+v1tLI1Qm8chpHq+E2/bneWz9dcp9g5x5CwXa2K5QTloEG2i
iHmZ/q1JEpnRzHXjjLg0ON72eFmwmhNBT1Pq2mlndjlFU5xWlb0QiZ56SGLvCVBc
0R0DtwKBgQClTrEOpYAfBBKVIUnieBbYkfVcTIjE6Dhquxc8yHdXhl9xeWOFTSYl
a0syRFl7WZv1IXdARYtD/YCz3FOAGQPyixLDG2jKPUeeJDWuumf/7BtbqzqlHSfb
6JL4FEHViN60nEE8x5+tF81loJeb/0a2faAORgMJfs0kj01Spp1wHw==
-----END RSA PRIVATE KEY-----
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArehpGoTexi70UPL7w2As
OTWj+HJrewxVkM3DMKU4CDzVpV4dCHRKciIA6rOYgrKnT/yH2z7UasHf9KBZENVN
dFqMCBTCNM4Pt4NDKZT4DNSw+2qSqxEtYb7gtOZfTJxv/MPhrZ1Fi672N8AfECqf
Zabf/HoVim4wtO1l+XMHdD/dpmQ7wQjF/BpIGLYQbI+0sgZWydOArwP7tDgeZJG7
nz+pgtgDhfkovLN9YYrgSr7IVydM71waCuC71ASQ6tTeUmlznP7mMJChND7trdNP
CSXuWt62JAPTSWqyjU3FW8pBNtOOhYaUe8RK8j1DuJxz94m6MEobnyUGiH5SCCnT
vQIDAQAB
-----END PUBLIC KEY-----
The public key can be safely shared with anyone but the private key must be kept secret. [7]
In RSA, there is no difference between ‘encryption’ and ‘decryption’: the same algorithm accepts the private or public key. The following table illustrates the result of encrypting input data.
input |
Operation |
output |
---|---|---|
hello, world |
apply(input, public_key) |
OAkrSvD8qcM5oFqY… |
hello, world |
apply(input, private_key) |
nVB/z/rIqlv/kb47… |
OAkrSvD8qcM5oFqY… |
apply(input, public_key) |
AHqRJWBGcmWaFDvf… |
OAkrSvD8qcM5oFqY… |
apply(input, private_key) |
hello, world |
nVB/z/rIqlv/kb47… |
apply(input, public_key) |
hello, world |
nVB/z/rIqlv/kb47… |
apply(input, private_key) |
U7F29J2MwRH9kVOO… |
Notice in the table above that the public key and private key act as opposites of each other. Applying the public key and then the private key will get you back to the input. The input is also recovered if the private key is applied first, followed by the public key. However, applying the public key twice (or any other key) will result in unrelated output.
This public/private key structure makes it possible for two strangers to communicate with each other privately. If I want to send you a private message, I use your public key to encrypt the message. As you are the only person with the matching private key, only you can decrypt the message. You can then reply to me by using my public key to send a response back: only I will be able to decrypt the message with my matching private key.
Asymmetric encryption also makes it possible to create digital signatures. A digital signature is the reverse of an encrypted message: it is a public message from a person that only the sender can generate. If I encrypt a message with my private key, anyone with the public key can decrypt the message. Of course, this doesn’t offer any secrecy to the message. Yet, it tells the world that I sent the message because I am the only person with the private key who could have encrypted the message.
Applications of the primitives
Password salting and hashing
One of the most common security problems occurs when an attacker gains access to a back-end database and can see every user’s password.
For example, there may be an internal database table as follows:
Username |
|
Password |
---|---|---|
Mike |
asdf1234 |
|
Carol |
password |
|
Greg |
chevrolet |
|
Marcia |
daveyjones |
|
Peter |
chevrolet |
One way to protect the password is to hash it. That way, attackers cannot see passwords even if they steal the database.
Username |
|
sha256(Password) |
---|---|---|
Mike |
312433c28349f63c4f387953ff337046e794bea0f9b9ebfcb08e90046ded9c76 |
|
Carol |
5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8 |
|
Greg |
9e48bad6e31c128258257bd997eacca666f0dae2049027b545ea48a0cf43ed71 |
|
Marcia |
46a5d9d9f45ef11fa3aada67a04467e8591f220cbe1fea881b7c525e50451359 |
|
Peter |
9e48bad6e31c128258257bd997eacca666f0dae2049027b545ea48a0cf43ed71 |
The system verifies a password by hashing the input text. If the hashed input matches the saved hash of the password, then the input must correctly match the original password.
While this increases security, there is still a problem. We can see that popular passwords have identical hashes. For example, because Greg and Peter have the same password, their hashed passwords are also identical. If Peter accesses this database, he knows that he can use his password to also log in as Greg.
The way to address this problem is to add some random values to the password before hashing. This random data is called salt. A simple (but still slightly flawed) way to hash with salt is to add the salt to the end of the password when hashing:
Username |
|
Salt (randomly generated) |
sha256(Password + Salt) |
---|---|---|---|
Mike |
ca978112ca |
e6496e1fc73992ceac14c9fa40886635d1713d149e5b3073c4dd44a9c98af512 |
|
Carol |
3e23e81600 |
02f03ff9db9707dcc1763f471e89b7ef00298cae90e9776175934f2fd42f34dc |
|
Greg |
2e7d2c03a9 |
125677dfd55a0e2a5c1b1ac3330451d010c78ecbef84607d5779c53dd199f0fd |
|
Marcia |
18ac3e7343 |
53527b305b23d6b74a894b5b65b3dc86a57e2ddde6bceb1c98890a39644ea9e7 |
|
Peter |
3f79bb7b43 |
dec183b815f56678e54fc7475f671923d48d4c7b24ad2f2b13aff0dedb99ba2e |
Verification of a password then happens in three stages:
-
Retrieve the user’s salt
-
Hash the combined input text and salt
-
Compare the hash to the stored salted hash
Note that every hash is different, even ones with identical passwords!
Of course, it is generally a bad idea to write your own cryptography code. A simple salting and hashing scheme will provide some security. To have state-of-the-art security, you should use libraries developed by professional cryptographers and have been ‘battle-tested’ by extensive use in major websites globally. The bcrypt library on NPM is an excellent choice and the most popular library. Other options include Argon2 (recommended by OWASP, but needs to be tuned) and PBKDF2 (recommended by NIST, but more difficult to configure).
Tip
|
If you are building a web application, create two separate accounts with the same password. If both accounts have the same password in the database (i.e., after hashing or salting), something is wrong! Every password should be unique after salting. |
Tokens and identifiers
Some data has no apparent primary key. Examples are listed below:
-
The shopping cart or session of a user who has not logged in (e.g., the shopping cart created on
amazon.com
when you visit as a guest) -
A newly created blank document in an online editor (e.g., a new document named “Untitled”, especially if the creation time might not be unique)
-
A message in a social network (e.g., a tweet or a post on a social network)
These situations require a unique identifier or token to represent the data. If this token is predictable, then access needs to be carefully checked. However, if the token is unpredictable and long, then the token itself can be used to grant access to the underlying information.
For example, a Google Doc can be made visible to ‘anybody with the link’:
https://docs.google.com/document/d/1UqutaK9WjvgBZ1Zo8qN8rZcaRzYQZTwSWcFL2pJMJ6w/edit?usp=sharing. This document URL contains a long random identifier (1UqutaK9WjvgBZ1Zo8qN8rZcaRzYQZTwSWcFL2pJMJ6w
) that is impossible to guess by chance.
Another example is the concept of sessions available in most web development frameworks. [8] Sessions allow developers to associate data with users. When a new user arrives, the session management system generates a random token to identify the user. The server sends the token to the user’s browser as a cookie and uses it to match subsequent HTTP requests back to their sessions.
An important use of secure random numbers is to generate identifiers and tokens. Using insecure methods (such as a counter, the current time or Math.random()
) can make the token predictable. Secure random numbers will preserve the security of systems that depend on identifiers or tokens being difficult to guess.
Warning
|
MongoDB automatically adds an _id when inserting in a collection. The value might look be something like ObjectId("5eb8ed79d16196d3e8ee2a3e") . While this looks like a random number, MongoDB uses a predictable algorithm based on the current time.
|
Hash-based message authentication code (HMAC)
A hash-based message authentication code (HMAC) is a form of digital signature, created from a secure hash and a secret key or password.
A common use of HMACs occurs in websites with many users. HMACs allow end-users’ web browsers to store critical data without concern about tampering.
For example, when Alice logs in, it might take a long time to retrieve account details from the database and verify her identity and access permissions. If this needs to happen often, the obvious solutions are not satisfactory:
-
Verify the access permissions on every separate request (this is slow)
-
Cache the access permissions of recent users (this may use lots of memory)
-
Have Alice’s browser store a cache of the access permissions (Alice could tamper with the data to change her access permissions)
Instead, HMACs provide a way to prevent tampering of data. The server can digitally sign the access permissions and send the permissions and signature to Alice. Alice can present the digitally signed permissions on subsequent requests. The server can use the signature to verify that Alice has not tampered with the permissions.
A very basic HMAC can be created by hashing both data and a secret key known only by the server. Tampering by the client can be detected because changing the data will cause the hash to change. The client cannot regenerate a valid HMAC themselves because they don’t know the secret key.
In practice, more sophisticated algorithms securely combine the data and secret key to avoid security issues.
JSON web token (JWT)
JSON web tokens (JWT) are an industry-standard method to combine data with a digital signature in a URL-friendly format that works well with other web standards.
A JSON web token encodes three kinds of information into a single value:
-
A header: the details of the encryption/signing algorithms (e.g., it might specify that it uses an HMAC based on SHA-256, or that it uses an RSA digital signature)
-
A claim set: the data in the message (it might also contain information about the JWT, such as when it the token should expire) [9]
-
A digital signature: an HMAC or signature calculated according to the algorithm specified in the header
SSL/TLS
In Chapter 3, I discussed how HTTP uses TCP sockets to send data over the network. TCP does not use encryption: any router handling TCP sockets can read and intercept messages.
HTTP requests for URLs that begin with the https://
scheme, use an additional layer of encryption on top of TCP. This layer was initially named Secure Sockets Layer (SSL). It has evolved in new standards, now called Transport Layer Security (TLS).
TLS is a complex standard, offering a range of operation modes. It serves several purposes:
-
Privacy: only the sender and receiver understand the data
-
Authentication: users can verify the server (so that nobody other than Google can claim to be
www.google.com
) -
Integrity: nobody can modify data sent between the sender and receiver
Symmetric encryption is usually very fast, but asymmetric encryption/decryption tends to be slower. For this reason, TLS combines both symmetric and asymmetric encryption. TLS uses symmetric encryption with a randomly chosen key to efficiently encrypt large amounts of data. The randomly chosen symmetric key is exchanged once at the start, using slower asymmetric encryption.
TLS also uses digital signatures to verify identity. Built into your web browser is the public key for a company called GlobalSign. This company is one of several companies that specialize in verifying identities. Google employees have provided GlobalSign with identification documents. In return, GlobalSign digitally signs Google’s public key. So, when you visit www.google.com
, the web server provides its public key as well as the digital signature from GlobalSign to prove the authenticity of the public key.