5.

The Protocol

So, given what we now know about asymmetric encryption, certificates and RSA, let’s put it
together in a single protocol:

1. Alice needs a certificate:

a. She chooses a public exponent ​e​ and generates a private key
b. She generates a public key from that private key
c. She generate a certificate signing request ➝ sends that to the CA
d. The CA generates a certificate for Alice and signs it ➝ sends it to Alice
2. Alice gets Bob’s certificate from him:
a. She verifies it using the CA’s certificate (already pre-installed on her computer)
b. She extracts Bob’s public key from it
c. She attempts to encrypt her large message using Bob’s public key ➝ error!
3. Alice picks a symmetric key:
a. She picks a strong symmetric key using a pseudo-random number generator
b. She encrypts it with Bob’s public key ➝ symkey.enc
c. She hashes it and then encrypts it with her private key ➝ signature.bin
d. She sends both symkey.enc and signature.bin to Bob
4. Bob deciphers and verifies the symmetric key:
a. He decrypts symkey.enc using his private key
b. He gets and verifies Alice’s certificate and extracts her public key
c. He decrypts signature.bin using Alice’s public key
d. He compares a hashed symkey with the decrypted signature, they must match
5. Alice encrypts her large message with that symmetric key:
a. She needs to use choose a symmetric key encryption algorithm
b. Bob decrypts using the symmetric key and that same algorithm
c. She can also use something like ​HMAC​ now for authentication (this won’t be
covered in this document but it’s similar to how she creates her signature, and
you can read more ​here​)

Note​: In real life, the protocols used are a little more complicated than this. You’ll notice that
both parties need to be using the same hashing and encryption algorithms, requiring more
initial communication (this is done in the TLS handshake for example).

9
6. OpenSSL Demo

Here we’ll implement all the steps of that protocol, using openssl terminal commands. In
practice you’re more likely to use openssl in the form of an API in another language- but
learning the terminal commands is still valuable as a transferable skill. Each command is
displayed with some explanations of its flags below.

If you want to follow along, you can make 3 folders, 1 for Alice, Bob and the CA respectively.
You need to repeat steps 1.a and 1.b for Bob and CA so they can have their own pair of keys.
And you need to generate a self-signed certificate for the CA (shown below).

Step 1.a - Alice generates a private key

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -pkeyopt

rsa_keygen_pubexp:3 -out privkey-A.pem

● genpkey ➝ ​generate a private key

● -algorithm RSA ➝​ use the RSA algorithm (can also take “EC” for elliptic-curve)
● -pkeyopt opt:value ➝​ set opt to value (see items below)
● rsa_keygen_bits:2048 ➝ ​sets the size of the key to 2048 bits (the default is 1024)
● rsa_keygen_pubexp:3 ➝ ​sets the public exponent ​e​ to 3 (default is 65, 537)
● -out privkey-A.pem ➝​ outputs to the file privkey-A.pem

Step 1.b - Alice generates a public key

openssl pkey -​in​ privkey-A.pem -pubout -out pubkey-A.pem

● pkey ➝ ​processes public or private keys

● -in privkey-A.pem ➝ ​read the key from filename privkey-A.pem
● -pubout ➝ ​output a public key (by default, a private key is output)

Aside: viewing the keys in plain text

The keys are saved in base64, and aren’t human readable if you open them in a text editor or
the terminal. Luckily, openssl provides us with a handy set of commands to convert them to
text. The (​-noout​) flag suppresses the command from printing out the base64 encoding as well.

openssl pkey -​in​ privkey-A.pem -text -noout

openssl pkey -pubin -​in​ pubkey-A.pem -text -noout

10
Note the size difference in the private key and the public key (one is a subset of the other,
afterall). There are some additional values stored in the private key that you won’t recognize
(exponent1, exponent2 and coefficient). These are stored by openssl to speed up decryption.

Step 1.c - Alice generates a certificate signing request

openssl req -new -key privkey-A.pem -out A-req.csr

● req ➝​ creates and processes signing requests

● -new ➝​ generates a new certificate request, will prompt Alice for some information
● -key privkey-A.pem ➝ ​signs the request with Alice’s private key

The command will prompt Alice with these questions:

● C​ountry code [​C​]: {Alice fills in her country code}
● Province/​ST​ate name [​ST​]: {Alice fills in her province name fully}
● City/​L​ocation [​L​]: {The city Alice’s business is registered in, for example}
● O​rganization Name [​O​]: {Alice’s business name, for example}
● O​rganizational ​U​nit Name [​OU​]: (Optional) {What part of the company is she?}
● C​ommon ​N​ame [​CN​]: the hostname+domain, i.e. “www.alice.com”
● A challenge password []: {this can be used as a secret nonce between Alice and CA}

Aside: generating a self-signed certificate for the CA

openssl req -x509 -new -nodes -key rootkey.pem -sha256 -days 1024 -out
root.crt

Step 1.d - CA generates and signs a certificate for Alice

openssl x509 -req -​in​ A-req.csr -CA root.crt -CAkey rootkey.pem

-CAcreateserial -out A.crt -days 500 -sha256

● x509 ➝ ​an x509 certificate utility (displays, converts, edits and signs x509 certificates)
● -req ➝ ​a certificate request is taken as input (default is a certificate)
● -CA root.crt ➝ ​specifies the CA certificate to be used as the issuer of Alice’s certificate
● -CAkey rootkey.pem ➝ ​specifies the private key used in signing (rootkey.pem)
● -CAcreateserial ➝ ​creates a serial number file which contains a counter for how many
certificates were signed by this CA
● -days 500 ➝ ​sets Alice’s certificate to expire in 500 days
● -sha256 ➝ ​specifies the hashing algorithm to be used for the certificate’s signature

11
Aside: viewing the certificate as text

openssl x509 -​in​ Alice.crt -text -noout

Step 2.a - Alice verifies Bob’s public certificate

openssl verify -CAfile root.crt Bob.crt

● verify ➝ ​a utility that verifies certificate chains

● -CAfile root.crt ➝ ​specified the trusted certificate (root.crt)
● Bob.crt ➝​ the certificate to verify
● If you get an OK, you know the certificate can be trusted

Step 2.b - Alice extracts Bob’s public key

openssl x509 -pubkey -​in​ Bob.crt -noout > pubkey-B.pem

● -pubkey ➝ ​outputs the certificate’s public key (in ​PEM format​)

Step 2.c - Alice tries to encrypt her largefile.txt with Bob’s public key

openssl pkeyutl -encrypt -​in​ largefile.txt -pubin -inkey pubkey-B.pem -out

ciphertext.bin

● pkeyutl ➝​ utility to perform public key operations

● -encrypt ➝​ encrypt the input data
● error! ​(recall: RSA is not meant for encrypting arbitrary large files- Alice needs to use
symmetric key encryption for that)

Step 3.a - Alice generates a symmetric key

openssl rand -base64 32 -out symkey.pem

● rand ➝​ generates pseudo-random bytes (seeded by default by $HOME/.rnd)

● -base64 32 ➝​ outputs 32 random bytes and encodes it in base64

12
 Step 3.b - Alice encrypts symkey.pem using Bob’s public key

openssl pkeyutl -encrypt -​in​ symkey.pem -pubin -inkey pubkey-B.pem -out

symkey.enc

Step 3.c - Alice hashes symkey.pem and encrypts it using her private key

openssl dgst -sha1 -sign privkey-A.pem -out signature.bin symkey.pem

● dgst -sha1 ➝​ hash the input file using the sha1 algorithm
● -sign privkey-A.pem ➝​ sign the hash with the specified private key
● symkey.pem ➝​ the input file to be hashed

Step 4.a - Bob decrypts symkey.enc using his private key

openssl pkeyutl -decrypt -​in​ symkey.enc -inkey privkey-B.pem -out

symkey.pem

● -decrypt ➝​ decrypt the input file

Step 4.b - Bob gets and verifies Alice’s certificate and extracts her public key

(This is simply a retread of what Alice did in step 2)

Step 4.c - Bob verifies the message is from Alice

Steps 4.c and 4.d in the protocol are combined in this step. Bob hashes symkey.pem, decrypts
signature.bin, and compares the two results in one command:

openssl dgst -sha1 -verify pubkey-A.pem -signature signature.bin symkey.pem

● -verify pubkey-A.pem ➝​ verify the signature using the specified filename

● -signature signature.bin ➝ ​specifies the signature to be verified
● symkey.pem ➝ ​the file to be hashed

Step 5.a - Alice encrypts her largefile.txt with the symmetric key

13
 openssl enc -aes-256-cbc -pass file:symkey.pem -p -md sha256 -​in
largefile.txt -out ciphertext.bin

● enc -aes-256-cbc ➝ ​encrypt a file using the aes-256-cbc symmetric key algorithm
● -pass file:symkey.pem ➝​ specified the file to get the symmetric key from
● -p ➝​ prints the key, salt, initialization vector to the screen
● -md sha256 ➝​ uses sha256 as part of the ​key derivation function​ (a f​ unction that
derives one or more secondary secret keys from a primary secret key)

Step 5.b - Bob decrypts ciphertext.bin with the same symmetric key

openssl enc -aes-256-cbc -d -pass file:symkey.pem -p -md sha256 -​in

ciphertext.bin -out largefile.txt

● -d ➝​ decryption flag

7. References

https://www.openssl.org/

https://prefetch.net/articles/realworldssl.html

https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html

https://www.oreilly.com/library/view/network-security-with/059600270X/

http://heartbleed.com/

14