In this talk, we will begin our journey looking at the RFCs behind these technologies. Next, we will use OpenSSL, CFSSL, and mkcert to validate what we have learned about X509 v3 certificates. Then we will use the certificates we make to bootstrap Consul, Vault, and Nomad clusters with mTLS enabled so we can get familiar with terminology and error messages. Finally, we will look at their source code to learn how we might implement the same ideas in our projects.
5. So what?
1. Managing certificates in 2020 is still hard.
2. We keep repeating the same errors in our software.
3. It all starts with education, and a foundation in the basics.
7. Agenda
Hands-on Practice
Use practical tools like mkcert
and HashiCorp Consul to get
familiar with how certificates
can be managed and used.
First Principles
Learn what makes up a
certificate and how those
pieces come together to for a
complete system.
Learn the Code
Examine a few code bases
which utilize mTLS to get work
done.
11. Certificate Basic Fields - 1/3
▪ Version Number
– The version of the encoded certificate.
▪ Serial Number
– A positive integer unique to the Certificate Authority.
▪ Signature Algorithm
– Contains the identifier for the cryptographic algorithm used by the CA to
sign the certificate.
12. Certificate Basic Fields - 2/3
▪ Issuer
– Identifies the entity that has signed and issued the certificate.
▪ Validity
– The time interval during which the CA warrants that it will maintain
information about the status of the certificate.
▪ Subject
– Identifies the entity associate with the public key stored in the subject
public key field.
13. Certificate Basic Fields - 3/3
▪ Subject Public Key Info
– Used to carry the public key and identify the algorithm with which the key
is used.
▪ Extensions
– Provide methods for associating additional attributes to certificates.
15. Let’s Look at a Certificate
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ba:26:56:af:26:bd:3c:1a:e5:05:9d:fa:0b:83:40:26
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Validity
Not Before: Feb 18 22:34:27 2020 GMT
Not After : Feb 18 22:34:27 2030 GMT
...
18. OpenSSL Command
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
This is the command which is used for displaying and signing X509 formatted
certificates.
19. OpenSSL Command
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
This specifies the input filename to read a certificate from.
20. OpenSSL Command
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
-text : Prints out the certificate in text form.
-noout : Prevents output of the encoded version of the certificate.
21. Version
The version of the encoded certificate.
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ba:26:56:af:26:bd:3c:1a:e5:05:9d:fa:0b:83:40:26
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Validity
Not Before: Feb 18 22:34:27 2020 GMT
Not After : Feb 18 22:34:27 2030 GMT
...
22. Serial Number
A positive integer unique to the Certificate Authority.
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ba:26:56:af:26:bd:3c:1a:e5:05:9d:fa:0b:83:40:26
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Validity
Not Before: Feb 18 22:34:27 2020 GMT
Not After : Feb 18 22:34:27 2030 GMT
...
23. Signature Algorithm
Contains the identifier for the cryptographic algorithm used by the CA to sign this certificate.
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ba:26:56:af:26:bd:3c:1a:e5:05:9d:fa:0b:83:40:26
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Validity
Not Before: Feb 18 22:34:27 2020 GMT
Not After : Feb 18 22:34:27 2030 GMT
...
24. Issuer
Identifies the entity that has signed and issued the certificate.
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ba:26:56:af:26:bd:3c:1a:e5:05:9d:fa:0b:83:40:26
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Validity
Not Before: Feb 18 22:34:27 2020 GMT
Not After : Feb 18 22:34:27 2030 GMT
...
25. Validity
The time interval during which the CA warrants that it will maintain information about the status of the certificate.
TERMINAL
> openssl x509 -in ~/.local/share/mkcert/rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ba:26:56:af:26:bd:3c:1a:e5:05:9d:fa:0b:83:40:26
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Validity
Not Before: Feb 18 22:34:27 2020 GMT
Not After : Feb 18 22:34:27 2030 GMT
...
26. Subject
Identifies the entity associate with the public key stored in the subject public key field.
TERMINAL
Subject: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (3072 bit)
Modulus: [large integer represented in colon-hexadecimal notation]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Subject Key Identifier:
B9:D5:B3:06:55:B4:E6:CE:CB:CB:56:B3:4A:35:96:A3:AA:5F:2D:C4
27. Subject Public Key Info
Used to carry the public key and identify the algorithm with which the key is used.
TERMINAL
Subject: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (3072 bit)
Modulus: [large integer represented in colon-hexadecimal notation]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Subject Key Identifier:
B9:D5:B3:06:55:B4:E6:CE:CB:CB:56:B3:4A:35:96:A3:AA:5F:2D:C4
28. Extensions: Key Usage
Defines the purpose of the key contained in the certificate.
TERMINAL
Subject: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (3072 bit)
Modulus: [large integer represented in colon-hexadecimal notation]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Subject Key Identifier:
B9:D5:B3:06:55:B4:E6:CE:CB:CB:56:B3:4A:35:96:A3:AA:5F:2D:C4
29. Extensions: Basic Constraints
Identifies whether the subject of the certificate is a CA and
the maximum depth of valid certification paths that include this certificate.
TERMINAL
Subject: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (3072 bit)
Modulus: [large integer represented in colon-hexadecimal notation]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Subject Key Identifier:
B9:D5:B3:06:55:B4:E6:CE:CB:CB:56:B3:4A:35:96:A3:AA:5F:2D:C4
30. Extensions: Subject Key Identifier
Provides a means of identifying certificates that contain a particular public key.
TERMINAL
Subject: O = mkcert development CA,
OU = ascherger@incontrol (Alan Scherger),
CN = mkcert ascherger@incontrol (Alan Scherger)
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (3072 bit)
Modulus: [large integer represented in colon-hexadecimal notation]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Subject Key Identifier:
B9:D5:B3:06:55:B4:E6:CE:CB:CB:56:B3:4A:35:96:A3:AA:5F:2D:C4
32. Blue Book
Volume VIII - Fascicle VIII.8
Data Communication Networks Directory
Recommendations X.500-X.521
🔥 236 glorious pages of explanation 🔥
34. Certificate Chain of Trust
A list of certificates (usually starting with an end-entity
certificate) followed by one or more CA certificates (usually the
last one being a self-signed certificate), with the following
properties:
1. The Issuer of each certificate (except the last one)
matches the Subject of the next certificate in the list.
2. Each certificate (except the last one) is supposed to be
signed by the secret key corresponding to the next
certificate in the chain (i.e. the signature of one
certificate can be verified using the public key contained
in the following certificate).
3. The last certificate in the list is a trust anchor: a
certificate that you trust because it was delivered to you
by some trustworthy procedure.
39. iOS 13 and macOS 10.15 - Gotchya
https://support.apple.com/en-us/HT210176
40. iOS 13 and macOS 10.15 - Gotchya
https://support.apple.com/en-us/HT210176
41. Install
Download the latest
Release
Use the -install flag to
generate an install
certificates.
TERMINAL
> chmod +x ~/Downloads/mkcert-v1.4.1-darwin-amd64
> ~/Downloads/mkcert-v1.4.1-darwin-amd64 -install
Created a new local CA at
"/Users/ascherger/Library/Application Support/mkcert" 💥
Sudo password:
The local CA is now installed in the system trust store!
⚡
Warning: "certutil" is not available, so the CA can't be
automatically installed in Firefox! ⚠
Install "certutil" with "brew install nss" and re-run
"mkcert -install" 👈
>
42. TERMINAL
> ls -al /Users/ascherger/Library/Application Support/mkcert
total 16
drwxr-xr-x 4 ascherger staff 128 Feb 16 22:09 .
drwx------+ 121 ascherger staff 3872 Feb 16 22:09 ..
-r-------- 1 ascherger staff 2488 Feb 16 22:09 rootCA-key.pem
-rw-r--r-- 1 ascherger staff 1728 Feb 16 22:09 rootCA.pem
>
Look at our
certificates.
rootCA-key.pem is the private key.
rootCA.pem is the public certificate.
44. TERMINAL
> openssl x509 -in rootCA.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
84:4a:83:84:72:ad:27:89:09:7e:48:44:b9:f6:30:6e
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=mkcert development CA,
OU=ascherger@Alans-MacBook-Pro.local,
CN=mkcert ascherger@Alans-MacBook-Pro.local
Validity
Not Before: Feb 17 04:09:41 2020 GMT
Not After : Feb 17 04:09:41 2030 GMT
Subject: O=mkcert development CA,
OU=ascherger@Alans-MacBook-Pro.local,
CN=mkcert ascherger@Alans-MacBook-Pro.local
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (3072 bit)
Modulus:
00:e0:70:56:33:aa:83:d5:ed:0f:46:f1:99:d5:81:
...
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Subject Key Identifier:
93:EF:36:51:3D:94:46:01:8F:01:F7:B9:22:09:75:F9:E7:63:93:F9
Signature Algorithm: sha256WithRSAEncryption
bb:0e:80:b4:35:b8:2a:58:9e:36:f3:4a:ce:87:5c:0b:86:54:
...
Inspect the
Root CA
Public
Certificate
45. Make a new
localhost
certificate.
TERMINAL
> mkcert localhost 127.0.0.1 ::1
Using the local CA at
"/Users/ascherger/Library/Application Support/mkcert" ✨
Created a new certificate valid for the following names
📜
- "localhost"
- "127.0.0.1"
- "::1"
The certificate is at "./localhost+2.pem" and the key at
"./localhost+2-key.pem" ✅
>
46. Serial Number
Remember “18:f1” we’ll see that again soon.
TERMINAL
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
18:f1:88:4e:19:56:0f:7a:ae:11:75:eb:e9:67:8d:57
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=mkcert development CA,
OU=ascherger@Alans-MacBook-Pro.local,
CN=mkcert ascherger@Alans-MacBook-Pro.local
Validity
Not Before: Jun 1 00:00:00 2019 GMT
Not After : Feb 19 07:05:58 2030 GMT
Subject: O=mkcert development certificate,
OU=ascherger@Alans-MacBook-Pro.local
47. Issuer
Matches our Root CA public key information.
TERMINAL
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
18:f1:88:4e:19:56:0f:7a:ae:11:75:eb:e9:67:8d:57
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=mkcert development CA,
OU=ascherger@Alans-MacBook-Pro.local,
CN=mkcert ascherger@Alans-MacBook-Pro.local
Validity
Not Before: Jun 1 00:00:00 2019 GMT
Not After : Feb 19 07:05:58 2030 GMT
Subject: O=mkcert development certificate,
OU=ascherger@Alans-MacBook-Pro.local
48. Validity
Patched to get around the macOS hiccup.
TERMINAL
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
18:f1:88:4e:19:56:0f:7a:ae:11:75:eb:e9:67:8d:57
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=mkcert development CA,
OU=ascherger@Alans-MacBook-Pro.local,
CN=mkcert ascherger@Alans-MacBook-Pro.local
Validity
Not Before: Jun 1 00:00:00 2019 GMT
Not After : Feb 19 07:05:58 2030 GMT
Subject: O=mkcert development certificate,
OU=ascherger@Alans-MacBook-Pro.local
49. Subject
No Common Name (CN), and different than CA information.
TERMINAL
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
18:f1:88:4e:19:56:0f:7a:ae:11:75:eb:e9:67:8d:57
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=mkcert development CA,
OU=ascherger@Alans-MacBook-Pro.local,
CN=mkcert ascherger@Alans-MacBook-Pro.local
Validity
Not Before: Jun 1 00:00:00 2019 GMT
Not After : Feb 19 07:05:58 2030 GMT
Subject: O=mkcert development certificate,
OU=ascherger@Alans-MacBook-Pro.local
50. Key Usage
Explicitly allows for Web Server usage, but not Client usage.
TERMINAL
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c0:ed:e2:11:01:66:60:d1:c6:50:cd:e0:7a:a3:
...
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
51. Basic Constraints
This certificate cannot act as a CA, so it cannot make child certificates.
TERMINAL
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c0:ed:e2:11:01:66:60:d1:c6:50:cd:e0:7a:a3:
...
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
52. Authority Key Identifier
Matches the Subject Key Identifier of the Root certificate.
TERMINAL
X509v3 Authority Key Identifier:
keyid:93:EF:36:51:3D:94:46:01:8F:01:F7:B9:22:09:75:F9:E7:63:93:F9
X509v3 Subject Alternative Name:
DNS:localhost, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1
Signature Algorithm: sha256WithRSAEncryption
4b:f5:d0:fa:27:43:c2:d8:ef:4c:be:5e:66:81:21:c1:c1:5f:
...
53. Subject Alternative Name
A list of DNS and IP addresses this certificate is allowed to represent.
TERMINAL
X509v3 Authority Key Identifier:
keyid:93:EF:36:51:3D:94:46:01:8F:01:F7:B9:22:09:75:F9:E7:63:93:F9
X509v3 Subject Alternative Name:
DNS:localhost, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1
Signature Algorithm: sha256WithRSAEncryption
4b:f5:d0:fa:27:43:c2:d8:ef:4c:be:5e:66:81:21:c1:c1:5f:
...
60. Consul + TLS
1. Consul CLI has a tls command for setting up a CA and certificates.
2. Consul has an auto_encrypt feature for auto-managing certificates
3. Consul’s Connect API contains the CA endpoints, and supports automated
CA rotation through cross-signing.
66. Consul - Encryption
If verify_server_hostname is set, then outgoing
connections perform hostname verification.
All servers must have a certificate valid for
server.<datacenter>.<domain> or the client will reject the
handshake.
67. package tlsutil - config.go L706-L726
CODE EDITOR
// Wrap a net.Conn into a client tls connection, performing any
// additional verification as needed.
//
// As of go 1.3, crypto/tls only supports either doing no certificate
// verification, or doing full verification including of the peer's
// DNS name. For consul, we want to validate that the certificate is
// signed by a known CA, but because consul doesn't use DNS names for
// node names, we don't verify the certificate DNS names. Since go 1.3
// no longer supports this mode of operation, we have to do it
// manually.
func (c *Configurator ) wrapTLSClient (dc string, conn net.Conn) (net.Conn, error) {
config := c.OutgoingRPCConfig ()
verifyServerHostname := c.VerifyServerHostname ()
verifyOutgoing := c.verifyOutgoing ()
domain := c.domain()
if verifyServerHostname {
// Strip the trailing '.' from the domain if any
domain = strings.TrimSuffix(domain, ".")
config.ServerName = "server." + dc + "." + domain
}
tlsConn := tls.Client(conn, config)
68. Nomad - Securing With TLS
To fulfill the desired security properties Nomad
certificates are signed with their region and role such as:
- client.global.nomad for a client node in the global
region
- server.us-west.nomad for a server node in the
us-west region
69. package tls - common.go L510-L521
CODE EDITOR
// VerifyPeerCertificate, if not nil, is called after normal
// certificate verification by either a TLS clientor server. It
// receives the raw ASN.1 certificates provided by the peer and also
// any verified chains that normal processing found. If it returns a
// non-nil error, the handshake is aborted and that error results.
//
// If normal verification fails then the handshake will abort before
// considering this callback. If normal verification is disabled by
// setting InsecureSkipVerify, or (for a server) when ClientAuth is
// RequestClientCert or RequireAnyClientCert, then this callback will
// be considered but the verifiedChains argument will always be nil.
VerifyPeerCertificate func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error
70. package nomad - server.go L451-L477
CODE EDITOR
// getTLSConf gets the server's TLS configuration based on the config supplied
// by the operator
func getTLSConf(enableRPC bool, tlsConf *tlsutil.Config, region string)
(*tls.Config, tlsutil.RegionWrapper , error) {
// omitted for slide
if tlsConf.VerifyServerHostname {
incomingTLS = itls.Clone()
incomingTLS .VerifyPeerCertificate =
rpcNameAndRegionValidator (region)
} else {
incomingTLS = itls
}
return incomingTLS , tlsWrap, nil
}
71. package nomad - server.go L479-L497
CODE EDITOR
// implements signature of tls.Config.VerifyPeerCertificate which is called
// after the certs have been verified. We'll ignore the raw certs and only
// check the verified certs.
func rpcNameAndRegionValidator(region string) func([][]byte, [][]*x509.Certificate)
error {
return func(_ [][]byte, certificates [][]*x509.Certificate) error {
if len(certificates) > 0 && len(certificates[0]) > 0 {
cert := certificates[0][0]
for _, dnsName := range cert.DNSNames {
if validateRPCRegionPeer(dnsName, region) {return nil}
}
if validateRPCRegionPeer(cert.Subject.CommonName, region) {
return nil
}
}
return errors.New("invalid role or region for certificate")
}
72. package nomad - server.go L499-L515
CODE EDITOR
func validateRPCRegionPeer (name, region string) bool {
parts := strings.Split(name, ".")
if len(parts) < 3 {
// Invalid SAN
return false
}
if parts[len(parts)-1] != "nomad" {
// Incorrect service
return false
}
if parts[0] == "client" {
// Clients may only connect to servers in their region
return name == "client."+region+".nomad"
}
// Servers may connect to any Nomad RPC service for federation.
return parts[0] == "server"
}