Self-hosting a static site with OpenBSD, httpd, and relayd

2022-07-08 - openbsd

My blog gets generated with Hugo, which I’m generally happy with. Until recently, I hosted the static files on Netlify but now decided to get my own little server again. There are two main reasons for this:

I actually missed doing some sysadmin work.
The Internet was supposed to be a federated system and I don’t want to outsource everything to a few tech giants.

Operating system choice

OpenBSD has always been one of my favorite (server) operating systems, for reasons that are nicely summarized at Why OpenBSD rocks. Most people seem to be drawn to it because of the promise of enhanced security (which others find debatable) but I primarily enjoy it for being a simple yet pretty full-featured system out of the box.

So when I decided to go back to hosting my own site, I went for a VM hosted with OpenBSD Amsterdam who donate part of each subscription to the OpenBSD Foundation.

SSL certificates

Since I didn’t mind a short downtime for my personal blog, I started by pointing my A and AAAA records at their new IPs and added a CAA record for Let’s Encrypt:

$ dig citizen428.net CAA +short
0 issue "letsencrypt.org"

OpenBSD comes with its own acme-client, configured via /etc/acme-client.conf (man page):

authority letsencrypt {
	api url "https://acme-v02.api.letsencrypt.org/directory"
	account key "/etc/acme/letsencrypt-privkey.pem"
}

domain citizen428.net {
	alternative names { www.citizen428.net }
	domain key "/etc/ssl/private/citizen428.net.key"
	domain full chain certificate "/etc/ssl/citizen428.net.crt"
	sign with letsencrypt
}

First, we set up Let’s Encrypt as a certificate authority, then we set up the certificates for my domain. Note the line that says

domain full chain certificate "/etc/ssl/citizen428.net.crt"

, we’ll circle back to that in a bit.

Web server

OpenBSD also comes with its own minimal web server, httpd. It’s very easy to configure (man page) and my initial configuration looked something like this:

server "citizen428.net" {
	listen on * port 80
	location * {
		block return 301 "https://$HTTP_HOST$REQUEST_URI"
	}
}

server "citizen428.net" {
	listen on * tls port 443
	tls {
		certificate "/etc/ssl/citizen428.net.pem"
		key "/etc/ssl/private/citizen428.net.key"
	}
	hsts
	log style combined
	root "/htdocs/citizen428.net"
	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
	}
}

This redirects all HTTP traffic to HTTPS and configures the SSL certificates and some other things, like the document root and the location for the Let’s Encrypt HTTP challenge. So far so good, this setup scored an A+ at SSL Labs.

Alas, the result on Security Headers was a lot less positive, I think my initial score was a D or something.

Adding a relay

Since httpd is kept simple on purpose, it doesn’t allow us to set the relevant security headers. Enter relayd, “a daemon to relay and dynamically redirect incoming connections to a target host.” We can use this to terminate TLS, forward requests to a local web server listening on port 8080, and set some response headers (abbreviated for clarity). Here’s my /etc/relayd.conf (man page):

include "/etc/ips.conf"

table <local> { 127.0.0.1 }

http protocol https {
	tls ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:..."
	tls keypair "citizen428.net"

	match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
	match request header append "X-Forwarded-Port" value "$REMOTE_PORT"

	match response header set "Referrer-Policy" value "same-origin"
	match response header set "X-Frame-Options" value "deny"
	match response header set "X-XSS-Protection" value "1; mode=block"
	match response header set "X-Content-Type-Options" value "nosniff"
	match response header set "Strict-Transport-Security" value "max-age=31536000; includeSubDomains; preload"
	match response header set "Content-Security-Policy" value "default-src 'none'; ..."
	match response header set "Permissions-Policy" value "accelerometer=(), .."
	match response header set "Cache-Control" value "max-age=86400"

	return error
	pass
}

relay wwwtls {
	listen on $ipv4 port 443 tls
	protocol https
	forward to <local> port 8080
}

relay www6tls {
	listen on $ipv6 port 443 tls
	protocol https
	forward to <local> port 8080
}

Of course, we also have to reconfigure httpd. We no longer terminate TLS and we need to listen on port 8080 where relayd will be relaying to:

server "citizen428.net" {
	listen on 127.0.0.1 port 8080
	root "/htdocs/citizen428.net"
	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
	}
}

# Redirect www to naked domain
server "www.citizen428.net" {
	listen on 127.0.0.1 port 8080
	block return 301 "$HTTP_HOST$REQUEST_URI"
}

# Redirect http to https
server "citizen428.net" {
	alias "www.citizen428.net"
	listen on * port 80
	block return 301 "$HTTP_HOST$REQUEST_URI"
}

(Re-)start both services and bingo, we are scoring an A on Security Headers again. However, there was one small issue, my SSL Labs score was now capped at a B, with the following explanatory message:

This server’s certificate chain is incomplete. Grade capped to B

No bueno, what to do? After some googling I found this blog post which pointed me in the right direction: relayd looks for certificate chains in /etc/ssl/private/name:port.key and /etc/ssl/name:port.crt, falling back to /etc/ssl/private/name.key and /etc/ssl/name.crt respectively. My original acme-client.conf did save the full chain with a .pem extension, whereas the .crt file only contained the certificate for the specific domain. There probably would have been more elegant ways to solve this, but the easiest solution was to just store the full chain in the .crt file as mentioned above:

domain full chain certificate "/etc/ssl/citizen428.net.crt"

Compressed HTTP responses

A couple days after my OpenBSD setup went live, I was delighted to find out that httpd can serve compressed files since OpenBSD 7.1. So I added an extra step to my Makefile which I also use to deploy the site with rsync:


clean:
	rm -r public

build: clean
	hugo

gzip: build
	find public/ -type f ! -name '*.png' -exec gzip -9k "{}" \;

...

.PHONY: serve clean build gzip deploy

The find command looks for everything that’s not a PNG, compresses it with the highest compression (-9) and keeps the original file around (-k).

To make use of this we need to update httpd.conf to include the gzip-static directive:

server "citizen428.net" {
	listen on $local port 8080
	root "/htdocs/citizen428.net"
	gzip-static # Newly added
	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
	}
}

Odds and ends

Since I don’t want to worry about forgetting certificate renewals, I added the following to /etc/daily.local (man page):

next_part "Refreshing Let's Encrypt certificates"
acme-client citizen428.net && rcctl reload relayd

This will check if a new certificate is available and restart relayd (where we’re terminating TLS connections) if necessary.

I also set up a very basic firewall with pf, primarily for blocking SSH brute-force attempts:

table <bruteforce> persist

set skip on lo

block quick from <bruteforce>
block return	# block stateless traffic
pass		# establish keep-state

pass quick proto tcp from any to any port ssh \
     flags S/SA keep state \
     (max-src-conn 5, max-src-conn-rate 5/30, \
     overload <bruteforce> flush global)

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

Last but not least, I added a .forward file (man page) in my user’s home directory so the mails generated by /etc/daily and the daily security scan get forwarded to my real address where I’m more likely to actually read them.

Summary

Overall this was a fairly quick and painless migration, and I’m very happy with the outcome. Not only am I now fully in charge of my own site again, it still scores an A+ on SSL Labs, an A on Security Headers, and an A+ on the Mozilla Observatory. It’s also still eligible for HSTS preload and generally scores well on PageSpeed Insights. Not too shabby for a little VM without a CDN. :-)

Resources

The following blog posts came in handy at various times and while I already credited some of them in the text I wanted to explicitly list them here one more time: