ewintr.nl

Dipping my toes in OpenBSD, in Amsterdam

Sometimes the water is cold at first.

$ nano
ksh: nano: not found
$ vim   
ksh: vim: not found

A couple of years ago, I was playing with an ancient laptop, a Toshiba 300 CDT. I can't remember exactly, but I believe it had an internal memory of 32 MB. Sources on the internet differ on the exact specification, perhaps there were different models, but we are definitely talking about megabytes, not gigabytes. I was wondering whether it would be possible to get such an old device connected to the internet securely and do "something real" with it. Could it still be useful in some way, however limited? It was released in 1998. Back then, people had to-do lists. They were using word processors, and they went online to visit websites. Why wouldn't that still be possible?

I started by looking into several small Linux distributions, like Damn Small Linux and Puppy Linux. Unfortunately, those proved to be too much for this old machine. I started a half-hearted attempt to do Linux From Scratch, figuring that that would be the most basic and minimal Linux possible. But that turned into a whole project in itself, and I got distracted by other kinds of OSes. I tried Haiku, Minix and many, many more. This list is just what pops up in my head now, years later. Naturally, I got to the BSDs as well, but all to no avail. Nothing would work on this ancient laptop, the limited memory being the culprit most of the time.

Well, nothing except OpenBSD. Installing and seeing it boot was a puzzling moment. After all the failures, this one simply... just worked? Funny. I just wanted something to play with, but because I was going to connect it to the internet, I wanted something secure and supported. And then the only OS that worked was the one people keep bringing up as the most secure OS available. That probably says something about the virtues of simplicity and minimalism.

I did not use it very long, though. It took 15 minutes to boot, thanks to one of the security features. I could likely have disabled that, but learning OpenBSD turned out to be a challenge. For a Linux user, it all looks so familiar. Yet, it is different enough that I just kept stumbling around and got nowhere.

$ pkg_add helix
pkg_add: pkg_add must be run as root
$ doas pkg_add helix
doas: doas is not enabled, /etc/doas.conf: No such file or directory

Learning can be difficult. Learning is taking in something new and giving that a place in the model of the world that you already have. To connect the new thing to the things you already know. If it is truly new, that might prove to be difficult. There is no obvious place in your mental model to put the new thing.

You may think you are open-minded, that you are flexible enough to learn new things, but often that only means that you are ready for square pegs of various new sizes. What happens if you encounter a round one? It may take several failed attempts before the concept of roundness even emerges in your head.

$ doas rcctl restart relayd
(failed)

Sometimes you are the round peg. Take the term KYC. It stands for Know Your Customer, and it means that organizations must take steps to make sure you, a potential customer, are not a fraudster. They are required by law to learn who you are, so they can determine if it is okay to start a business relationship with you.

For me, getting to know someone simply means talking to them. Preferably over some beer or coffee. For many companies, "getting to know you" nowadays means asking you to stare into your webcam so that an AI can take a good look at you. Or to submit a scan of your passport, that they promise to process securely because your privacy is very, very important to them. Somehow, I never believe it when they say that.

When I started looking for a new way to host this site because I was done playing around at the previous hoster, and now wanted something simple and stable, I encountered many of those companies that wanted to get to know me in the most awkward way possible. These days, buying European is all the rage, but European companies are just the same. Soulless websites with forms that require intimate information. They really, really value my pri... yeah, right.

But then I stumbled upon OpenBSD.Amsterdam. A local offering of OpenBSD VMs. The organization pledges to use part of the money they charge for sponsoring the development of OpenBSD, which is cool. It felt no-nonsense and rock solid, like the OS itself. I have never met the people behind it, but I bet they like coffee, or beer.

$ doas relayd -n            
/etc/relayd.conf:37: cannot load keypair ewintr.nl for relay www4tls
/etc/relayd.conf:45: cannot load keypair ewintr.nl for relay www6tls
no actions, nothing to do

OpenBSD, there it is again. Remembering my earlier experience with the old laptop, I figured I needed to give it another try. After all, besides security, enthusiasts always keep raving about the documentation and the focus on creating a simple and coherent system. Perhaps I gave up on it too soon. Surely, I should be capable of understanding that? I decided I should have another go.

The process turned out to be more difficult than expected. I got there in the end. The web page you are reading right now was served to you by OpenBSD's httpd. Open the Developer Tools in your browser to verify. But the road to get there was longer than I anticipated. And the weird thing is, looking back, I cannot tell you why it took more effort, other than by the analogy of adapting to round pegs.

I would skim the documentation for the command to run, or the configuration to enter, only to get error messages in response. Then a period of frantic searching and trial and error would follow. Until I finally got it right. Once it worked, I reread the original documentation and saw that the answers were right there all along. Crystal clear. And yet somehow it did not register the first time. This happened on multiple occasions.

$ doas rcctl restart httpd
httpd(ok)
httpd(ok)
$ doas rcctl restart relayd  
relayd(ok)
relayd(ok)

Originally I planned to make this post a simple recipe. Just follow these steps, and you have your website running in no time. But now I know it won't work like that. At least not for everyone. Instead, I'll just give you the list of resources I used and the first version of the configuration files that worked. The ingredient you need to add is some time and effort.

I think it is worth it, though. It all feels rock solid. Technology that is simple and boring in the best way possible. Something you can depend on.

Let it all sink in for a while. Give it some time. The water is nice, once you get used to it. Beautiful, round pufferfish swim there. They're fun to watch.

Resources

All these pages discuss the same few topics, but from a slightly different perspective. In no particular order:

Configuration files

This is what I ended up with. Remember, I am a newbie. Most certainly there are better ways to do this:

/etc/httpd.conf

types {
  include "/usr/share/misc/mime.types"
}

# https variant, behind relayd
server "ewintr.nl" {
  listen on 127.0.0.1 port 8080
  root "/htdocs/ewintr.nl" # this is /var/www/htdocs/ewintr.nl on the file system because httpd runs in a jail
  log style forwarded
  gzip-static

  # RSS and linklog redirects
  location match "/linklog/%d*/links%-([%d%-]*)" {
    block return 302 "https://ewintr.nl/linklog/#links-%1"
  }
  location "/feed/" {
    block return 302 "https://ewintr.nl/atom.xml"
  }

  # Older redirects
  location "/quick-go-test-clycle-with-reflex" { 
    block return 301 "https://ewintr.nl/posts/2020/quick-go-test-cycle-with-reflex/" 
  }
  # (skipped another 30+ of those for brevity) 
}
  
# http variant, directly
server "ewintr.nl" {
  listen on * port 80
  location "/.well-known/acme-challenge/*" {
    root "/acme"
    request strip 2
  }

  location "/*" {
    block return 301 "https://ewintr.nl$REQUEST_URI"
  }
}

# https variant, behind relayd
sserver "vrijkorteverhalen.nl" {
  listen on 127.0.0.1 port 8080
  root "/htdocs/vrijkorteverhalen.nl"
  log style forwarded
  gzip-static
}

# http variant, directly
server "vrijkorteverhalen.nl" {
  listen on * port 80
  location "/.well-known/acme-challenge/*" {
    root "/acme"
    request strip 2
  }
  location "/*" {
    block return 301 "https://vrijkorteverhalen.nl$REQUEST_URI"
  }
}

/etc/relayd.conf

ext_addr4="46.23.93.208"
ext_addr6="2a03:6000:93f4:605::208"
local="127.0.0.1"

table <webhosts> { $local }

http protocol https {
  match request path "/*.woff2" tag "CACHEYEAR"
  match response tagged "CACHEYEAR" header set "Cache-Control" value "public, max-age=31536000"
  match request path "/*.(css|png|jpg|jpeg)" tag "CACHEMONTH"
  match response tagged "CACHEMONTH" header set "Cache-Control" value "public, max-age=2628000"

  match request header set "X-Forwarded-For" value "$REMOTE_ADDR"
  match request header set "X-Forwarded-Port" value "$REMOTE_PORT"
  
  tls keypair "ewintr.nl"
  tls keypair "vrijkorteverhalen.nl"
}

relay www4tls {
  listen on $ext_addr4 port 443 tls
  protocol https

  forward to <webhosts> port 8080
} 

relay www6tls {
  listen on $ext_addr6 port 443 tls
  protocol https

  forward to <webhosts> port 8080
}

/etc/acme-client.conf

# Copy the example from /etc/examples/acme-client.conf and adapt the domain section:

domain ewintr.nl {
	#alternative names { secure.example.com }
	domain key "/etc/ssl/private/ewintr.nl.key"
	domain full chain certificate "/etc/ssl/ewintr.nl.fullchain.pem"
	# Test with the staging server to avoid aggressive rate-limiting.
	#sign with letsencrypt-staging
	sign with letsencrypt
}

domain vrijkorteverhalen.nl {
	#alternative names { secure.example.com }
	domain key "/etc/ssl/private/vrijkorteverhalen.nl.key"
	domain full chain certificate "/etc/ssl/vrijkorteverhalen.nl.fullchain.pem"
	# Test with the staging server to avoid aggressive rate-limiting.
	#sign with letsencrypt-staging
	sign with letsencrypt
}

Note that these are not the file paths relayd can discover automatically. I needed to add these symlinks:

# in /etc/ssl
ewintr.nl.crt -> ewintr.nl.fullchain.pem
vrijkorteverhalen.nl.crt -> vrijkorteverhalen.nl.fullchain.pem

Makefile

In the website project:

build:
	zola build

gzip:
	find public/ -type f \( -iname "*.html" -o -iname "*.txt" -o -iname "*.css" -o -iname "*.xml" \) -exec gzip -f -9k {} +

stage-drafts:
	zola serve --drafts

stage:
	zola serve

deploy: build gzip
	rsync -a public/ ewintr.nl:/var/www/htdocs/ewintr.nl/