OpenBSD, Services, and Signals

Introduction

I’ve been rebuilding out this server (zacbrown.org) with OpenBSD over the last few weeks. As part of that, I’ve had need to write a couple of services for the operating system for things like Prometheus exporters and so on.

I had a pretty decent understanding of services on OpenBSD, but I’d never written a program to run as a service before. I wanted to create an HTTP API server for something and realized that I didn’t know what POSIX signals my program would need to handle during its lifetime.

While I’ve written daemons before, I’d never written one to run on OpenBSD. The following is a brief account of that journey.

OpenBSD’s rcctl, rc.d, and rc.conf.local

Services/daemons on OpenBSD are managed by rc. There are three components we care about for writing our own services:

I’m not going to get into the specifics of how each of those work but we’ll interact with those components directly or indirectly as we learn.

Creating a new service

To make it easier to understand what signals our service is processing, I’ve created a very simple Golang binary called signalfun. You can view the source code for the program here: git.sr.ht/~zacbrown/signal-fun/tree/main/item/main.go

Briefly, the signalfun binary listens to all POSIX signals sent to it and prints them out. If it is one of the following signals, it will exit:

If it receives SIGHUP it will note that it should reload config files and reopen log files. signalfun won’t exit in this case but it does treat it as special.

Although I mentioned SIGINT and SIGKILL above, they aren’t relevant to our daemon. We typically only see SIGINT when we’ve launched a program as the foreground program from the shell/tty and type CTRL^C. SIGKILL is one we’d send explicitly using the kill -s SIGKILL command (aka kill -9).

rc.d service control script

In order to register the signalfun binary as an OpenBSD service, we need an rc.d control script. A very simple version of a control script looks like the following:

#!/bin/ksh

daemon="/home/zbrown/signal-fun/bin/signalfun"

. /etc/rc.d/rc.subr

rc_start() {
    ${rcexec} "${daemon} ${daemon_flags} 2>&1 | logger -t signalfun &"
}

rc_cmd $1

Basically, the script includes default subroutines for rc.d control scripts. Then it defines a function called rc_start which launches the daemon. Notably, the latter part of the command pipes stdout/stderr for the daemon to the logger binary which logs output to /var/log/messages.

In order to install our service, we need to make sure our binary lives at the right location. In the case of my server, it’s at /home/zbrown/signal-fun/bin/signalfun. Then we need to copy the rc.d control script to /etc/rc.d/signalfun.

To enable and start the signalfun service, run:

root@fugu ~# rcctl enable signalfun
root@fugu ~# rcctl start signalfun
signalfun(ok)
root@fugu ~#

To see the logs of the signalfun binary, run:

root@fugu ~# tail -f -n 50 /var/log/messages | grep signalfun
Mar 28 19:43:31 fugu signalfun: my PID is 64085

How rcctl interacts with your new service

rcctl stop signalfun

Now let’s try calling stop on the service to see what signal we get:

root@fugu ~# rcctl stop signalfun
signalfun(ok)
root@fugu ~# tail -n 20 /var/log/messages | grep signalfun
Mar 28 19:43:31 fugu signalfun: my PID is 64085
Mar 28 19:46:01 fugu signalfun: signal received: terminated
Mar 28 19:46:01 fugu signalfun: breaking for SIGTERM

rcctl start signalfun; rcctl reload signalfun

How about if we start and then reload it:

zbrown@fugu ~/signal-fun (main)> sudo rcctl start signalfun
signalfun(ok)
zbrown@fugu ~/signal-fun (main)> sudo rcctl reload signalfun
signalfun(ok)
zbrown@fugu ~/signal-fun (main)> tail -n 3 /var/log/messages | grep signalfun
Mar 28 19:59:03 fugu signalfun: reloading and continuing for SIGHUP
Mar 28 19:59:03 fugu signalfun: signal received: urgent I/O condition
Mar 28 19:59:03 fugu signalfun: continuing despite urgent I/O condition

rcctl restart signalfun

Finally, let’s try restart:

zbrown@fugu ~/signal-fun (main)> sudo rcctl restart signalfun
signalfun(ok)
signalfun(ok)
zbrown@fugu ~/signal-fun (main)> tail -n 3 /var/log/messages | grep signalfun
Mar 28 20:04:27 fugu signalfun: signal received: terminated
Mar 28 20:04:27 fugu signalfun: breaking for SIGTERM
Mar 28 20:04:28 fugu signalfun: my PID is 77682
zbrown@fugu ~/signal-fun (main)>

What if we ignore SIGTERM?

Finally, what happens if we ask signalfun to just ignore SIGTERM?

zbrown@fugu ~/signal-fun (main)> sudo rcctl set signalfun flags -ignoreterm
zbrown@fugu ~/signal-fun (main)> sudo rcctl start signalfun
signalfun(ok)
zbrown@fugu ~/signal-fun (main)> sudo rcctl stop signalfun
signalfun(failed)
zbrown@fugu ~/signal-fun (main) [1]> tail -n 5 /var/log/messages | grep signalfun
Mar 28 20:18:57 fugu signalfun: my PID is 98919
Mar 28 20:19:07 fugu signalfun: signal received: terminated
Mar 28 20:19:07 fugu signalfun: lol ignoring SIGTERM
Mar 28 20:19:07 fugu signalfun: signal received: urgent I/O condition
Mar 28 20:19:07 fugu signalfun: continuing despite urgent I/O condition
zbrown@fugu ~/signal-fun (main)> ps aux | grep signalfun
root     90736  0.0  0.0   356  1312 p0  I+p     7:44PM    0:00.01 grep signalfun
root     98919  0.0  0.0 103432  1568 p1  I       8:18PM    0:00.03 /home/zbrown/signal-fun/bin/signalfun -ignoreterm
root     71399  0.0  0.0   232  1108 p1  Ip      8:18PM    0:00.02 logger -t signalfun
zbrown   42699  0.0  0.0   288  1284 p1  S+p     8:20PM    0:00.01 grep signalfun

Well, that’s not really surprising given how OpenBSD generally behaves. rcctl sends a SIGTERM to signalfun and since we just ignore it, eventually it times out. The process remains running and we’ll have to kill it with SIGINT or SIGKILL:

zbrown@fugu ~/signal-fun (main)> ps aux | grep signalfun
root     98919  0.0  0.0 103432  1568 p1  I       8:18PM    0:00.04 /home/zbrown/signal-fun/bin/signalfun -ignoreterm
root     71399  0.0  0.0   232  1108 p1  Ip      8:18PM    0:00.02 logger -t signalfun
zbrown    1055  0.0  0.0   288  1280 p1  S+p     8:34PM    0:00.01 grep signalfun
zbrown@fugu ~/signal-fun (main)> sudo kill -s SIGINT 98919
zbrown@fugu ~/signal-fun (main)> ps aux | grep signalfun
zbrown   29388  0.0  0.0   284  1272 p1  S+p     8:34PM    0:00.01 grep signalfun
zbrown@fugu ~/signal-fun (main)>

In Summary…

So what have we learned today? Well, creating a service on OpenBSD is pretty simple. In order fo the service binary to behave properly, there’s two main signals that we need to be aware of:

Most of this was unsurprising to me except for what I learned about SIGHUP. I didn’t know what its original purpose was or what its modern use is.

PS…

You can find all the code I used for this blog post here: git.sr.ht/~zacbrown/signal-fun

Posted on 2021-03-28