Securely deploying zacbrown.org

When I transitioned zacbrown.org from a Ghost blog to a static set of HTML files generated by zsitegen, I needed a way to publish new content. I decided that each new commit to the git repo should result in a deploy of the website. In the case of zacbrown.org, it’s just a set of static HTML, CSS, images, and font files. That makes the site a prime candidate for using rsync over SSH to deploy.

My preferred source control hosting site is sourcehut and it handily provides a build service. Builds are described in a relatively simple build definition. Here’s my .builds.yml:

arch: amd64
image: openbsd/latest
packages:
  - curl
  - git
  - go
  - rsync--
environment:
  deploy: deploy@zacbrown.org
repositories: {}
secrets:
  - e65a4d24-852d-4939-bb84-485a55d930a9
  - b5fe5cdb-0976-4494-bfdd-b7ac161d8c65
shell: false
sources:
  - https://git.sr.ht/~zacbrown/zacbrown.org-site
tasks:
  - setup: |-
      go get git.sr.ht/~zacbrown/zsitegen
  - build: |-
      cd zacbrown.org-site
      $(go env GOPATH)/bin/zsitegen -in site/ -out out/
  - deploy: |-
      cd zacbrown.org-site
      sshopts="ssh -p 8738 -i ~/.ssh/b5fe5cdb-0976-4494-bfdd-b7ac161d8c65 -o StrictHostKeyChecking=no";
      rsync -avzh --progress -e "$sshopts" out/ $deploy:/home/deploy/htdocs/;
triggers: []

Initially, I created the deploy user and generated a unique SSH key for deployment. This key is only ever used to deploy from the infrastructure to that server. However, it’s still a full blown user even if locked down with minimal permissions.

An even safer method would be to make use of an OpenBSD chroot for the deploy user to ensure it has the bare minimum of capabilities needed to complete a deployment. There’s four things we’ll need to do:

Setting up the chroot

Most of this content is based on this very helpful post: https://mwl.io/archives/842

Assuming we already have a deploy user, we’ll create the standard devices that a user would need access to for a valid tty:

# cd ~deploy
# mkdir dev
# cd dev
# /dev/MAKEDEV std
# ls
arandom klog ksyms null stdin tty zero
console kmem mem stderr stdout xf86
# rm console klog kmem ksyms mem xf86
# ls
arandom null stderr stdin stdout tty zero

We’ll also need a shell to spawn rsync from:

# cd ~deploy
# mkdir bin
# cd bin/
# file /bin/ksh
/bin/ksh: ELF 64-bit LSB shared object, x86-64, version 1
# cp /bin/ksh .

While the output from file notes that this is a shared object, we can use ldd to make sure it doesn’t actually rely on any shared libraries:

# ldd /bin/ksh
/bin/ksh:
	Start            End              Type  Open Ref GrpRef Name
	00000bbf47d1f000 00000bbf47dc5000 dlib  1    0   0      /bin/ksh

We’re using the deploy user as an rsync target so we need to make sure rsync is available. Let’s check what rsync’s dependencies are:

# ldd /usr/local/bin/rsync
/usr/local/bin/rsync:
	Start            End              Type  Open Ref GrpRef Name
	0000029e43f1d000 0000029e43fa7000 exe   1    0   0      /usr/local/bin/rsync
	000002a0e7fc0000 000002a0e81f9000 rlib  0    1   0      /usr/lib/libcrypto.so.46.1
	000002a0fde86000 000002a0fdf7a000 rlib  0    1   0      /usr/lib/libc.so.96.0
	000002a141766000 000002a141766000 ld.so 0    1   0      /usr/libexec/ld.so

Alright, so we need a copy of each of those filenames within the /home/deploy/ chroot:

# mkdir -p /home/deploy/home/deploy/usr/local/bin/
# mkdir -p /home/deploy/home/deploy/usr/lib/
# mkdir -p /home/deploy/home/deploy/usr/libexec/
# cp /usr/local/bin/rsync /home/deploy/home/deploy/usr/local/bin/
# cp /usr/lib/libcrypto.so.46.1 /home/deploy/home/deploy/usr/lib/
# cp /usr/lib/libc.so.96.0 /home/deploy/home/deploy/usr/lib/
# cp /usr/libexec/ld.so /home/deploy/home/deploy/usr/libexec/

A chrooted user shouldn’t have write access to their directory, so we’ll reset the ownership to root:wheel. We’ll then change the ownership of the sub home/deploy directory we’re using for the chroot to deploy:deploy:

# chown root:wheel /home/deploy/
# mkdir -p /home/deploy/home/deploy
# chown deploy:deploy home/deploy

If we login, we’ll see something like the following:

# ssh deploy@chroothost
ksh: No controlling tty (open /dev/tty: Device not configured)
ksh: warning: won't have full job control
$

Just to be sure everything works as expected, let’s launch rsync and make sure there’s no segfaults or other issues:

$ rsync
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, no ACLs, no xattrs, optional protect-args,
    no iconv, symtimes, no prealloc, stop-at, no crtimes
Optimizations:
    no SIMD, asm, openssl-crypto
Checksum list:
    md5 md4 none
Compress list:
    zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

...<TRUNCATED>...

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See https://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at main.c(1732) [client=3.2.3]

That looks like it launched. Yippee!

Important: If you’re using SSH keys, make sure you move the /home/deploy/.ssh folder for the deploy user to /home/deploy/home/deploy/.ssh and change the ownership to deploy:deploy.

Creating a local NFS share for /var/www/htdocs/www.zacbrown.org

While Linux has the mount --bind option and FreeBSD/NetBSD have the mount_nullfs utility, OpenBSD doesn’t have an equivalent. OpenBSD 3.7 was the last version to have mount_nullfs. From what I’ve read, the code wasn’t in great shape and didn’t offer substantial improvements over just using local NFS mounts.

First, we’ll need to setup our NFS exports:

echo '/var/www/htdocs/www.zacbrown.org/ -alldirs -mapall=root' > /etc/exports

To setup a local NFS mount, we’ll need to enable portmap, mountd, and nfsd:

rcctl start portmap mountd nfsd

Mounting /var/www/htdocs/www.zacbrown.org/ inside the chroot

Now, let’s make the directory to mount the htdocs to:

# mkdir -p /home/deploy/home/deploy/htdocs
# chown -R deploy:deploy /home/deploy/homde/deploy/

We can test the mount on the local machine with:

mount 127.0.0.1:/var/www/htdocs/www.zacbrown.org/ /some/directory

Finally, we need to make sure we mount the directory on boot using /etc/fstab. Open the file to edit and add the following line:

127.0.0.1:/var/www/htdocs/www.zacbrown.org/ /home/deploy/home/deploy/htdocs/ nfs rw,suid,sync 0 0

On boot, that will ensure that it gets mounted inside the chroot at /home/deploy/htdocs/.

Updating the OpenSSH to put deploy inside a chroot

This part is the easiest - you’ll need to open /etc/ssh/sshd_config and at the bottom, add the following:

Match User deploy
ChrootDirectory /home/%u/
AuthorizedKeysFile /home/%u/home/%u/.ssh/authorized_keys

Then restart sshd:

rcctl restart sshd

The deploy user can now be used as a safe chrooted target for deploying the static website assets to:

rsync -avzh --progress -e ssh files/ deploy@chroothost:/home/deploy/htdocs/;

Finishing Up

Now we’ve got an even safer way to deploy using rsync to our production server. It was already pretty secure with a unique strong SSH key but this ensures that even if the key were compromised, the user would only be able to rsync files into the chroot.

This post was made possible by:



Posted on 2021-04-09