Real time constant backups with ZFS+Zrepl

This is a guide to making your home network backups seamless, secure, and awesome. It’s comparable in many ways to Apple Time Machine.

Prerequisites:
A machine in the cloud with lots of disk space
Mine comes from zfs.rent, where I sent some physical hard drives and rent a small machine with them attached.
CA Infrastructure.
Creating your own private CA is a whole topic of it’s own, one I hope to make simpler. For the short term, creating a private CA just for Zrepl is the best option. Create the key with cfssl genkey -initca zrepl-ca.json | cfssljson -bare zrepl-ca

# /etc/zrepl/zrepl-ca.json - On your backup host

{
  "CN": "Personal Zrepl Root CA",
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "Thomas Hahn",
      "OU": "Zrepl",
      "ST": "Washington"
    }
  ]
}


Setup
Create certificates for your hosts. They should use the hostname as the CN, but also have appropriate Subject Alternative Names. I recommend CFSSL to do so.

Example contents of your cfssl csr json:

# /etc/zrepl/zrepl.json
{
  "CN": "timemachine",
  "hosts": [
    "timemachine.internal",
    "timemachine.zfs.rent",
    "timemachine.gauntletwizard.net"
  ],
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "Thomas Hahn",
      "OU": "Zrepl",
      "ST": "Washington"
    }
  ]
}
mkdir /etc/zrepl
cfssl genkey zrepl.json | cfssljson -bare zrepl-$HOSTNAME

You should now have a zrepl-key.pem and a zrepl.csr in your /etc/zrepl folder, on each of your machines. Copy all of the .csr files to your CA for signing. Don’t touch the -key.pem files! These are your private keys, and need to be secret. Once you have the csrs on your CA’s machine, sign them. Sign them with:

host=HOSTNAME # Replace HOSTNAME with the name of each host
cfssl sign -ca ca.pem -ca-key ca-key.pem "zrepl-${host}.csr" | cfssljson -bare ${host}

Signing them with the above will leave you with a set of .pem files, named host.pem. You should verify that they signed correctly: openssl verify -CAfile ca.pem host.pem. It should print ‘host.pem: ok’. Copy these files back to their respective hosts, and rename them on that host to zrepl.crt. Also copy the ca.pem file to each host as /etc/zrepl/ca.pem

Next, set up the backup as a sink:

global:
  logging:
    # use syslog instead of stdout because it makes journald happy
    - type: syslog
      format: human
      level: warn
  monitoring:
    - type: prometheus
      listen: ':9811'


jobs:
  - name: backups
    type: sink
    root_fs: tank/backups
    recv:
      placeholder:
        encryption: inherit
    serve:
      type: tls
      listen: ":8826"
      ca: "/etc/zrepl/ca.pem"
      cert: "/etc/zrepl/zrepl.crt"
      key: "/etc/zrepl/zrepl-key.pem"
      client_cns:
        - "desktop"
        - "laptop"
        - "timemachine"

zrepl uses the CN field for disambiguation. Add each host you signed above to the client_cns section in zrepl.yaml. zrepl will create a new zfs filesystem under your root_fs for each of these client cns upon their first backup, i..e. tank/backups/desktop and so on.

Next, configure your machines as sources:

global:
  logging:
    # use syslog instead of stdout because it makes journald happy
    - type: syslog
      format: human
      level: warn
  monitoring:
    - type: prometheus
      listen: ':9811'


jobs:
  - name: desktop
    type: push
    filesystems:
      "desktop/home<": true
    send:
      encrypted: false
    connect:
      type: tls
      address: "timemachine:8826"
      ca: /etc/zrepl/ca.pem
      cert: /etc/zrepl/zrepl.crt
      key:  /etc/zrepl/zrepl-key.pem
      server_cn: "timemachine"

    snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 5m
    pruning:
      keep_sender:
        - type: not_replicated
        - type: regex
          regex: ".*"
      keep_receiver:
      - type: grid
        grid: 1x1h(keep=all) | 24x1h | 30x1d | 6x30d
        regex: "^zrepl_"