KVM Virtual Machine Backups with Restic and guestfish

#

Moving from ESXi to KVM

I used to use VMware ESXi on an HP Microserver for virtualization along with Veeam for backups. More recently I built a new host for my virtual machines and decided to run them under the Linux KVM with qemu, libvirt, etc.

Veeam is a great backup solution for virtual machines but it does need to run inside a Windows VM and only supports KVM guests using an agent. I wanted an open source backup solution for my KVM host with the following features:

  • Live backups
  • Only changed disk image blocks backed-up
  • Ability to restore a complete VM with disk images and machine configuration
  • Restoration of individual files from within disk images

Backup running VMs with libvirt and restic

Live backup of VMs is easy with libvirt external snapshots and active block commit. This creates a snapshot file to collect new disk blocks for the running machine while the original disk image remains consistent during the backup. After the backup is finished the original image and snapshot are merged and the snapshot deleted.

Here is an example script that backs up my nagios VM:

#!/bin/bash

#Dump the machine config XML
virsh dumpxml nagios > /mnt/data1/virt-pool/nagios/nagios.xml

#Create the external snapshot
virsh snapshot-create-as --domain nagios nagios-state \
    --diskspec vda,file=/mnt/data1/virt-pool/nagios/nagios-state.qcow2 \
    --disk-only --atomic --quiesce --no-metadata

#Backup the disk image and VM configuration
restic backup --quiet --tag vmbackup /mnt/data1/virt-pool/nagios/nagios-sda \
    /mnt/data1/virt-pool/nagios/nagios.xml

#Merge the snapshot
virsh blockcommit nagios vda --active --verbose --pivot

#Remove the snapshot
rm /mnt/data1/virt-pool/nagios/nagios-state.qcow2

Restic will only backup any blocks, or ‘chunks’, that have changed since the last backup.

Restore a disk image using restic

$ restic restore latest --include /original/path-to-image/backlup --target /location-to-restore-to

Standard restic file restore from the latest snapshot. To restore from an earlier snapshot, list the snapshots and choose a hash instead of latest.

Once the image is restored it can mounted with guestfish and files copied out.

This is useful, but large disk images will be slow to restore.

Restore files using guestfish without restoring the disk image

Restic allows us to mount the backup repository to a local directory using fuse, so a restore is not needed to access a disk image:

$ restic mount /mnt/archive/restore &

Next, use guestfish to:

  • mount the disk image from the restic snapshot. It is important to mount the image read-only in guestfish, otherwise the VM created in the next step will crash. You would want your restic repository on local or near-line storage. I don’t think mounting your B2 or S3 bucket based repo would work too well!
  • run a temporary VM
  • list available file systems
  • mount the filesystem we want to restore from
  • copy a file or directory out of the mounted image to a local directory.
$ guestfish

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> add-ro /mnt/archive/restore/snapshots/latest/mnt/data1/virt-pool/attalus/data
><fs> run
><fs> list-filesystems
/dev/data/data1: ext4
><fs> mount /dev/data/data1 /
><fs> copy-out /home/steve/Work/ABC/Quotation.pdf .
><fs> exit

Return to the restic mount process:

$ fg

…then ctrl-c to stop the fuse mount.

Archive

2021