Backing up a running virtual machine on Ubuntu 15.10

published Apr 08, 2016 02:45   by admin ( last modified Dec 31, 2016 09:42 )

 

Update 2016-05-09:

Folding back a snapshot in kvm works in Ubuntu 16.10! The whole chain as described below now works. I have not checked if my 16.10 is running on the hand installed debs or if they've been superceded by 16.10 versions.

Update 2016-04-13:

It seems that the last step, folding back a snapshot in kvm, does not work in Ubuntu 15.10 and indeed has a lot of problems on other systems too. Folding back:

virsh blockcommit guestName vda --active --pivot --wait

fails with:

error: internal error: unable to execute QEMU command 'block-commit': Device 'drive-virtio-disk0' is busy: block device is in use by block job: commit

It works if you stop the guest, but that kind of defeats the purpose since it is not backing up a running machine anymore. One may try to use RedHat instead and see if patches there have been quicker, since the Qemu/Kvm people seem to be aware of the problems.

Summary 2016-04-08: KVM can be used and can do this (but last step does still not work, see above), but the configuration and packages in Ubuntu 15.10 are buggy, so you have to replace stuff, and disable stuff:

 

Install KVM, disable AppArmor,  install by hand updated versions of libvirt-bin (libvirt-bin_1.2.16-2ubuntu11.15.10.4_amd64.deb) and libvirt0 (libvirt0_1.2.16-2ubuntu11.15.10.4_amd64.deb), install qemu guest agent in the guest, open up a communications channel between host and guest by creating a channel device called org.qemu.guest_agent_0 in the configuration of the guest.
 
Read on for more info on how to do  this (you can skip to the KVM heading if you don't want the background).

Backing up a running virtual machine on Ubuntu 15.10 turned out to be not so simple. A lot of the information that Google return is outdated, giving recommendations pertaining to 8 year old Ubuntu systems or claiming that KVM cannot run GUI systems.

Virtualbox

I started out with Virtualbox, and Virtualbox has something called snapshots. As far as I understand, when you take a snapshot, the main vdi file becomes read-only and all changes go into a new snapshot file. Several snapshots can be made and they can reference each other and branch, and be folded back into the main vdi file and into each other. I followed this guide:

Arne's Blog: Backup of running VirtualBox machines:

!/bin/bash
VBOXMANAGE="/usr/bin/VBoxManage -q"

if [ $# != 1 ]
then
    echo "Usage: $0 VBoxName"
    exit
fi

echo "Renaming old snapshot..."
$VBOXMANAGE snapshot "$1" edit previous --name deleteme
echo "Renaming current snapshot..."
$VBOXMANAGE snapshot "$1" edit current --name previous
echo "Taking new snapshot..."
$VBOXMANAGE snapshot "$1" take current
echo "Deleting old snapshot..."
$VBOXMANAGE snapshot "$1" delete deleteme
 

But I ended up with errors and inconsistent snapshots that were corrupt according to Virtualbox. Which is probably my fault somehow. Digging deeper I found a blog post referencing the same script, OSC — Backing up Virtual Machines in VirtualBox  and that guy writes that he has gone back to shutting down the virtual machines and then back them up:

Sadly, for me personally, live VirtualBox snapshots haven’t been a terribly robust backup strategy. I’ve unfortunately seen several snapshots fail. Or, worse, had VirtualBox crash while a snapshot was taking place.

Btrfs

So next solution, why not put the virtual machine on a snapshotting file system, that is guaranteed to copy the vdi file in one fell swoop? This may not be enough, in the sense that there might be state stored in other files or in RAM. But it is worth a shot. One option could be btrfs, which I have been experimenting with and which has a very smooth snapshotting mechanism: You can snapshot any file or director on a btrfs volume. But timely enough, this was posted on Reddit's /r/linuxadmin:

BTRFS for VM images? : linuxadmin

which thread says things like:

Yes. I do not recommend this. I spent more time maintaining the one client I had running VMs on a BTRFS store than I did the fifty or so others I had running VMs on ZFS stores, for roughly a year. The replication is unreliable, the performance is incredibly hit-or-miss

The poster lauds ZFS though. ZFS is supported well in Ubuntu 15.10, but the ZFS snapshotting is not as simple as btrfs. Just using LVM should also be an option.

VM options

So, what about changing the virtual machine from Virtualbox to something else? Three options are:

  • VmWare
  • Xen
  • KVM

VmWare costs money for the snapshotting version. I did however vaguely remember that Linode switched from one VM technology to another recently. A Google search turned up that they switched from Xen to KVM.

KVM

Ok KVM it is. What is available snapshot wise there then?

Well firstly there is a way of freezing the state of a VM, backup and then unfreeze it again Backup of running KVM qcow2 VPS - Server Fault.. Secondly it seems to have the same ability to do snapshots with mutable snapshot files as VirtualBox, but there is more documentation giving examples of what I think I want.

This is the best I have found so far [libvirt-users] Backup a VM (using live external snapshot and blockcommit:

# Create snapshot
virsh snapshot-create-as --domain $VM_NAME snap --diskspec
vda,file=$VM_DIR/"$VM_NAME"-snap.qcow2 --disk-only --atomic
--no-metadata --quiesce

# Copy frozen backing file
cp $VM_DIR/"$VM_NAME".qcow2 $SNAP_FILEPATH

# Blockcommit snapshot back into backing file
virsh blockcommit $VM_NAME vda --active --pivot

# Remove snapshot file
rm $VM_DIR/"$VM_NAME"-snap.qcow2

# Variables should be self-explanatory: 
# - VM_DIR is the directory where the VM are stored
# - VM_NAME is the name of the VM, and its qcow2 file is called
# VM_NAME.qcow2
# - SNAP_FILEPATH is the full path (including name) where the backup
# should be created

This seems to be essentially the same KVM – Live backups with qcow2 | Gonzalo Marcote | Open source, open mind.

Some more info here too: Live-disk-backup-with-active-blockcommit - Libvirt Wiki

The above script will only work if the following criteria are fulfilled:

Download links for amd64 are (to the best of my understanding and what I used):

Install with sudo dpkg -i libvirt*

(assuming you only have those two files in the directory)

Unfortunately, without tuning or disabling AppArmor there is currently no way to flatten the snapshot back into the image, as of the last comment currently at Bug #1517539 “Libvirt KVM can not create snapshot (with qemu-gue...” : Bugs : libvirt package : Ubuntu:

"Strictly speaking, the virsh command that prompted this ticket has been fixed. So, I can now successfully create a live snapshot using the proposed packages. Unfortunately, I have no way to flatten the snapshot back into the base image now."

However after having disabled AppArmor, it works, it seems.

Moving from VirtualBox to KVM

Was disarmingly simple. As long as you follow the up-to-date guides and use the virt-manager GUI, you can actually just use the vdi file directly in KVM. However then you will not get the snapshotting abilities. For that you need the qcow2 disk format. Which you can convert to directly from vdi like this:

qemu-img convert -f vdi -O qcow2 vm.vdi vm.qcow2