From Ceph wiki
The qemu-rbd module was originally developed by Christian Brunner. It allows striping a VM block device over objects stored in the Ceph distributed object store. This gives you shared block storage to facilitate VM migration between hosts and fancy things like that.
See also rbd.
The prerequisites for building QEMU with rbd support are both librados and librbd (librados-dev and librbd-dev packages in Debian). It goes something like this:
$ git clone git://git.qemu.org/qemu.git $ cd qemu $ ./configure --enable-rbd $ make; make install
Creating rbd image
Using the rbd as described in Rbd is one way to create images.
There are two other options that use the qemu tools:
- Creating a new image:
This will create a 10GB data image named 'foo' inside the RADOS data pool.
$ qemu-img create -f rbd rbd:data/foo 10G
- Converting an old image:
This will create an image named 'lenny' in the data pool that clones our old qcow2 image:
$ qemu-img convert -f qcow2 -O rbd /data/debian_lenny_amd64_small.qcow2 rbd:data/lenny
Now we can run kvm using this image for booting:
$ qemu-system-x86_64 --drive format=rbd,file=rbd:data/lenny
We can also migrate the running vm between two hosts that can access the RADOS cluster: On the source host we run the previous command and on the destination host (called dest_host) we'd run:
$ qemu-system-x86_64 --drive format=rbd,file=rbd:data/lenny -incoming tcp:0:4444
Now we can migrate. On the source host we'll enter the qemu console (Ctrl+Alt+2) and run:
% migrate -d 0:dest_host:4444
libvirt is a library for managing virtual machines. Using version 0.8.7 or later, you can define network disks which allow you to use rbd via libvirt.
An example disk configuration would look like:
<disk type='network' device='disk'> <source protocol='rbd' name='poolname/imagename'> <host name='mon.example.org' port='6789'/> <host name='mon.example.org' port='6790'/> <host name='mon.example.org' port='6791'/> </source> <target dev='vda' bus='virtio'/> </disk>
The hosts in this configuration are the ceph monitors. If these are set in /etc/ceph/ceph.conf, they may be omitted from the libvirt disk configuration. Note: With Ubuntu apparmor blocks access to /etc/ceph/ceph.conf by default causing a permission denied error, the quick fix is to change /etc/apparmor.d/abstractions/libvirt-qemu to allow access.
While QEMU-RBD supports snapshots, you can't simply snapshot running Virtual Machines via rbd nor qemu-img, when doing so, the Qemu process won't be notified, thus not knowing about the snapshot.
Upstream libvirt supports snapshotting RBD devices since 0.9.12. The version included in Ubuntu 12.04 also includes this capability.
To snapshot a running virtual machine you could use libvirt's virsh snapshot-create method which notifies the running Qemu process.
root@client01:~# virsh snapshot-create alpha Domain snapshot 1283504027 created root@client01:~#
Now we want to list the available snapshots:
root@client01:~# virsh snapshot-list alpha Name Creation Time State --------------------------------------------------- 1283504027 2010-09-03 10:53:47 +0200 running root@client01:~#
Starting with Ceph versions 0.46 caching has been added, it can be enabled by setting rbd_cache=1 as in the following example.
<disk type='network' device='disk'> <source protocol='rbd' name='poolname/imagename:rbd_cache=1'/> <driver name='qemu' type='rbd'/> <target dev='vda' bus='virtio'/> </disk>
In 0.47 you can change the cache size by setting rbd_cache_size, and set it to write-through mode by setting rbd_cache_max_dirty=0, i.e.:
<disk type='network' device='disk'> <source protocol='rbd' name='poolname/imagename:rbd_cache=1:rbd_cache_size=67108864:rbd_cache_max_dirty=0'/> <driver name='qemu' type='rbd'/> <target dev='vda' bus='virtio'/> </disk>
For more details see http://article.gmane.org/gmane.comp.file-systems.ceph.devel/6402