We (my company) have been long-time contributors to Ceph (since version 0.17!), and will be using it in our own cloud product. Support for Ceph didn’t exist in CloudStack… So we built it!

I’m co-owner of a Dutch webhosting company called PCextreme B.V. My role as CTO is to do our Research & Development and that enables me to play with Ceph (a lot).

Quite some time ago we were convinced we wanted to use Ceph with RBD in our VPS product, but we weren’t sure how. Were we going to write our own cloud management software? OpenStack seemed like a good choice since it already had RBD integration, but while looking at OpenStack we came across CloudStack. I’m not going to do the OpenStack vs CloudStack discussion, but we decided CloudStack suited us better. It however lacked RBD support!

To make this integration work, a few things needed to be done:

• Add RBD storage pool support to libvirt (Since version 0.9.14)
• Update libvirt-java bindings (Secret handling and misc fixes)
• Make a few minor libvirt fixes (Small cephx fix)
• Add RBD support to CloudStack (Add RBD primary storage)

This work has been completed and merged, and will all be part of the new CloudStack 4.0 release, which is slated for the end of October. Between now and then, we’d like people to try it!

To get started, take a look at the related documentation. If you encounter any problems, feel free to ask for help on the Ceph or CloudStack mailing lists. Or join the #ceph (OFTC) or #cloudstack (Freenode) IRC channels, I’m idling there for most of the time.

So what can you do with Ceph? For those who like to read the ending first, the answer turns out to be “a blindingly awesome ton.” Thankfully that doesn’t spoil it for the rest of us, because it’s the details that make it fun. In an email discussion of these details, it was Inktank’s chief suit, Bryan Bogensberger, who managed to succinctly summarize many of the available options while still citing examples and supporting data. (How do you like that, a business guy who has a solid handle on the tech. How lucky are we!?) Without immediately overwhelming you with all the supporting details, his list was as follows:

• Enable the Public Cloud
• Enable the Private Cloud
• Support service providers replacing legacy storage
• Act as a replacement for HDFS
• Act as a replacement for enterprise storage
• Serve as a Lustre replacement
• Provide a platform for Application Development

I would actually add one more:

• Act as the basis for loads of academic research, development, and experimentation

The cool part about this is that a number of these categories already have early adopters that took one look at Ceph and decided to dive right in, building amazing things on top of it. The combination of Ceph as open source technology and Inktank’s seasoned enterprise veterans have allowed those responsible for Ceph to engage with two disparate communities at a very deep level. Open source enthusiasts have helped with edge cases, testing, patches, and even active development and support. Additionally, many businesses have provided their own external expertise as we build bridges from Ceph to other technologies, interesting problems to solve, as well as active development contributions. This combination is starting to allow Ceph as a technology to spread like wildfire, challenging many expensive alternatives just as cloud challenged traditional infrastructure.

Enabling the Public Cloud

Several service providers are already availing themselves of the insanely low comparative cost per gigabyte that Ceph allows, building object storage products to compete with incumbent offerings. Because of Ceph’s extensible nature, powerful integrations, and ease of management these service providers are finding themselves with a lot of room to grow, sidestepping the inherent limitations of proprietary technologies. Dreamhost is a prime example of this with their new DreamObjects offering. DreamObjects is aimed at being an inexpensive, object-based, cloud storage service that allows users to connect via Amazon S3 and OpenStack Swift compatible APIs. Ceph is giving them the ability to bring this new object store to market quickly, cheaply, and in a way that is extremely easy to scale.

Enabling the Private Cloud

Over the last few years many businesses have been moving to virtual infrastructure as a way to save both time and money. Unfortunately this trades the problem of managing the sprawl of physical racks for the complexity of maintaining the digital sprawl of your virtual infrastructure. IT managers are always looking for a good way to get the most bang for their buck, as well as scale resources across applications. Ceph is a platform with many capabilities, and not just a simple object store; many IT professionals are finding that it can solve many problems at once, extending its usefulness and their budget.

Inktank is already supporting a few companies who are developing their own cloud projects with a cost effective alternative for reliable, scalable cloud storage. Ceph’s modularity and auto-rebalancing of the cluster through intelligent object storage daemons (OSDs) are especially important in the private cloud. They allow a business to spin nodes up or down based on need without downtime or effects on performance. This goes a long way towards having a truly dynamic infrastructure that is so important in today’s virtualized data center. Additionally, there are a number of different ways you can access a Ceph cluster which allows a massively flexible, and self-service, approach for your application developers. Whether you are taking advantage of one of the native APIs or simply mounting your cluster with CephFS, once the cluster is available, application developers can interface with it in a way that makes the most sense for their individual project. This allows ops people to focus on ops and application developers to focus on building the best software possible.

A great example of what Ceph can do in a private cloud can be seen in Piston Cloud’s OpenStack product. Piston Cloud has created a tool that auto-configures a whole rack of servers as OpenStack nodes with Ceph providing the block storage. This offering installs in under 10 minutes and allows customers to evaluate all of OpenStack’s services without having to configure several different machines. Piston’s implementation also promises both easy deployment and seamless upgrade from a pilot deployment to a fully supported production environment.

Support service providers replacing legacy storage

Many enterprises are long overdue for upgrading their storage needs, due in no small part to the cost associated with doing so via existing technologies. Ceph has a leg up on traditional enterprise storage through being both open source and driven primarily by commodity hardware. Several companies have already come to the conclusion that while reliability and scalability can be enhanced, the real driving force is the long term effects on the bottom line. Ceph is a technology that can grow with a business, through the expert help of someone like Inktank or the internal expertise developed as with any other technology.

HDFS / Enterprise Storage / Lustre Replacement

Ceph actually originated as an alternative to Lustre that would provide better scalability and performance, and has offered a number of other interesting enhancements as well. One of the interesting parts about Ceph is that it has the ability to maintain multiple metadata server daemons, which makes file system access much more efficient. This particular approach is also what makes it an ideal replacement for the Hadoop Distributed File System (HDFS), especially when compared to HDFS’s single name-node architecture. Several folks are already experimenting with blending a Ceph back end with a Hadoop front end and we can’t wait to see the results.

In addition to the strictly technical advantages, Ceph’s open source nature and ability to run on commodity hardware also makes it a very attractive offering for Enterprise data storage, especially as it relates to the bottom line. The ability to deploy Ceph as a strictly software solution in an existing infrastructure allows businesses to remove the ongoing cost and difficulty of maintaining a separate appliance-based solution. This allows for many options when it comes to extensibility and flexibility, which are key to the long-term viability of an enterprise environment. The level of data ownership and control available to an integrated solution also mitigates a lot of risk when it comes to future migrations or data access requirements.

Application development

As mentioned earlier, Ceph is so much more than just an object store: it is a whole platform that can be used in a myriad of different ways. Two of the key ways we see developers engaging with Ceph are through the client library (directly via librados) and via our RADOS gateway (radosgw). Librados gives developers full control over Ceph’s object store via C, C++, Java, Python, Ruby, or PHP and offers the most advanced functionality of the four types of interfaces with RADOS. The massive horizontal scaling and fault tolerance of object stores are ideal for many of the big data operations that businesses are finding problematic from storing and loading virtual machine images to archival of video surveillance. Regardless of what interesting applications people decide to build though, librados gives you the best tools to build them.

The RADOS gateway provides a REST interface compatible with applications written for Swift or Amazon’s S3. This allows developers to take advantage of the Ceph platform without having to rewrite their applications. Developers can immediately realize the advantages offered by Ceph like the financial benefits of commodity hardware or the time savings of self-healing storage devices. Amazon has done a tremendous job of building and proving out the cloud model for application development; now anyone can build their own version of S3 and take advantage of, or improve upon, those benefits.

Research, development, and experimentation

Ceph’s unique characteristics and modular architecture also make it an ideal candidate as the subject of purely academic study. The most obvious research pathway is the “Controlled Replication Under Scalable Hashing” (CRUSH) algorithm. CRUSH can be described as follows:

“CRUSH works by describing the storage cluster in a hierarchy that reflects its physical organization. For example, let’s say each host has three disks, each rack has 30 hosts, and we have some number of racks. The result is a hierarchy of racks, hosts, and devices.”

The nature of this CRUSH algorithm allows you to pass three things in (the placement group, latest state of the cluster, and a crush map) and it can always calculate the location of your data object. CRUSH is both repeatable and fluid, since the cluster can change and CRUSH will always know how to adapt to the new layout. CRUSH is also fully configurable allowing you to specify things like how many times your data should be replicated and what kind of weighting should be applied. This allows us to ask a few potentially interesting questions, like “to what other types of problems could the CRUSH algorithm be applied?” or “how can we extend or refine the CRUSH algorithm?”

In any good academic study, it is important to have very controlled circumstances (to the extent possible). The modular nature of Ceph allows you to limit the number of variables in the equation by only running the parts of the platform that you require. If your particular case doesn’t require a RESTful interface, for instance, you have the ability to turn off an entire part of the platform (in this case, radosgw). This distinct separation of function allows you to scale out the pieces you need without any unnecessary overhead or interdependency. This is helpful for study and also useful in production environments!

Another application of pure study could be uses and extensions to the object store, RADOS (Reliable, Autonomic, Distributed Object Store). RADOS is comprised of two kinds of components: the monitors (ceph-mon), which keep track of which nodes are in operation at any given time, and the object storage daemons (ceph-osd) themselves. The incredibly cool part about RADOS is that the storage nodes have a certain level of “intelligence” built into them and have the ability to be self-healing, self-managing, and smarter than your average bear. The potential for other applications (or just enhancing this intelligence) is quite vast, especially as it relates to Ceph’s object model. Objects in Ceph have properties (as objects do), but you can also build extensions that give them methods like makeThumbnail() or MD5encrypt(). Tools like this could provide an enterprising developer hours of enjoyment, and we look forward to helping them experiment!

Other avenues of study could incorporate things like cluster power efficiency, multi data center research, plain old optimization tweakery, or many other things that we haven’t even thought of yet. Ceph has provided a veritable “wild west” of opportunities for research, development, and experimentation, and our community has responded in kind with the best creativity and ingenuity an open source project could hope for.

Conclusion

Now that you have read the details, you can see our skip-to-the-end conclusion of “a blindingly awesome ton” was pretty accurate, even with today’s list. This list grows every day thanks to the creativity of our community. We are all deeply excited to see what fancy new cloud apps, massive data applications, or other incredibly creative new tools might be built on top of Ceph tomorrow! If you have questions, ideas, or requests please feel free to snag us at one of the stops on our rigorous trade show schedule, on irc (irc.oftc.net #ceph), or on Twitter (@Ceph or @Inktank). We’d love to hear from you.

• instance – a virtual machine
• image – a template for a virtual machine
• volume – a block device
• Cinder – OpenStack service for managing block devices (replaces nova-volumes from previous versions)
• Glance – OpenStack service for storing images and metadata about them (image type, size, owner, etc.)

In previous releases, you could create volumes and attach them to virtual machines, and you could even boot from them, but there was no way to put data on them without going and doing it manually yourself. To boot from a volume, you’d have to:

1. start an instance
2. create a volume
3. attach the volume to the instance
4. write a bootable image to the volume from within the instance
5. detach the volume
6. boot from the volume

Now you may be wondering, why would I want to boot from a volume? By default, you’re booting from a file on a local disk. That’s ok, but if you boot from a volume, you can take advantage of more advanced features of volume storage:

Persistence

Volumes in OpenStack are persistent by default, so you can have vms that behave more like traditional servers which do not disappear when you reboot them.

Not tied to a single host

• vms can be migrated.
• compute and storage resources can be scaled independently.
• compute hosts can be diskless.

Advanced storage system features are available

• different types of storage (e.g. fast vs slow, replicated vs single copy, etc) can be accessed from the same compute host.
• snapshots can be created directly by the storage system instead of going through several other layers and acting more like a backup than a snapshot (this is what ‘snapshots’ of instances are in OpenStack).
• copy-on-write clones of existing volume snapshots can be created, providing a fast and easy way to restore to a past state of an instance.

In Folsom, using volumes is much easier. You can use a single API request to create a volume and populate it with the contents of an image from Glance, and you’re ready to boot from it. Thus, the 6 step manual process has been reduced to a 2 step automatic one.

Even better, if both your Glance images and your volumes are stored as Ceph block devices (RBD), creating a volume from an image can be a copy-on-write clone. This means you get a thin-provisioned, highly available disk along with fast instance creation. Plus it supports all the advanced features mentioned above.

You can find instructions for setting this up on the main Ceph.com site.

If you’d like to learn more about and Ceph and Cinder, you should check out my talk at the OpenStack conference.

In the immediate future you can find us both at the Open World Forum coming up this week in Paris and next week at the newly streamlined OpenStack Summit in San Diego.

Open World Forum

In Paris Ross Turk will be speaking on several panels. For those of you who don’t know Ross, any presentation from this seasoned Open Source veteran is well worth the time away from precious bits and/or internet cat pictures, so make sure you catch all of his appearances! Ross will be delivering both of the following talks:

The Dangers of Apathetic Abstraction

What began as a vague shape on a diagram (indicating something so incomprehensibly vast the diagrammer decided against trying to draw it) has changed everything. Yesterday, the network was out of your control and too complicated to understand; now your entire operation is. But one thing hasn’t changed: people don’t care how it works, just that it continues to.

Every day, scores of excited entrepreneurs build new companies on cloud infrastructure without a second thought. Every day, established companies relinquish their data centers. The reason is clear: it’s far easier to build something amazing when you can stop paying attention to what makes large portions of it work.

Abstraction is not having to worry how underlying technology functions; apathetic abstraction is not caring. This session will provide a discussion about the danger of relying on something you don’t control and don’t understand.

DreamObjects: Using Ceph To Build Open Cloud Services

Last month, US-based hosting provider DreamHost announced the public beta of DreamObjects, a inexpensive, scalable, reliable object storage service for web and application developers. DreamObjects is built upon Ceph, an open source storage platform designed for reliability, performance, and scalability.

DreamHost runs Ceph on everyday Linux servers, transforming three petabytes of commodity disks into an elastic, self-managing object store. Ceph works with applications written for S3 and Swift, allowing users to migrate applications seamlessly to DreamObjects.

In this talk, attendees will be introduced to the Ceph distributed storage platform and learn what makes it unique. The session will also provide insight into the design and operation of DreamObjects, the world’s largest deployment of Ceph to date.

OpenStack Summit

The OpenStack Summit will feature one of the core developers on Ceph, Josh Durgin. Lately Josh has been working on the Ceph RBD (RADOS block device) as well as how that intersects with OpenStack’s Cinder. To hear more about Josh’s epic code slinging make sure you stop by for his talk:

Storing VMs with Cinder and Ceph RBD.

Ceph is an open source distributed object store, network block device, and file system. Ceph can be used for object storage through its S3-compatible REST interface. It can also provide storage for network block devices, with the thin provisioning and copy-on-write cloning features necessary to support large-scale virtualization. With the Folsom release, Cinder makes block storage for backing VMs a first class feature in OpenStack. Block devices can be created from images stored in Glance, and with RBD behind both, new VMs can be created faster while using less space. This session will cover the current status of the integration, and discuss the technical implications and the advantages of block storage within the OpenStack cloud operating system.

If you are attending either Open World Forum or the OpenStack Summit you’ll definitely want to catch all the tidbits from the Inktank crew.

Why do this? A long long time ago, when you issued a write to a disk, it would ACK the write when the data was written. No more. Now, the ACK means the data is either the drive’s cache or on disk. You don’t know data is safe/durable until you issue a separate flush command. Now RBD behaves similarly: writes are acked immediately (up to some number of bytes, at least), and a flush will wait for all previous writes to commit. The only real difference between this and a real drive cache is that a real drive will try to coalesce small writes into a single operation, while RBD sends them all straight through to the backend cluster.

To make this work with qemu/KVM you need:

• Ceph v0.35 or later.
• Set the rbd_writeback_window to the number of bytes (something on the order of what you’d expect a physical disk cache to be.. say, 8 MB). This means using a qemu drive string like

  rbd:rbd/myimage:rbd_writeback_window=8000000

• You need qemu with commit 7a3f5fe, which wires up the qemu flush function properly.  It is not included in v0.15, but should be in the next release.

This is not yet implemented in the kernel RBD driver. As a result, effective performance using that device is still relatively poor. We hope to have similar behavior ready when the v3.2 merge window opens.

The QEMU-RBD block device has been merged upstream into the QEMU project. QEMU-RBD was created originally by Christian Brunner, and is binary compatible with the linux native RBD driver. It allows the creation of QEMU block devices that are striped over objects in RADOS — the Ceph distributed object store. As with the corresponding Linux device driver, the QEMU driver gets all the RBD goodies: thin provisioning, reliability, scalability, and snapshots!

libvirt

libvirt is a virtualization library that allows controlling virtual machines (such as QEMU based VMs, but also others) using a single API. There are many tools already built around it (e.g., virsh, virt-manager, etc.), and adding the ability to configure RBD devices via the library makes RBD work in the existing tools. With the help of the Sheepdog project (whom also merged their QEMU block device upstream into QEMU recently), we were able to get RBD (and Sheepdog, and also nbd) support upstream into libvirt. Basically a new “network” disk type was added, and there are currently 3 possible types for such a disk: nbd, sheepdog, or rbd. For each you can specify a host name. E.g., for rbd the host name(s) would hold the ip address and tcp port for the ceph cluster monitor(s).

libvirt support for the Linux native kernel rbd driver is also in the works, which will allow rbd to be used with non-qemu VMs supported by libvirt (e.g., Xen, VirtualBox, VMware, etc.)

• For more information about using QEMU-RBD and libvirt, see http://ceph.newdream.net/wiki/QEMU-RBD

Linux Kernel

As we posted before, the RBD native linux device was merged into the upcoming Linux kernel version (2.6.37) which will be out in a few weeks. Since the original merge we’ve modified the RBD sysfs interface so that it’d conform better with the sysfs requirements: originally, the RBD driver was based on another linux block device called osdblk and it inherited its sysfs interface, which was monolithic and kept a single sysfs entry per config option for all the devices. This was both wrong and cumbersome, as we needed to specify the device id for each operation. The new interface moves the sysfs rbd subdir to a better location (/sys/bus/rbd) and creates a subdir per device, so that all operations for a single device are grouped together, and there’s no need to specify the device name. We also create a subdir per snapshot under the device that holds all its information, and we dropped the one-big-list-for-all entry.

All in all, it was a relatively big change to introduce well into the release cycle, but we believe it was worth it.