Ceph » Dev notes

New Ceph Backend to Lower Disk Requirements

scuttlemonkey — Tue, 11 Jun 2013 16:20:11 +0000

I get a fair number of questions on the current Ceph blueprints, especially those coming from the community. Loic Dachary, one of the owners of the Erasure Encoding blueprint, has done a great job taking a look at some of issues at hand.

When evaluating Ceph to run a new storage service, the replication factor only matters after the hardware provisioned from the start is almost full. It may happen months after the first user starts to store data. In the meantime a new storage backend ( erasure encoded ) reducing up to 50% of the hardware requirements is being developped in Ceph.

It does not matters to save disk from the beginning : it is not used anyway. The question is to figure out when the erasure encoded will be ready to double the usage value of the storage already in place.

When looking for a new storage solution the hardware requirements are an important factor. If Ceph is configured with three replicates, 1PB of usable storage requires 3PB of actual storage. The users are expected to occupy an increasing amount of disk space over time:
            ^
       10PB |
            |
            |
        6PB |
            |                                          /--
            |                                     /----
        4PB |                                /----
            |                           /----   usage
            |                      /----
        2PB |                 /----
            |             /---
            |        /----
            |   /----
            +----------------+----------------+------------>
                          A months          B months
Hardware provisioning is expected to follow the usage curve. In the following, 4PB are provisionned initialy, an additional 2PB after A months of operation etc.
            ^
       10PB |                                 +-----------
            |                                 |
            |                                 |
        6PB |                +----------------+
            |                |    provisioning         /---
            |                |                    /----
        4PB +----------------+               /----
            |                           /----  usage
            |                      /----
        2PB |                 /----
            |             /---
            |        /----
            |   /----
            +----------------+----------------+------------>
                          A months          B months
An erasure encoded Ceph backend could reduce the requirements for raw storage : 1PB of usable storage fits in 1.5PB of raw storage. If it was available the curve would not grow as fast and the need for provisioning more hardware would happen at a later time.
            ^
       10PB |
            |
            |
        6PB |                                    +---------
            |                                    |
            |                                    |
        4PB +------------------------------------+
            |                  provisioning
            |                                    /---------
        2PB |                          /---------  usage
            |                /---------
            |        /-------
            |   /----
            +----------------+----------------+------------>
                          A months          B months
The implementation of an erasure encoded backend for Ceph started in may 2012 and when it is released, it will progressively lower the disk space requirements. In the example above it will save money if it happens before A months. However, even if it happens later, it will still save money by reducing the storage footprint and make better use of the existing hardware.
            ^
       10PB |
            |
            |
        6PB |
            |
            |
        4PB +-----------------
            |
            |
        2PB |
            |
            |
            |
            +----------------+
                          A months
In any case, it does not save any money to have erasure encoding from the start because the provisionned hardware is completely empty. Up to A months, the investment to provision 4PB was done anyway.

Originally posted by Loic Dachary.
scuttlemonkey out

Incremental Snapshots with RBD

scuttlemonkey — Tue, 14 May 2013 14:02:34 +0000

While Ceph has a wide range of use cases, the most frequent application that we are seeing is that of block devices as data store for public and private clouds managed by OpenStack, CloudStack, Eucalyptus, and OpenNebula. This means that we frequently get questions about things like geographic replication, backup, and disaster recovery (or some combination therein, given the amount of overlap on these topics). While a full-featured, robust solution to geo-replication is currently being hammered out there are a number of different approaches already being tinkered with (like Sebastien Han’s setup with DRBD or the upcoming work using RGW).

However, since one of the primary focuses in managing a cloud is the manipulation of images, the solution to disaster recovery and general backup can often be quite simplistic. Incremental snapshots can fill this, and several other, roles quite well. To that end I wanted to share a few thoughts from RBD developer Josh Durgin for those of you who may have missed his great talk at the OpenStack Developer Summit a few weeks ago.

For the purposes of disaster recovery, the idea is that you could run two simultaneous Ceph clusters in different geographic locations and instead of copying a new snapshot each time, you could simply generate and transfer a delta. The incantation would look something like this:

rbd export-diff --from-snap snap1 pool/image@snap2 pool_image_snap1_to_snap2.diff

This creates a simple binary file that stores the following information:

original snapshot name (if applicable)
end snapshot name
size of the image at ending snapshot
the diff between snapshots

The format of this file can be seen in the RBD doc.

After exporting a diff you could either simply back up the file somewhere offsite or import the diff on top of the existing image on a remote Ceph cluster.

rbd import-diff /path/to/diff backup_image

This will write the contents of the differential to the backup image and create a snapshot with the same name as the original ending snapshot. It will fail and do nothing if a snapshot with this name already exists. Since overwriting the same data is idempotent, it’s safe to have an import-diff interrupted in the middle.

These commands can work with stdin and stdout as well, so you could do something like:

rbd export-diff --from-snap snap1 pool/image@snap2 - | ssh user@second_cluster rbd import-diff - pool2/image

You can see which extents changed (in plain text, json, or xml) via:

rbd diff --from-snap snap1 pool/image@snap2 --format plain

There are a couple of limitations in the current implementation, however.

There’s no guarantee you’re importing a diff onto an image in the right state (i.e. the same image at the same snapshot as the diff was exported from).
There’s no way to inspect the diff files to see what snapshots they refer to, so you’d have to depend on the filename containing that information.

While the implementation is still relatively simple, you can see how this could be quite useful in managing not only cloud images, but any of your Ceph block devices. This functionality hit the streets with the recent ‘cuttlefish‘ stable release, but if you have questions or enhancement requests please let us know.

To learn more about some of the new things coming in future versions of Ceph you can check out the current published roadmap of work Inktank is planning on contributing. Also if you missed the virtual Ceph Developer Summit, the videos have been posted for review. In the meantime, if you have questions, comments, or anything for the good of the cause feel free to stop by our irc channel or drop a note to one of the mailing lists.

scuttlemonkey out

Adding Support for RBD to stgt

dmick — Thu, 21 Mar 2013 10:51:26 +0000

tgt, the Linux SCSI target framework (well, one of them) is an iSCSI target implementation whose goals include implementing a large portion of the SCSI emulation code in userland. tgt can provide iSCSI over Ethernet or iSER (iSCSI extensions for RDMA) over Infiniband. It can emulate various SCSI target types (really
“command sets”):

SBC (normal “disk” type devices)
SMC (“jukebox” media changer)
MMC (CD/DVD drive)
SSC (tape device)
OSD (the ‘object storage device’)

It can use either a raw block device or a file as backing storage for any of these device types.

Well, since Ceph provides a distributed reliable storage pool on the network, having a way to access that storage as an iSCSI device seems natural; this way clients that speak iSCSI don’t even need to be aware that their storage is on the Ceph cluster (except to know that it’s highly available and safe). Virtual machine providers and cloud software of many types can speak iSCSI, and if Ceph could export storage as an iSCSI device, it would be easy to glue all those providers to a Ceph cluster.

To that end, I’ve written a backend for tgt that can use a RADOS block device (rbd) image as the storage for the iSCSI target device. Now, you may be saying, “but wait, I can already create an RBD image on the Ceph cluster, map it in my kernel as a block device, and use tgt or LIO or other iSCSI tools to export it as an iSCSI target”, and that’s correct; people do this with success today. However, the completely-userland approach has several benefits:

the userland code for rbd typically leads the kernel implementation with respect to new features. At the time of this writing, userland rbd can use copy-on-write cloning and ‘fancy striping’, which are still being implemented in the kernel
userland code can be compiled and installed on older kernels that may not have the kernel rbd module available at all, or may have an older, less-stable version
avoiding the kernel can be useful for throttling memory/bandwidth, management in general, delegating access, security, avoiding kernel crashes, etc.
Without risk of memory deadlock, we can perform much better caching in the userland librbd

Of course these advantages don’t come for free; there can also be a cost in performance. The tgt project has taken some care to try to mitigate performance effects, but your mileage may vary. However, the ease of the port, and the ease of modifying it for new features makes this a worthwhile effort even in the face of possible performance hits.

The bs_rbd backing-store driver

Adding rbd support to tgt was fairly simple due to its modular design and simple backing-store drivers. Starting from the bs_rdwr backing-store driver, which backs the daemon-provided instances with either a file in a filesystem or a block device, using normal open/close /read/write functions, I added the initialization to open a connection to the RADOS cluster using librados, and a small function to parse the rbd pool, imagename, and snapshot name out of the arguments (see below). Then the POSIX calls were translated into librbd calls for rbd_open, rbd_close, etc. librbd is very similar to POSIX file operations in both its synchronous and asynchronous forms, so the translation was obvious and easy.

The patch to add bs_rbd has been accepted into the mainline repository as of mid-February 2013, along with some very brief README information on how to use it. Here’s a little expansion on that brief usage:

tgtd is configured by the tgtadm command; to select an RBD image as the backend storage for a tgtd instance, you use the –bstype rbd option to tell tgtd that it should access the storage using bs_rbd. Also, use the –backing-store option to select the (already-existing) rbd image in the usual Ceph syntax: –backing-store [pool/]image[@snap] to select an rbd image named ‘image’, optionally in pool ‘pool’, and optionally a readonly snapshot of that image @snap. You can create the image in the usual way, using the rbd command-line tool.

You must give the device you’re creating a name; a typical name form would be an ‘IQN’ (iSCSI qualified name), of the form: iqn.<year>-<month>.<domain>:<domain-specified-string> but no particular form seems to be required, so ‘testrbd’ works just as well. In my testing I created a target named simply “rbd”.

Using bs_rbd with tgtd

So a typical setup using manual commands might go like this: First, create an image on your running Ceph cluster:

rbd create iscsi-image --size 500       # a 500 MB image named iscsi-image

tgtadm/tgtd will access the cluster using the configuration supplied via the default Ceph configuration files (by default, /etc/ceph/$cluster.conf, ~/.ceph/$cluster.conf, and ./$cluster.conf, where $cluster is, by default, ‘ceph’), or by the CEPH_CONF environment variable; make sure your configuration is accessible through one of those settings.

Next, create a new target for the tgtd daemon to emulate:

tgtadm --lld iscsi --mode target --op new --tid 1 --targetname rbd

Create a LUN on this target bound to an rbd image:

tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 0 --backing-store iscsi-image --bstype rbd

Allow access to that lun:

tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

Verify that the image can be seen by a local iscsi initiator. Here I’m using iscsiadm, part of the open-iscsi package:

iscsiadm -m discovery -t st -p localhost

Log into node, which will create a /dev/sdX block device:

iscsiadm -m node --login

Now you can access the device locally as /dev/sdX using iSCSI. You can also perform the last two steps from a different network host, specifying -p <tgtd-hostname>, of course.

When you’re done, you can terminate the session and remove the device:

iscsiadm -m node --logout

Details of the bs_rbd backing-store driver, possible future work

As a first implementation, I wrote the bs_rbd driver to handle up to 20 simultaneous rbd images (just an arbitrary fixed-size-array limit), and used the bs_rdwr module as a starting point, so that I/O is synchronous to the RADOS cluster. However, tgtd itself maintains a thread worker pool of, by default, 16 threads, so while I/O blocks in the RADOS cluster, the daemon itself maintains multiple outstanding requests. The thread count can be adjusted with -t or –nr-iothreads when creating the LUN.

It’s possible that using librbd’s asynchronous-I/O support would improve performance or CPU utilization; this is something that could be the basis of experiments. I chose the simpler implementation as a working proof-of-concept; performance studies and experiments would be welcomed.

The driver links against librbd and librados, so those must be installed on your machine to build, and you must select the configuration option CEPH_RBD by, for example, “make CEPH_RBD=1″.

Try it!

So that’s the story of rbd support in tgt! Please try it out and let us know what you think; report any bugs to the stgt mailing list and the ceph development list: stgt@vger.kernel.org and ceph-devel@vger.kernel.org.

Ceph’s New Monitor Changes

joao — Thu, 07 Mar 2013 23:11:16 +0000

Back in May 2012, after numerous hours confined to a couple of planes since departing Lisbon, I arrived at Los Angeles to meet most of the folks from Inktank. During my stay I had the chance to meet everybody on the team, attend the company’s launch party and start a major and well deserved rework of some key aspects of the Ceph Monitor. These changes were merged into Ceph for v0.58.

Before getting into details on the changes, let me give some background on how the Monitor works.

Monitor Architecture

As you may already know, the Monitor is a critical piece in any Ceph cluster: without at least one monitor, the cluster just won’t do anything useful. And by that I mean nothing will happen. Ever.

Think of the monitors as that central piece of the cluster that keeps track of who and where the other pieces of the cluster are and what is happening with them. Through a single monitor, a Client is able to obtain the location of the remaining monitors, where the object storage daemons (OSDs) or the metadata servers (MDS) can be found, or figure out where the data lies; and it is to the monitors that OSDs and MDS’ will report.

The monitor tracks a lot of information essential to the cluster’s operation, much of which is at some point provided by the other components in the system. Some of this information is kept in the form of maps — OSDMap and PGMap, to name a couple –, and each map may have multiple versions. For instance, the OSDMap contains the location of the OSDs, the CRUSH map, and numerous statistics; the PGMap keeps track of PGs and where they are located at any given moment, with different versions providing different insights on the cluster history. So one might want to consider having multiple monitors in the same cluster, not only to guarantee redundancy of this information in case the monitor’s data store suffers a terrible death, but also to guarantee availability if something should happen to the monitor (power or network failure on the monitor’s server or rack, for instance).

However, keeping multiple monitors means that the information must be equally shared by them all. Any potential inconsistencies, may they be lost or corrupted versions, could lead to incorrect cluster behavior or even data-loss. In order to enforce the consistency requirements throughout the monitor cluster, Ceph resorts to Paxos (http://en.wikipedia.org/wiki/Paxos_(computer_science)), a distributed consensus algorithm. Each time a map is modified, a new version is created and run through a quorum of monitors. When a majority acknowledges the change, and only then, the new version will be considered committed. Throughout the documentation and the mailing list archives one can find numerous reasons to maintain more than one monitor (and an odd numbers at that), but I believe that Mike Lowe described it the best in an email to the list

(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-February/000224.html):

“Think of yourself as a mob boss and the mon is your mob accountant. While you may have all of the account numbers where you have stashed your ill gotten gains only your accountant knows which bank those account numbers belong to. If you or somebody else whacks your sole accountant then your money is gone. Oh, and your accountants may lie to you so best to have an odd number and let the majority rule.”

ARCHITECTURAL REWORK

There were three major architectural reworks on the monitor, which I will explain in further detail in the next sections:

Shifting from the legacy file-based data store onto a Key/Value store;
Introducing a single Paxos instance instead of an instance per monitor service; and,
Performing a store-wide sync to catch up with cluster state

K/V STORE INSTEAD OF THE LEGACY FILE-BASED STORE

Up until v0.58, the monitor’s data store was comprised of a set of files and directories. This approach benefited from the simplicity of inspecting the data store with tools like ‘ls’, ‘cat’, and the likes. However, this simplicity would also enable some creative problem-solving approaches, and every now and then we would get ahold of someone whose monitors would crash and burn because there had been some tampering with monmap versions, for instance. Let me make this clear: black boxing the monitor is not a workaround to avoid this kind of approach — once the users understand that they should not do it, they won’t ever repeat the feat –, but black boxing does provide other benefits that are far more important.

The file-based monitor store has some other drawbacks that cannot be avoided by simply informing the user. For instance, there is no way to atomically change a set of files. This issue is not uncommon in file systems, and several techniques have been developed to work around it, but they are annoying and there are several circumstances when they don’t really work out. Take how the monitor applies a version on its data store: reads V, the latest committed version in the store; creates a new file under foo/V+1 with the new version’s contents; writes V+1 to the latest committed version file. Now say that the store runs out of space when writing the new version’s contents to disk. There is a chance that only part of the contents ever reached disk, and we might end up with a corrupted version. And you say, “but we didn’t mark that version as being the last committed version, so there’s no problem, right?!?”. Well, that is certainly true, to some extent of true. The real story is that, during recovery, the monitor might check if there is an uncommitted version in the store, and if so try to run it through Paxos, and in this case the version might be corrupted. So one would say that this is a bug, the store’s fault, and one would certainly be correct: it could be avoided by stashing the version’s crc before we went out to write it out, and we could check if the crc matched the read version before we did anything with it.

Sure, we could have kept on working on the file-based store, adding features as we deemed necessary, and that’s what we probably would have done if we weren’t about to perform a major rework on the monitor. Therefore, instead of focusing on extending the existing file-based store, we decided it was time to move on to a key/value store with all the properties we were looking for, and given that we have already been using one such store in Ceph, we just went ahead and used leveldb (http://code.google.com/p/leveldb/).

In fact, the legacy file-based store acted much like a key/value store when it came to data placement. It was mainly comprised of files holding data, the filename acting as the key, the data acting as the value. Thus, moving to a key/value store didn’t pose much of an ordeal, and it gave us something we were really looking forward to use on the remaining architectural rework of the monitor: transactions, being able to perform multiple modification operations in one single atomic batch.

PAXOS & MONITOR SERVICES

Figure 1

We have already discussed how the monitor maintains map consistency throughout the cluster by resorting to Paxos, but we didn’t give much detail on it. Without getting into excruciating detail, in fact the monitor can be seen as being divided in 6 services, each responsible to handle one kind of information: authentication, logging, MDS, Monitor, PG and OSD maps. Each of these services are what we call ‘Paxos Services’, given that they pretty much behave as paxos machines, each one maintaining a Paxos instance (see Figure 1). This means that at any given moment, it would be theoretically possible to have 6 parallel modifications going on, granted each one would be of a different type. In reality they are not really parallel, as the monitor only handles one message at a time, but it is possible to keep multiple concurrent Paxos proposals.

Having a Paxos instance per service guarantees that each service will keep track of its own versions, and will be responsible for their maintenance that may differ from service to service, depending on different requirements and criteria. Basically, this approach confers a great deal of autonomy to each service, at the expense of some redundancy by having multiple Paxos instances when just one would be enough. In Figure 1 we roughly depict how each service used perform their read and write operations on the monitor data store. In a nutshell, most modifications would be made through their Paxos instance, while reads would be directly performed by the service. We say most modifications because we would only resort to Paxos when dealing with a new version on the cluster. There were several other modifications that would be done directly on the store, as long as they were considered as not affecting the global Paxos state, such as version trimming (i.e., getting rid of old, unnecessary versions).

Disregarding the conceptual architecture and diving for a moment into the implementation point-of-view, each service also involved quite a bit of effort when accessing its own data, as they were required to some extent to explicitly use the file-based monitor store interface to access their allocated namespace within the file system. Little to no abstraction was provided.

This brings us to our ultimate goal with the architectural rework: use one single Paxos instance across all services, while keeping their autonomy and sandboxing their store accesses to their own namespace using a clean and simple-to-use interface.

ONE PAXOS TO RULE THEM ALL

Figure 2

Although using a single Paxos instance makes sense, it involved some serious reworking on how the services perceive their world, as well as how Paxos is used within the monitor.

Instead of keeping up with the previous approach of using Paxos solely to run new versions of a specific service through the other monitors in the cluster, we now use it to perform any change whatsoever across the cluster, thus guaranteeing that all the monitors are constantly in sync — and this means trimming too, which is now enforced to happen at the same time across the cluster. Therefore, with the Single Paxos approach we make sure that every write is run through Paxos prior to be applied onto the store, although services can read the store directly (see Figure 2).

This approach posed one major challenge: given that service versions (and by that we mean, for instance, map epochs) were directly associated with the Paxos version, ranging from [1,n] in incremental fashion, how would we now deal with this given that we have only a single Paxos instance? Would we end up with gaps in map epochs? Were this a headline, I could easily refer to Betteridge’s law of headlines (http://en.wikipedia.org/wiki/Betteridge’s_law_of_headlines); given it’s not, I will just have to answer No! and explain why.

In dissociating Paxos from the services, the Paxos’ version became analogous to a global version, representing a given proposal’s version instead of a map epoch. The services kept their responsibility of managing their own versions, and are absolutely oblivious to the fact that there is only one single Paxos instance — they really don’t care, they just push their changes up the chain, and propose them to the cluster. The same goes to Paxos. By leveraging the new key/value store’s capability to perform transactions, not only are we able to abstract the services from however Paxos deals with versions, but we are able to abstract Paxos from whatever the services propose, which didn’t happen before — the Paxos/service relation was so tight that a Paxos proposal took the form of set version ‘foo’ with contents ‘bar’ for service ‘baz’.

Figure 3

With the support of transactions however, we can make a service generate a transaction containing the operations it wants to perform on its namespace — which will be properly adjusted to reflect the service’s namespace without the service being aware. The transaction will then be encoded into a byte array (Ceph has all the data structures allowing this to happen effortlessly), and submitted to Paxos. Take Figure 3, where we depict this process. Once the service’s transaction reaches Paxos, a new transaction will be created, reflecting the new Paxos version. In Figure 3 we can see that Paxos will create a new version 42 with the contents of the service’s encoded transaction — Paxos won’t care what the contents really are though; they are meaningless from its point-of-view. Once the proposal is acknowledge by a majority of monitors, each monitor will perform one single transaction comprised of the Paxos transaction’s operations and the service’s transaction’s operations — all of them applied in one single atomic batch.

This approach is also used for pretty much any operation requiring to be applied throughout the cluster in a consistent manner. For instance, while we used to let each service, on each monitor, decide when to trim their versions, we now delegate that decision only to the Leader on the monitor quorum. Periodically, the Leader will assess which versions, either Paxos or service-specific, need to be trimmed, generating a transaction comprised of erase() operations over Paxos versions, alongside with service-specific versions (if any). Similarly to what happens with other modifications, this transaction is proposed through Paxos, which will create a new version containing the encoded proposed transaction, finally applying it throughout the cluster.

One might have noticed that we just stated that trimming versions is also a Paxos proposal that will lead to a new Paxos version. Well, that is by design, and comes as wonderfully useful when recovering drifted monitors.

A monitor is considered as having drifted if it is behind a given number of Paxos versions. If this number is small enough such that its last committed version is within the interval of available versions on the remaining cluster monitors, then the monitor is able to recover without much effort, simply by obtaining the missing Paxos versions and re-applying them on the store — some of these versions can simply add new information to the store, or erase old versions; regardless, the monitor will obtain a consistent state with the remaining cluster.

However, at times there is a chance that the monitor drifted so much that no longer shares any Paxos version with the remaining cluster. At this point, the monitor must perform a store-wide synchronization.

STORE-WIDE SYNCHRONIZATION

Prior to v0.58, when a Paxos service drifted beyond a given number of versions, a mechanism called slurp would be triggered. In a nutshell, this mechanism consisted of establishing a connection with the quorum Leader and obtain every single version the Leader had, for every service that had drifted. Such approach was adequate to the one-Paxos-per-service architecture, but wouldn’t fare so well on a single Paxos architecture. The reason is simple and follows the behavior of Paxos as it was described in the previous section: Paxos versions no longer represent service versions, and only synchronizing them would certainly lead to a corrupted state, with lots and lots of information missing.

So we got rid of slurp. Instead, we leveraged leveldb’s snapshots and iterators, and we now perform a store-wide sync. This means that once a monitor (hereafter known as Requester) finds out it has drifted beyond salvaging, it will request some other monitor (hereafter known as Provider) to perform a sync. The Provider will then take a snapshot of its store and iterate over it, bundling all the key/values it can find into transactions and sending them to the Requester. The Requester will apply each received transaction and once it receives the last chunk it will be ready to join the cluster.

The great thing about this new mechanism, is that unlike the slurp, the Requester doesn’t really need to synchronize directly from the quorum’s Leader. Instead, it may synchronize from any given monitor in the quorum, and there may be any given number of syncs being performed simultaneously, without overloading the Leader.

BUT, BUT… IS UPGRADING COMPLEX? IS IT POSSIBLE TO REVERT?

Well… no and kind of.

Sometime around Bobtail, the monitor started recording a Global Version for each version a service proposed through their Paxos instance. After some time running, the monitor be holding a mapping from service-specific Paxos versions to a Global Version, and would then set a flag on its store stating just that: we are now able to map any Paxos version to a global id.

This was slipped into the monitor in order to allow us to upgrade a monitor from the one-Paxos-per-service to the single Paxos architecture. So, basically, as long as one has been running a Bobtail monitor for some time, upgrading to the new monitor should be as simple as restarting it and waiting for the store conversion to finish. This conversion will be triggered automatically, and may take some time if the store is big enough. So, no, upgrading is not complex, granted you are coming from Bobtail; otherwise, you will have to upgrade to Bobtail and take it from there.

If upgrading fails for some reason, we would really appreciate if you’d let us know on the mailing-list and/or on IRC. In any case, there is no need for despair. Given that during conversion we only perform read operations on the legacy file-based store, and we convert everything into a leveldb sub-directory on the monitor’s data directory, you can easily revert to your original data store simply by running your old monitor. However, if your monitor did not fail, if you successfully upgrade all your monitors and they form a quorum, from that point onward there is no going back (unless you are okay with just reverting to an older state). Furthermore, you should be aware that this upgrade does not allow for mixed monitor clusters, so there is no point in trying to upgrade just part of your monitor cluster: it won’t work as the post-rework code is unable to understand pre-rework way of doing business.

SUMMARY

Over the past ten months the Ceph Monitor undertook a major rework, from its backend data store moving from a file-based format to a key/value store supporting atomic transactions, to the way versions are created, unifying all services under a single Paxos instance and sandboxing their access to the data store. Such rework allowed us to suppress some limitations of the previous architecture, and to create an architecture that by dissociating Paxos from the monitor services it will allow us to disseminate information throughout the cluster in a seamless way, allowing to simplify how new capabilities can be built around and within the monitor. Future versions of the monitor may for instance include a generic key/value store, such that a user could stash and retrieve information deemed necessary, while benefitting from the distributed and high-availability nature of the monitor. There’s work being developed towards such an implementation, taking advantage of the mechanisms now in place, leveraging Paxos as a conduit of modifications throughout the cluster.

If you would like to know more, feel free to take a look at the commit messages of the patches introducing the whole architectural rework (single Paxos, trimming through Paxos, and store sync), dive into the source code, or chat us up on the mailing list or IRC!

CephFS MDS Status Discussion

gfarnum — Tue, 05 Mar 2013 16:45:48 +0000

There have been a lot of questions lately about the current status of the Ceph MDS and when to expect a stable release. Inktank has been having some internal discussions around CephFS release development, and I’d like to share them with you and ask for feedback!

A couple quick notes: first, this blog post is from the perspective of Inktank’s development. We aren’t the only ones generating metadata server (MDS) patches, and other parties might make contributions with different priorities! Second, this is a discussion about MDS development — look for a blog about what the MDS does and how it works coming soon!

Current Status

Over the past year, we at Inktank have regretfully stepped back from the filesystem — we still believe its feature set and capabilities will revolutionize storage, but we realized it required a lot more work to become a stable product than RBD and RGW, so we focused our efforts on the software we could give to customers. That is still Inktank’s organizational focus, but at the turn of the year something wonderful (for me personally) happened! We created an internal CephFS team and I and Sam Lang have been devoting an increasing amount of our time to work on the MDS and filesystem development. This renewed focus has emphasized what kinds of issues remain. There are a few brave organizations using CephFS in testing or production capacities, but the more important its use is to them the less functionality they rely on. For community members my recommendation has been to test CephFS under your workload for two weeks, inject some failures (node restarts, etc), and if it works through that then it should continue working — some people have run systems for months without issues, but others run into trouble on their second or third command. Basically, if your workload happens to look like one of the test suites we regularly run it should be good — but if it deviates even a little there are hidden traps lying in wait from bugs that we haven’t yet discovered.

Initially our goal was to stabilize the features the filesystem already has and develop fsck, but through recent discussions we realized we hadn’t sat down and figured out what our users and customers actually needed CephFS to do in order to put it into production deployments. More than that, while we’ve viewed CephFS for years as this big ball of awesomeness with features like snapshotting and multiple active servers and unlimited directory sizes, we don’t know which of those features are actually necessary for a first release — and the bugs we’ve been working on have reminded us that some of them require a lot more stability work than others.

Keeping that thought in mind, we’re now starting a discussion with customers, users, and the community at large to discuss what a “minimum viable product” for CephFS would look like. Our starting point is just what’s easy from a development perspective, and we would welcome feedback from users on if this works for them, or how it would need to change before they could deploy it.

Minimum Viable Product Proposal

As we put more engineering resources back into CephFS, we are looking at what we would consider as the minimal useful feature set in order to get CephFS into the hands of production users as soon as possible. We are currently considering it to be a single active MDS, with a maximum number of entries in a single directory, no fsck, and no snapshots. This delivers a POSIX-compliant filesystem that can be mounted on thousands of clients, scales to arbitrarily large data throughputs, allows an unlimited number of files in the hierarchy, is location-aware, and can be used through a number of interfaces (Ganesha NFS, Hadoop, the in-kernel and FUSE-based clients, Samba, etc).

Let me break down what each of those assertions means in more detail.

Single active MDS

One of CephFS’ flagship features is its horizontal scalability across very large numbers of metadata server daemons. This will continue to be a flagship feature in the future, but right now it introduces significant system instability so it will not be a part of our initial supported release.

However, the standby and active standby features are very stable and will be part of the first release. This means that the fast failover features (30 seconds or much less, depending on user settings, hardware, and tolerance for unnecessary failovers) will function, allowing users to provision as many servers as they wish in case of a hardware failure, or to take over during maintenance.

The primary limits implied by a single-MDS configuration are the number of metadata operations/second the system can handle, the number of simultaneous client connections it can handle, and the amount of metadata the MDS can store in memory. This last provides a limit on how many files can be in use simultaneously with good performance, but not on total number of files in the system (which remains effectively unlimited). As always, the amount of RAM and CPU available to the MDS node will have a dramatic impact on where precisely these limits fall.

Aside: We currently default to 100,000 inodes in the cache, but that is extremely conservative and fits inside the low hundreds of MB of RAM. We don’t yet have recent specific values on memory consumption per inode.

Maximum number of entries in a single directory

CephFS includes preliminary support for directory “fragmenting” (or sharding), which allows us to both split up a single directory on-disk and to split it up between multiple MDS servers. Again though, while the code exists it requires a significant amount of validation and debugging, so our first release will not provide support and we will need to limit the total number of entries allowed within a single directory. This is a soft limit open to negotiation — the MDS needs to be able to hold the whole directory in-memory whenever it is read off disk, and if the directory holds more entries than the MDS cache can hold the cache efficiency and overall performance will naturally degrade (and, if more than one directory is in use they will feed back on each other).

However, “manually” sharding directories by splitting them up according to any given heuristic (which splits them finely enough) works just fine, and this limit does not imply a limit on the total number of files in the system. (As long as one considers the maximum amount of active metadata discussed above.)

No fsck

As with many distributed systems, CephFS does not currently provide an fsck. Initial design work has been done but not yet implemented. CephFS does of course inherit RADOS’ underlying reliability methods, which include a periodic scrub of the data for consistency between replicas, checksum-based validity checks (upcoming in the Cuttlefish release), and replication of data and recovery when it degrades. Unfortunately this does not completely insure CephFS — objects which are completely lost will translate into file holes, and will not necessarily trigger alarms. Any lost directories in the filesystem hierarchy (which will be detected) must be repaired manually.

On the positive side, due to Ceph’s design, any portions of the hierarchy which have not been damaged will continue to function even if data has been lost.

No snapshots

While CephFS has preliminary support for snapshots of directory hierarchies, it too requires significant hardening and debugging. We will not support them in our initial release. When they are released, it will be a pioneering feature among distributed filesystems.

POSIX-compliant

CephFS is POSIX-compliant, always has been, and always will be. It provides proper consistency (rather than open-to close as NFS does), and supports even less commonly-used features such as file locking.

Hard links are also supported, although in their current implementation each link requires a small bit of MDS memory and so there is an implied limit based on your available memory. We have designed but not implemented a new solution to avoid this problem.

Can be used by thousands of clients simultaneously

In past testing (during Sage Weil’s PhD thesis work), MDS servers have had no trouble handling 1000 clients each, and while we haven’t tested recently we expect that number to have improved rather than degraded.

Scales to arbitrary data throughputs

In CephFS, once the client has opened a file, the MDS does not play a further role in the data path. That means that if your clients can send the data, and your OSDs can write the data, the single-MDS limit will not directly impact the aggregate bandwidth available. The implied limits are those based on how much each client can send out and the total number of active files the MDS can handle.

Allows an unlimited number of files in the hierarchy

Although there are limits to the size of a single directory as discussed above, Ceph does not require that every file in the system take up MDS memory at all times. This means that unlike many other systems, it does not and never will have a hard limit on the total number of files available.

Is location aware

CephFS is built on RADOS, which has a failure domain-based layout engine (which by default naturally maps onto the physical host, rack, row, room layouts of the data center). CephFS allows clients to query this layout data for files and optionally to read from local replicas. Systems which are interested in location awareness will also appreciate the ability to set custom layouts on every file, specifying the underlying object size and pool (which further dictates the the striping strategy in use).

Many interfaces

Native ceph clients are available in the upstream Linux kernel and in userspace as both a library and a FUSE module. In addition to the regular interface options available through those standard mechanisms, the library has been integrated into the Ganesha NFS server and Samba; fully integrates with Hadoop; and can be integrated into any custom application.

Feedback

As I said, we would love to get your feedback on these ideas. I’m starting a discussion thread on ceph-users as this blog goes up; you can comment here; or you can drop by irc and ping any of us. We’d appreciate any information you can provide!

-Greg out

Deploying Ceph with Juju

scuttlemonkey — Thu, 21 Feb 2013 17:55:03 +0000

The last few weeks have been very exciting for Inktank and Ceph. There have been a number of community examples of how people are deploying or using Ceph in the wild. From the ComodIT orchestration example, to the unique approach of Synnefo delivering unified storage with Ceph and many others that haven’t made it to the blog yet. It is a great time to be doing things with Ceph!

We at Inktank have been just as excited as anyone in the community and have been playing with a number of deployment and orchestration tools. Today I wanted to share an experiment of my own for the general consumption of the community, deploying Ceph with Canonical’s relatively new deployment tool, ‘Juju,’ that is taking cloud deployments by storm. If you follow this guide to the end you should end up with something that looks like this:

Juju is a “next generation service deployment and orchestration framework“. The cool part about Juju is you can use just about anything to build your Juju “charms” (recipes) from bash and your favorite scripting language, all the way up to Chef and Puppet. A good portion of the knowledge for the Ceph charms developed by Clint Byrum and James Page actually came from both the Chef cookbooks and the work on ceph-deploy, which we’ll cover in later installments.

For the purposes of this experiment I decided to build the environment using Amazon’s EC2 but you can also use an OpenStack deployment or on your own bare metal in conjunction with Canonical’s MAAS product. The client machine used to spin up the bootstrap environment and then later spin up all the other servers will be an Ubuntu Quantal (12.10) LTS image, but could be any Ubuntu box, including your laptop. The rest of the working machines will be spun up using Quantal as well.

Juju is very generous about spinning up new boxes (typically one per service) so I chose to make all of my boxes spin up using the ‘t1.micro’ machine size so anyone playing with this guide wouldn’t incur massive EC2 charges. Now, on to the meat!

Getting Started

As I said, start the process by spinning up an Ubuntu 12.10 LTS image as your client, this way you don’t have to dump a bunch of software/config on your local machine. This will be the client you use to spin everything else up. Once you have your base Ubuntu install lets add the PPA and install Juju.

> sudo apt-add-repository ppa:juju/pkgs
> sudo apt-get update && sudo apt-get install juju

Now that we have Juju installed we need to tell it to generate a config file.

> juju bootstrap

This will throw an error, but creates ~/.juju/environments.yaml for you to edit. Since we’re using EC2 we need to tell Juju about our credentials so it can spin up new machines and deploy new services. You’ll notice that I’m using the default-series of ‘quantal’ for all of my node machines. This is important since this tells juju where and how to grab the important bits of each charm.

> vi ~/.juju/environments.yaml

default: cephtest
environments:
  cephtest:
    type: ec2
    access-key: YOUR-ACCESS-KEY-GOES-HERE
    secret-key: YOUR-SECRET-KEY-GOES-HERE
    control-bucket: (generated by juju)
    admin-secret: (generated by juju)
    default-series: quantal
    juju-origin: ppa
    ssl-hostname-verification: true

Setting up the Bootstrap Environment

Now that Juju can interact with EC2 directly we need to get a bootstrap environment set up that will hold our configs and deploy our services. Since I can’t set the global configs yet, I need to tell it manually that this box needs to be a ‘t1.micro’ instance.

> juju bootstrap --constraints “instance-type=t1.micro”

This will take a few minutes to spin up the machine and get the environment set up. Once this is completed you should be able to see the machine via the ‘juju status’ command.

> juju status

2012-11-07 13:06:30,645 INFO Connecting to environment...
2012-11-07 13:06:42,313 INFO Connected to environment.
machines:
  0:
    agent-state: running
    dns-name: ec2-23-20-70-201.compute-1.amazonaws.com
    instance-id: i-d79492ab
    instance-state: running
services: {}
2012-11-07 13:06:42,408 INFO 'status' command finished successfully

Now we have a bootstrap environment and we can tell it that all boxes should default to ‘t1.micro’ unless otherwise specified. There are a number of settings that you can monkey with, take a look at the constraints doc for more details.

> juju set-constraints instance-type=t1.micro

Make it Pretty!

For those who like to see a visual representation of what’s happening, or just feel like letting someone else watch what’s going on, Juju now has a GUI that you can use. While I wouldn’t recommend using the GUI as a replacement for the command line to deploy the charms below, you can certainly use it to watch what’s happening. For more mature charms (and in the future) this GUI should be more than capable of managing your resources. In any case, it’s neat to have pretty pictures as you tapdance on the CLI.

If you would like to install the GUI feel free to grab my version of the ‘juju-gui’ charm (at the time of this article the main charm wasn’t on quantal yet):

> juju deploy cs:~pmcgarry/quantal/juju-gui

Once that completes (and it could take a while for everything to download and install) you’ll need to ‘expose’ it so you can get to it:

> juju expose juju-gui

This will give you the ability to access the box publicly via a web browser at the ec2 address shown in ‘juju status’. The detault user name and password are ‘admin’ and the ‘admin-secret’ value from your ~/.juju/environments.yaml file. Feel free to leave that up while you do the rest of this work to watch the magic happen.

Prep for Ceph Deployment

Our Juju environment is now ready to start spinning up our Ceph cluster, we just need to do a little leg work so Juju has all the important details up-front. First we need to grab a few Ceph tools:

> sudo apt-get install ceph-common && sudo apt-get install uuid

We need to generate a uuid and auth key for Ceph to use.

> uuid

insert this as the $fsid below

> ceph-authtool /dev/stdout --name=$NAME --gen-key

insert this as the $monitor-secret below.

Now we need to drop these (and a few other) values into our yaml file:

> vi ceph.yaml

ceph:
    source: http://ceph.com/debian-bobtail/ quantal main
    fsid: d78ae656-7476-11e2-a532-1231390a9d4b
    monitor-secret: AQDcNRlR6MMZNRAAWw3iAobsJ1MLoFBLJYo4yg==

ceph-osd:
    source: http://ceph.com/debian-bobtail/ quantal main
    osd-devices: /dev/xvdf

ceph-radosgw:
    source: http://ceph.com/debian-bobtail/ quantal main

You’ll notice we’re also passing a ‘source’ item to Juju, this tells the charm where to grab the appropriate code for Ceph, in this case the latest release (Bobtail 0.56.3 when this was written) from Ceph.com.

Tail Those Logs!

Since a good portion of this setup is experimental it’s a good idea to tail the logs. Thankfully, Juju makes this extremely easy for you to do. Simply open a second term window, ssh to your client machine, and type:

>juju debug-log

This will aggregate all of the logs from your cluster into a single output for easy browsing in case something goes wrong.

Deploying Ceph Monitors

Time to start deploying our Ceph cluster! In this case we’re going to deploy the first three machines with ceph-mon (Ceph monitors) since we typically recommend at least three in order to reach a quorum. You’ll want to wait until all three machines are up before moving on.

> juju deploy -n 3 --config ceph.yaml cs:~pmcgarry/quantal/ceph

You’ll notice that while these charms are in the charm store (cs:) they are off on my own user space. This is because I had to make a few tweaky changes for these charms to deploy happily on ec2 and use bobtail and quantal. These charms are still a bit new so if you have tweaks or changes feel free to give me a shout, or play with the main Ceph charms on jujucharms.com. In the future you’ll be able to deploy using just ‘ceph’ instead of anyone’s user space.

EXAMPLE: > juju deploy -n 3 --config ceph.yaml ceph

This could take a while, so just keep checking ‘juju status’ until you have the machines running AND the agents set to ‘started.’ You should also see the debug-log go through a flurry of activity when it starts getting close to the end.

Once we have the monitors up and running you can take a look at what your deployment looks like. If you want to you can even ssh in to one of the machines using Juju’s built-in ssh tool.

> juju status

machines:
  0:
    agent-state: running
    dns-name: ec2-50-16-15-64.compute-1.amazonaws.com
    instance-id: i-2b45f657
    instance-state: running
  1:
    agent-state: running
    dns-name: ec2-50-19-23-167.compute-1.amazonaws.com
    instance-id: i-3b368547
    instance-state: running
  2:
    agent-state: running
    dns-name: ec2-107-22-128-107.compute-1.amazonaws.com
    instance-id: i-1f368563
    instance-state: running
  3:
    agent-state: running
    dns-name: ec2-174-129-51-96.compute-1.amazonaws.com
    instance-id: i-15368569
    instance-state: running
services:
  ceph:
    charm: cs:~pmcgarry/quantal/ceph-0
    relations:
      mon:
      - ceph
    units:
      ceph/0:
        agent-state: started
        machine: 1
        public-address: ec2-50-19-23-167.compute-1.amazonaws.com
      ceph/1:
        agent-state: started
        machine: 2
        public-address: ec2-107-22-128-107.compute-1.amazonaws.com
      ceph/2:
        agent-state: started
        machine: 3
        public-address: ec2-174-129-51-96.compute-1.amazonaws.com

> juju ssh ceph/0 sudo ceph -s

   health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
   monmap e2: 3 mons at {ceph232118103=10.243.121.227:6789/0,ceph501969115=10.245.210.114:6789/0,ceph5423414494=10.245.89.32:6789/0}, election epoch 6, quorum 0,1,2 ceph232118103,ceph501969115,ceph5423414494
   osdmap e1: 0 osds: 0 up, 0 in
    pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
   mdsmap e1: 0/0/1 up

From this status we can see that there are three monitors up (“monmap e2: 3 mons at {…}”) and no OSDs (“osdmap e1: 0 osds: 0 up, 0 in”). Time to spin up some homes for those bits!

Deploying OSDs

Once our monitors look healthy it’s time to spin up some OSDs. Feel free to drop as many in as you please, for the purposes of this experiment I chose to spin up three.

> juju deploy -n 3 --config ceph.yaml cs:~pmcgarry/quantal/ceph-osd

That will take a little bit to complete so you may want to go grab an infusion of caffeine at this point. One thing to keep in mind is that earlier in our ceph.yaml we defined the physical devices for our OSDs as /dev/xvdf. If you are familiar with EC2 you will know that that device doesn’t exist yet, so our OSD deploy command will spin up and configure boxes, but we’re not quite there yet.

When you get back, if you take a look with juju status you should now see a bunch of new machines and a new section called ceph-osd:

> juju status

…
  ceph-osd:
    charm: cs:~pmcgarry/quantal/ceph-osd-0
    relations: {}
    units:
      ceph-osd/0:
        agent-state: started
        machine: 4
        public-address: ec2-174-129-82-169.compute-1.amazonaws.com
      ceph-osd/1:
        agent-state: started
        machine: 5
        public-address: ec2-50-16-0-95.compute-1.amazonaws.com
      ceph-osd/2:
        agent-state: started
        machine: 6
        public-address: ec2-75-101-175-213.compute-1.amazonaws.com

Now we need to actually give it the disks it needs. Via your EC2 console (or using ec2 command line tools) you need to spin up 3 EBS volumes and attach one to each of your OSD machines. If you need help there is a pretty decent, concise walkthrough at:

http://www.webmastersessions.com/how-to-attach-ebs-volume-to-amazon-ec2-instance

Once you have the volumes attached we need to tell Juju to go back and use them:

> juju set ceph-osd "osd-devices=/dev/xvdf"

This will trigger a rescan and get your OSDs functioning. All that’s left now is to connect our monitor cluster with the new pool of OSDs.

> juju add-relation ceph-osd ceph

We can ssh into one of the Ceph boxes and take a look at our cluster now:

>juju ssh ceph/0

> sudo ceph -s

   health HEALTH_OK
   monmap e2: 3 mons at {ceph232118103=10.243.121.227:6789/0,ceph501969115=10.245.210.114:6789/0,ceph5423414494=10.245.89.32:6789/0}, election epoch 6, quorum 0,1,2 ceph232118103,ceph501969115,ceph5423414494
   osdmap e10: 3 osds: 3 up, 3 in
    pgmap v115: 208 pgs: 208 active+clean; 0 bytes data, 3102 MB used, 27584 MB / 30686 MB avail
   mdsmap e1: 0/0/1 up

Congratulations, you now have a Ceph cluster! Feel free to write a few apps against it, show it to all of your friends, or just nuke it and start refining your chops for a production deployment.

Extra Credit

Since that Juju GUI screen looked so empty I decided I wanted to play a bit more with the tools at my disposal. If you would like to take this exercise a bit further we can also add a few RADOS Gateway machines and load-balance them behind an haproxy machine. To do this is only a few more commands with Juju:

> juju deploy -n 3 --config ceph.yaml cs:~pmcgarry/quantal/ceph-radosgw
> juju expose ceph-radosgw

> juju deploy cs:~pmcgarry/quantal/haproxy
> juju expose haproxy

> juju add-relation ceph-radosgw haproxy

That should be it! You’ll notice that I have my own copy of the haproxy, this is simply because it isn’t technically released for quantal yet, but my (unmodified) version seems to run just fine.

Troubleshooting

Juju actually makes troubleshooting and iterative development VERY easy (one of my favorite things about it). If you would like to delve deeper into playing with Juju I highly recommend reading their docs, which are quite good. However, one of the most useful tools (beyond the debug-log I mentioned earlier) is the ability to step through the hooks as juju tries to run them. For example, lets say we tried to deploy Ceph and ‘juju status’ was telling us there was an ‘install-error.’ We could use our second term window to execute the following:

> juju debug-hooks ceph/0

This allows us to debug the execution of the hooks on a specific machine (in this case ceph/0). Now in our main window we can type:

> juju resolved --retry ceph/0

We get a preformatted setup in our ‘debug-hooks’ window with an indication at the bottom that we’re on the “install” hook. From here we can change to the hooks directory and rerun the install hook:

> cd hooks
> ./install

From here we can troubleshoot errors on this box before going back and pushing a patch to Launchpad.net. I wont try to recreate the expansive documentation on the jujucharms site, but fiddling with Juju has been far less frustrating that some other orchestration frameworks I have poked at recently. Good luck, and happy charming!

Cleaning Up

If you would like to close up shop you can either destroy just the services (if you want to keep the machines running for deploying other Juju tests):

> juju destroy-service ceph
> juju destroy-service ceph-osd
> juju destroy-service ceph-radosgw
> juju destroy-service haproxy

…or just drop some dynamite on the whole thing (this will kill everything but your client machine, including your bootstrap environment):

> juju destroy-environment

Wrap Up

You are now a seasoned veteran of Ceph deployment, what more could you want? If you do have questions, comments, or anything for the good of the cause we would love to hear about it. Currently the best way to get help or give feedback is in our #Ceph irc channel but our mailing lists are also pretty active. For Juju-specific feedback you can also hit up the #Juju irc channel. If you see any egregious errors on this writeup or would like to know more about Ceph community plans feel free to send email to patrick at inktank dot com.

scuttlemonkey out

What’s New in the Land of OSD?

sajust — Tue, 18 Dec 2012 13:54:21 +0000

It’s been a few months since the last named release, Argonaut, and we’ve been busy! Well, in retrospect, most of the time was spent on finding a cephalopod name that starts with “b”, but once we got that done, we still had a few weeks left to devote to technical improvements. In particular, the OSD has seen some new and interesting developments.

OSD Internals Overview

Let’s start with some background for those not familiar with ceph internals. Objects in a Ceph Object Store are placed into pools, each of which is comprised of some number of placement groups (PGs). An object “foo” in pool “bar” would be mapped onto a set of osds as follows:

The first mapping hashes foo to 0x3F4AE323 and maps “bar” to its pool id: 3. The next mapping maps this to PG 3.23 (pg 23 in pool 3) by taking 0x3F4AE323 mod 256 (the number of PGs in pool “bar”). This pgid is then mapped onto the osds [24, 3, 12] via CRUSH. osd 24 is the primary; 3 and 12 are the replicas. PGs serve several critical roles in the ceph-osd design. First, they are the unit of placement. If we calculated placement directly on a per-object basis, changes in the cluster might require us to recalculate the location of each and every object! This way, we only need to re-run CRUSH on a per-PG basis when the cluster changes. Second, writes on objects are sequenced on a per-PG basis. Each PG contains an ordered log of all operations on objects in that PG. Finally, recovery is done on a per-PG basis. By comparing their PG logs, two osds can agree on which objects need to be recovered to which OSD.

Scrub

With that out of the way, let’s move to some work on keeping your cluster’s data honest. It turns out that data redundancy isn’t particularly useful if you fail to notice a corrupted object until you finally go to read it, possibly months after the last copy has finally become unreadable. To deal with this, ceph has long included a “scrub” feature which, during periods of low IO, chooses PGs in sequence and compares their contents across replicas. Alas, our implementation suffered from two shortcomings. The first is that we compared the set of objects contained in each PG across replicas as well as object metadata, but not the object contents. In the upcoming Bobtail release, we hash the object contents as we scan and compare the hashes from across replicas to detect corrupt copies.

The second shortcoming is that, in the name of simplicity, we essentially scrubbed an entire PG at once. The tricky part of efficiently performing a scrub is that comparing the contents of the primary and the replica is only useful if the scans are performed at the same version! Scrubbing while writes are in flight might result in a scrub scanning the replica at version 200 and scanning the primary at 197 because the replica happens to be a bit ahead of the primary. A simple way to ensure that the versions match is simply to stop writes on the entire PG and wait for them to flush before scanning the primary and replica stores. In fact, the Argonaut approach is a bit more sophisticated — we scan the primary and replica collections without stopping writes, and then stop writes to rescan any objects which changed in the meantime. However, for a large PG, that last step could take a long time, so a better approach is needed. Enter ChunkyScrub! In Bobtail, we scrub a PG in chunks, only pausing writes on the set of objects we are currently scrubbing. This way, no object has writes blocked for long.

OSD Internals Refactor

The ceph-osd daemon internals received a bit of a rework as well. As mentioned above, PGs act as the unit of sequencing for object operations. This is reflected in the code: each PG the OSD is responsible for maps onto a PG object. The OSD object’s primary responsibility is to shuffle messages from clients and other OSDs over to the appropriate PG object. A happy consequence of this is that operations on different objects on the same OSD can be processed independently (and in parallel!) as long as the objects are in different PGs. There is, however, one annoying detail which tends to prevent us from fully exploiting this opportunity for parallelism: that pesky OSDMap.

The OSDMap is required for the “CRUSH MAGIC” arrow in the above diagram to work. CRUSH really takes two inputs: a pgid and a description of the cluster. These together determine the OSDs on which the PG will reside. This description is encoded in the OSDMap. Changes to the cluster, such as the death of of an osd, are encoded into a new OSDMap by the ceph-mon cluster and sent out to the OSDs. The maps are given sequential epoch numbers. Essentially every decision within an OSD depends on the contents of the OSDMap. Complicating the situation even further is that OSDMap updates don’t reach all OSDs at the same time. The ceph-mon cluster sends out maps as they are created to a few OSDs, and then the OSDs gossip the new map around to other OSDs as they discover OSDs with old maps. Every OSD-OSD (including regular heartbeats) or OSD-client message includes the sender’s current OSDMap epoch, allowing the receiver to respond with whatever maps the sender is missing. So, how do we handle an OSDMap update arriving while other threads are busy with client requests for various PGs?

Originally, the OSD halted the threads responsible for handling PG requests (including client IO) while updating the global map. This was a useful simplification since each PG might need to update local state due to the map change, and it would be complicated to coordinate that update with in-progress operations. However, it was also a somewhat expensive simplification since halting all IO during the map switchover tends to be costly. Bobtail includes a rework of how the OSD processes PG messages. First, the PG internal code has been reworked to rely as little as possible on global OSD state. Second, each PG has its own notion of the “current” OSDMap epoch distinct from that of other PGs and from the OSD as a whole. Each PG’s internal map state is updated to the current OSDMap epoch before processing a message. The OSD can therefore update its OSDMap related state without bothering the PG threads and then publish the new map epoch atomically for PG thread consumption once it’s ready. The end result of all of this is that the OSD should handle map changes much more efficiently. This might not seem like much, but map changes tend to happen quickly when the cluster is experiencing heavy load due to OSD failures — exactly when you don’t want extraneous overhead!

Filestore Performance

Another area that got a fresh coat of paint is the backend io system synchronization design. The ceph-osd daemon uses standard file systems such as xfs or btrfs as its backing store. However, as you might imagine, it’s much simpler to work in terms of transactions on an abstract data store than to work directly on top of a file system (particularly considering the differences between xfs, btrfs, and ext4). Thus, the ceph-osd daemon talks to the file system via the FIleStore, which presents a uniform transactional interface in terms of objects and flat collections on top of the user’s underlying filesystem.
The journal is crucial to providing these transactional guarantees. In xfs (btrfs is somewhat different), the FileStore writes out each transaction to the journal prior to applying it to the file system. Each write must pass through:

OSD op thread (responsible for handling client requests)
FileStore journal thread (responsible for appending writes to the journal)
FileStore work queue (responsible for applying writes to the backing file sytem)
Messenger (responsible for managing inter-node communication) for the client reply.

It’s crucial to maximize throughput and minimize latency in this pipeline if we want to avoid torpedoing performance. To approach this problem, we took a shiny new server with 192GB of memory and started running benchmarks against the FileStore module in isolation mounted on a ramdisk. We were able to shove small writes through at a rate of around 6k iops. This is pretty good if we plan on running on a ~150iop spinning disk. It is considerably less good if we plan on running on 20k+ iop ssds. So, we went to work. Instrumenting our Mutex object to add up time spent waiting on each lock yielded several promising “problem locks”, each of which, for reasons of simplicity, protected several unrelated structures. Restructuring the code for finer synchronization around these structures, along with disabling in-memory logging, bumped us up to around 22k iops. For the next release, we’ll continue on to attacking latency and throughput bottlenecks in the upper layers of the OSD daemon.

Recovery QOS

One of Ceph’s nicer properties is self-healing. The death of OSD 10 eventually triggers a new OSDMap to be generated with OSD 10 marked down and out, which in turn triggers any PGs which had lived on OSD 10 to rebalance to a new set of OSDs. Of course, there is no escaping the fact that recovering OSD 10’s PGs to new OSDs must involve copying the objects from OSD 10’s PG’s surviving replicas to new OSDs. With Argonaut’s default settings, this looks like a long series of 1MB transfers from surviving replicas to new replicas. So, how might these large transfers interact with, say, the flurry of latency sensitive 4k writes generated by the VMs running on RBD on your cluster?

Argonaut already has some facilities you may be familiar with for limiting the impact of recovery on client workloads. Most prominently, “osd recovery max active” limits the number of concurrent recovery operations any single OSD will start. Regrettably, this only limits the number started at any single OSD. It does not, for example, prevent 20 OSDs from simultaneously pushing “osd recovery max active” objects each to a single OSD you have just added to your cluster! That’s where Bobtail’s new “osd max backfills” configurable comes in. “osd max backfills” defines a limit on how many PGs are allowed to recover to or from a single OSD at any one time.

Argonaut also includes a simple mechanism for prioritizing Messages for processing at the OSD. Each message is tagged with a numerical priority. Messages are processed in order first by priority, and then by time of arrival. So all messages of priority 128 will be processed before any of priority 63. This is useful for some pieces of the OSD. For example, replies from replicas to the primary indicating that the replica has persisted a client op are given a high priority to reduce client op latency since they are quick to process. However, if we give client messages a higher priority than recovery messages using this mechanism, any significant amount of client io will tend to starve recovery. You really don’t want that since the longer a PG goes without re-replicating it’s data, the more likely a second or third OSD death takes it out completely!

Bobtail introduces a more flexible Message prioritization scheme. Messages can be sent such that a message with priority 40 will be allowed through at twice the rate of messages with priority 20, but won’t starve them. This has been leveraged to allow recovery messages to be sent with a lower priority than client io messages without causing starvation. As a nice bonus, consider trying to read an object which has not yet been recovered to the primary. The primary must complete the recovery operation on that object before serving the read. Now, we can give that recovery operation the priority of client io to allow it to bypass any lower priority recovery operations queued at other osds! Future work in this area will focus on reducing the burden of coordinating a PG’s recovery operations on the PG’s primary OSD. This burden still has an impact on client io coming in to that OSD.

So, those are some of the new developments in the OSD. And that’s just the OSD! RBD is getting layering and write-back cache! CephFS is getting substantial stability and performance enhancements! And if we stay on track, our next release will have even more exclamation points! That is, it will if we have time left after we come up with a cephalopod name that starts with “c”…

Atomicity of RESTful radosgw operations

yehuda — Mon, 07 Nov 2011 21:42:26 +0000

A while back we worked on radosgw doing atomic reads and writes.

The first issue was making sure that two or more concurrent writers that write to the same object don’t end up with an inconsistent object. That is the “atomic PUT” issue.

We also wanted to be able to make sure that when one client reads an object via radosgw while another client writes to the same object, the result is consistent. That is, when reading an object a client should get either the old or the new version of the object, and never a mix of the two. That is the “atomic GET” issue.

Radosgw is built directly on top of RADOS and is a prime example of a librados user. The basic issue is that radosgw streams the objects from or to the RADOS objects with a series of relatively small reads or writes. For the atomic PUT and atomic GET we didn’t want to introduce locking. Locking would solve the issue, but implementing it on top of RADOS would not have been trivial, and would have affected scalability and the relative simplicity of the gateway. The Ceph distributed file system implements locking in the metadata server (as part of its POSIX file locking support), and introducing that in the gateway would require holding state on each object and synchronizing it between the different gateway instances. We didn’t want to reimplement the MDS again.

Atomic PUT

When radosgw reads or writes an object it can issue multiple read or write librados requests to the RADOS backend. One RADOS feature is that each single operation is atomic. The problem is that for sufficiently large object (which are not too large in any case) we issue multiple write operations, and could end up with an interleaved object.

The solution for the atomic PUT is to write the object into a temporary object. Once the temp object is completely written, we issue a single librados clone-range operation that atomically clones the entire temp object to the destination. Once the data is there we remove the temp object. This is equivalent to write to a temporary file and renaming it over the target when we finish.

Since the RADOS backend is distributed, we need to make sure that both the temp object and the target object will be located in the same placement group (and on the same OSD). Usually the object location is determined by the object name, but for this purpose we used the “object locator” feature, which allows us to provide alternative string that is fed into the hash function. In this case we use the target object name as the object locator for the temporary object, ensuring that both objects end up on the same placement group on the same node so that the clone operation can work.

Atomic GET

With atomic PUT we know that the objects are consistent. However, this doesn’t help with clients reading when an object is being written. Since there can be multiple librados read operations for a single GET, some of the reads may happen before the object is replaced and some may happen after that, leading to an inconsistent “torn” result.

In addition to atomic operations, RADOS has a nice feature called compound operations which allow you to send a few operations that are bundled together and applied atomically. If one of the operations fail, nothing is applied. We use this for atomic PUT in order to set both data and metadata on the target object in a single atomic operation.

For the atomic GET we introduce an object “tag,” which is a random value that we generate for each PUT and store as an object attribute (xattr). When radosgw writes to an object it first checks for an existing object and fetches its tag (which it can do atomically). If the object exists it clones it to a new object with the tag as a suffix (taking necessary steps to avoid name collisions) and the original object name as the locator. The compound clone operation looks like:

check to see if object tag attribute is
clone to _

The first operation is a guard to make sure that the object hasn’t been rewritten since we first read it. (Had it been rewritten, we need to restart the whole operation and reread the tag.) We put the same guard when we write the new object instance, to make sure that there was no racing operation.

A client that reads the object also starts by reading the tag, and putting the same guard before each subsequent read operation. If the guard fails, the client knows that the object has been rewritten. However, it also knows that since it has been rewritten, the object that it started reading can now be found at _. So, reading of an object named foo looks like this:

read object foo tag -> 123
verify object foo tag is “123″; read object foo (offset = 0, size = 512K) -> ok, read 512K
check object foo tag is “123″; read object foo (offset = 512K, size = 512K) -> not ok, object was replaced
read object foo_123 (offset = 512K, size = 512K) -> ok, read 512K

The final component is an intent log. Since we end up creating multiple instances of the same object under different names, we need to make sure that these object are cleaned up after some reasonable amount of time. We added a log object which we record each such object that needs to be removed. After a sufficient amount of time (however long we expect very slow GETs to still succeed), a process iterates over the log and removes old objects.

RBD Status Update

yehuda — Tue, 11 Oct 2011 02:34:24 +0000

Just a quick update on the current status of RBD.

The main recent development is that librbd (the userspace library) can ack writes immediately (instead of waiting for them to actually commit), to better mimic the behavior of a normal disk.

Why do this? A long long time ago, when you issued a write to a disk, it would ACK the write when the data was written. No more. Now, the ACK means the data is either the drive’s cache or on disk. You don’t know data is safe/durable until you issue a separate flush command. Now RBD behaves similarly: writes are acked immediately (up to some number of bytes, at least), and a flush will wait for all previous writes to commit. The only real difference between this and a real drive cache is that a real drive will try to coalesce small writes into a single operation, while RBD sends them all straight through to the backend cluster.

To make this work with qemu/KVM you need:

Ceph v0.35 or later.
Set the rbd_writeback_window to the number of bytes (something on the order of what you’d expect a physical disk cache to be.. say, 8 MB). This means using a qemu drive string like
```
rbd:rbd/myimage:rbd_writeback_window=8000000
```
You need qemu with commit 7a3f5fe, which wires up the qemu flush function properly. It is not included in v0.15, but should be in the next release.

This is not yet implemented in the kernel RBD driver. As a result, effective performance using that device is still relatively poor. We hope to have similar behavior ready when the v3.2 merge window opens.

Linus vs FUSE

sage — Fri, 08 Jul 2011 21:47:46 +0000

I can’t decide whether Linus is amused or annoyed by the extent to which people hang on his every word, or go nuts over his random rants about this or that. People still talk about his pronouncement about O_DIRECT and tripping monkeys (which has now found a home on the open(2) man page). The latest hullabaloo is about his decree that all FUSE-based file systems are toys.

Clearly, as many have pointed out, calling all such systems “toys” isn’t completely fair. But then it wouldn’t be fun to say it if it were strictly true. There are real systems (big and fast) built on FUSE, just as there are such systems built with Java, Visual BASIC, Cobol, and every other platform/technology we love to mock.

I haven’t seen PLFS come up yet in the discussion, but I think it’s worth mentioning just because it is such a good example of optimizing for the cases that actually matter for your workload. For those not familiar, PLFS (parallel log-structured file system) is a FUSE-based file system built at LANL for their huge many-thousand node clusters that turns all random IO sequential by building a mess of intermediate indices. It sounds like it would be a disaster, but in practice it speeds up their workloads by several orders of magnitude, simply because the underlying parallel file systems on which it is stacked are so bad at those workloads.

Anyway, there are just a few points I wanted to make about the kernel vs userspace file systems, having implemented the Ceph client using both. At the risk of stating the obvious:

There is nothing you can do in userspace that you can’t also do in the kernel. Sure, development can be harder in the kernel, but you have unparalleled access to the system. The only significant technical disadvantage of a kernel implementation is fault isolation: a buggy FUSE-based file system won’t take down the system with it.
Implementation is easier with FUSE. At least for something basic. There are some key problems that are harder to solve because of limitations in the interface.
Memory management is easier in the kernel. AB is right when he says that the memory management and file system need to work together. The problem is that it is difficult to push memory management into userspace when you are not the only tenant on the machine. (I suspect that in most of the big production environments where userspace file systems are used, the fs either is the sole tenant or is given some fixed amount of RAM to work with.) The kernel VM, on the other hand, will apply cache pressure dynamically based on the demands of all users of the system. Trying to do that in userspace is extremely awkward at best.
Managing cache coherency is easier in the kernel. Some people don’t care about this (e.g., see NFS, or any of the “toys” Linus was referring to), but we do. This is mainly a result of the limited FUSE interface. You can probably avoid the issue by simply not using the kernel dentry and page caches and reimplementing it all in userspace. That’s a simple enough approach, but is slow, and fails to leverage years of work invested in the core Linux VFS code.
FUSE may be partly to blame. Jeff Darcy has made the point that many of the FUSE shortcomings aren’t inherent to userspace storage, but artifacts of the current interface and kernel politics. Maybe that’s the case, but that is the world we live in. No file system that doesn’t work on Linux (or maybe *BSD) is relevant. And for what it’s worth, most of the people I see complaining about kernel community intransigence haven’t even tried to work upstream; it’s easier than you think, as long as the code you’re pushing isn’t crap.

Which is better for any given project in the end is probably more of a business decision: technical investment, performance, time to market, ease of deployment. If you’re talking purely about the technical limitations of the environment, however, it’s hard to beat the kernel.

Or, if you can, implement both. It makes these sorts of debates that much more fun.