The Ceph Blog

Featured Post

v0.83 released

Another Ceph development release! This has been a longer cycle, so there has been quite a bit of bug fixing and stabilization in this round. There is also a bunch of packaging fixes for RPM distros (RHEL/CentOS, Fedora, and SUSE) and for systemd. We’ve also added a new librados-striper library from Sebastien Ponce that provides a generic striping API for applications to code to.


  • The experimental keyvaluestore-dev OSD backend had an on-disk format change that prevents existing OSD data from being upgraded. This affects developers and testers only.
  • mon-specific and osd-specific leveldb options have been removed. From this point onward users should use ‘leveldb_‘ generic options and add the options in the appropriate sections of their configuration files. Monitors will still maintain the following monitor-specific defaults:

    leveldb_write_buffer_size = 32*1024*1024 = 33554432 // 32MB leveldb_cache_size = 512*1024*1204 = 536870912 // 512MB leveldb_block_size = 64*1024 = 65536 // 64KB leveldb_compression = false leveldb_log = “”

    OSDs will still maintain the following osd-specific defaults:

    leveldb_log = “”


  • ceph-disk: fix dmcrypt support (Stephen Taylor)
  • cephtool: fix help (Yilong Zhao)
  • cephtool: test cleanup (Joao Eduardo Luis)
  • doc: librados example fixes (Kevin Dalley)
  • doc: many doc updates (John Wilkins)
  • doc: update erasure docs (Loic Dachary, Venky Shankar)
  • filestore: disable use of XFS hint (buggy on old kernels) (Samuel Just)
  • filestore: fix xattr spillout (Greg Farnum, Haomai Wang)
  • keyvaluestore: header cache (Haomai Wang)
  • librados_striper: striping library for librados (Sebastien Ponce)
  • libs3: update to latest (Danny Al-Gaaf)
  • log: fix derr level (Joao Eduardo Luis)
  • logrotate: fix osd log rotation on ubuntu (Sage Weil)
  • mds: fix xattr bug triggered by ACLs (Yan, Zheng)
  • misc memory leaks, cleanups, fixes (Danny Al-Gaaf, Sahid Ferdjaoui)
  • misc suse fixes (Danny Al-Gaaf)
  • misc word size fixes (Kevin Cox)
  • mon: drop mon- and osd- specific leveldb options (Joao Eduardo Luis)
  • mon: ec pool profile fixes (Loic Dachary)
  • mon: fix health down messages (Sage Weil)
  • mon: fix quorum feature check (#8738, Greg Farnum)
  • mon: ‘osd crush reweight-subtree …’ (Sage Weil)
  • mon, osd: relax client EC support requirements (Sage Weil)
  • mon: some instrumentation (Sage Weil)
  • objecter: flag operations that are redirected by caching (Sage Weil)
  • osd: clean up shard_id_t, shard_t (Loic Dachary)
  • osd: fix connection reconnect race (Greg Farnum)
  • osd: fix dumps (Joao Eduardo Luis)
  • osd: fix erasure-code lib initialization (Loic Dachary)
  • osd: fix extent normalization (Adam Crume)
  • osd: fix loopback msgr issue (Ma Jianpeng)
  • osd: fix LSB release parsing (Danny Al-Gaaf)
  • osd: improved backfill priorities (Sage Weil)
  • osd: many many core fixes (Samuel Just)
  • osd, mon: config sanity checks on start (Sage Weil, Joao Eduardo Luis)
  • osd: sharded threadpool to improve parallelism (Somnath Roy)
  • osd: simple io prioritization for scrub (Sage Weil)
  • osd: simple scrub throttling (Sage Weil)
  • osd: tests for bench command (Loic Dachary)
  • osd: use xfs hint less frequently (Ilya Dryomov)
  • pybind/rados: fix small timeouts (John Spray)
  • qa: xfstests updates (Ilya Dryomov)
  • rgw: cache bucket info (Yehuda Sadeh)
  • rgw: cache decoded user info (Yehuda Sadeh)
  • rgw: fix multipart object attr regression (#8452, Yehuda Sadeh)
  • rgw: fix radosgw-admin ‘show log’ command (#8553, Yehuda Sadeh)
  • rgw: fix URL decoding (#8702, Brian Rak)
  • rgw: handle empty extra pool name (Yehuda Sadeh)
  • rpm: do not restart daemons on upgrade (Alfredo Deza)
  • rpm: misc packaging fixes for rhel7 (Sandon Van Ness)
  • rpm: split ceph-common from ceph (Sandon Van Ness)
  • systemd: wrap started daemons in new systemd environment (Sage Weil, Dan Mick)
  • sysvinit: less sensitive to failures (Sage Weil)
  • upstart: increase max open files limit (Sage Weil)


Earlier Posts

v0.80.5 Firefly released

This release fixes a few important bugs in the radosgw and fixes several packaging and environment issues, including OSD log rotation, systemd environments, and daemon restarts on upgrade.

We recommend that all v0.80.x Firefly users upgrade, particularly if they are using upstart, systemd, or radosgw.


  • ceph-dencoder: do not needlessly link to librgw, librados, etc. (Sage Weil)
  • do not needlessly link binaries to leveldb (Sage Weil)
  • mon: fix mon crash when no auth keys are present (#8851, Joao Eduaro Luis)
  • osd: fix cleanup (and avoid occasional crash) during shutdown (#7981, Sage Weil)
  • osd: fix log rotation under upstart (Sage Weil)
  • rgw: fix multipart upload when object has irregular size (#8846, Yehuda Sadeh, Sylvain Munaut)
  • rgw: improve bucket listing S3 compatibility (#8858, Yehuda Sadeh)
  • rgw: improve delimited bucket listing (Yehuda Sadeh)
  • rpm: do not restart daemons on upgrade (#8849, Alfredo Deza)

For more detailed information, see the complete changelog.


Lots Going on with Ceph

While we knew that after the acquisition of Inktank life would accelerate again, it seems like the Ceph community is quickly approaching ludicrous speed, and it shows no sign of slowing down. We have had some amazing participation in the various endeavors, but it would be completely understandable if you had missed something amidst the avalanche of Ceph-related news.

Just in case something flew by you, I wanted to take a few minutes to recap some of the highlights of recent history. If you would like to keep a closer eye on what has been going on feel free to follow one (or all!) of our informational feeds:

Twitter || Facebook || Google+ || Lists/IRC

ludicrous speed

read more…

OSCON has arrived (although if you came in for the Ceph tutorial session that’s old news to you)! As a part of our participation in OSCON, and as a way to celebrate the fact that Ceph turned 10 years old this year, we have decided to have our party be a distributed one.

We would love to have our users send us pictures of whatever they might be doing to celebrate the 10th anniversary of ceph. Are you busy racking in 3 petabytes of storage to add to your Ceph cluster? Did you create a culinary masterpiece in the form of a squid cake? Are you sitting alone in the middle of the OSCON show floor with a party hat and a cupcake? We want to see! As thanks for sharing your birthday celebration efforts with the community we’ll be picking one lucky winner to receive a desktop Ceph test cluster built by our very own Mark Nelson (Ceph performance guru extraordinaire!).

While the cluster wont break any speed records, and only a madman would use it for anything even remotely production ready, it will give you a Ceph cluster to play with and can sit on your desk to invoke feelings of envy in your coworkers. For more details check out the (new) contest page on the Ceph wiki. If you have any questions please contact me or just tweet @Ceph. Thanks, and happy birthday to Ceph!

scuttlemonkey out

Ceph is coming back to OSCON next week (July 20-24 in Portland, OR). The difference however, is that this year we need two digits to tell people how old we are. Stop by for some mild festivities at the Ceph booth (P2) as we share cupcakes, and t-shirts that salute the hard work of all our committers since day one.

Photo credit:

Originally we had much bigger plans for a guerrilla show floor birthday spectacle, but summoning Cthulhu just seemed like far too much work and could have been potentially disruptive to our fellow attendees. So instead we’re just going to enjoy hanging out with our community and sharing memories and calories.

read more…

v0.80.4 Firefly released

This Firefly point release fixes an potential data corruption problem when ceph-osd daemons run on top of XFS and service Firefly librbd clients. A recently added allocation hint that RBD utilizes triggers an XFS bug on some kernels (Linux 3.2, and likely others) that leads to data corruption and deep-scrub errors (and inconsistent PGs). This release avoids the situation by disabling the allocation hint until we can validate which kernels are affected and/or are known to be safe to use the hint on.

We recommend that all v0.80.x Firefly users urgently upgrade, especially if they are using RBD.


  • osd: disable XFS extsize hint by default (#8830, Samuel Just)
  • rgw: fix extra data pool default name (Yehuda Sadeh)

For more detailed information, see the complete changelog.



v0.80.3 Firefly released


This is the third Firefly point release. It includes a single fix for a radosgw regression that was discovered in v0.80.2 right after it was released.

We recommand that all v0.80.x Firefly users upgrade.


  • radosgw: fix regression in manifest decoding (#8804, Sage Weil)

For more detailed information, see the complete changelog.


This is the second Firefly point release. It contains a range of important fixes, including several bugs in the OSD cache tiering, some compatibility checks that affect upgrade situations, several radosgw bugs, and an irritating and unnecessary feature bit check that prevents older clients from communicating with a cluster with any erasure coded pools.

One someone large change in this point release is that the ceph RPM package is separated into a ceph and ceph-common package, similar to Debian. The ceph-common package contains just the client libraries without any of the server-side daemons.

We recommend that all v0.80.x Firefly users skip this release and use v0.80.3.


  • ceph-disk: better debug logging (Alfredo Deza)
  • ceph-disk: fix preparation of OSDs with dmcrypt (#6700, Stephen F Taylor)
  • ceph-disk: partprobe on prepare to fix dm-crypt (#6966, Eric Eastman)
  • do not require ERASURE_CODE feature from clients (#8556, Sage Weil)
  • libcephfs-java: build with older JNI headers (Greg Farnum)
  • libcephfs-java: fix build with gcj-jdk (Dmitry Smirnov)
  • librados: fix osd op tid for redirected ops (#7588, Samuel Just)
  • librados: fix rados_pool_list buffer bounds checks (#8447, Sage Weil)
  • librados: resend ops when pool overlay changes (#8305, Sage Weil)
  • librbd, ceph-fuse: reduce CPU overhead for clean object check in cache (Haomai Wang)
  • mon: allow deletion of cephfs pools (John Spray)
  • mon: fix default pool ruleset choice (#8373, John Spray)
  • mon: fix health summary for mon low disk warning (Sage Weil)
  • mon: fix ‘osd pool set <pool> cache_target_full_ratio’ (Geoffrey Hartz)
  • mon: fix quorum feature check (Greg Farnum)
  • mon: fix request forwarding in mixed firefly+dumpling clusters 9#8727, Joao Eduardo Luis)
  • mon: fix rule vs ruleset check in ‘osd pool set … crush_ruleset’ command (John Spray)
  • mon: make osd ‘down’ count accurate (Sage Weil)
  • mon: set ‘next commit’ in primary-affinity reply (Ilya Dryomov)
  • mon: verify CRUSH features are supported by all mons (#8738, Greg Farnum)
  • msgr: fix sequence negotiation during connection reset (Guang Yang)
  • osd: block scrub on blocked objects (#8011, Samuel Just)
  • osd: call XFS hint ioctl less often (#8241, Ilya Dryomov)
  • osd: copy xattr spill out marker on clone (Haomai Wang)
  • osd: fix flush of snapped objects (#8334, Samuel Just)
  • osd: fix hashindex restart of merge operation (#8332, Samuel Just)
  • osd: fix osdmap subscription bug causing startup hang (Greg Farnum)
  • osd: fix potential null deref (#8328, Sage Weil)
  • osd: fix shutdown race (#8319, Sage Weil)
  • osd: handle ‘none’ in CRUSH results properly during peering (#8507, Samuel Just)
  • osd: set no spill out marker on new objects (Greg Farnum)
  • osd: skip op ordering debug checks on tiered pools (#8380, Sage Weil)
  • rados: enforce ‘put’ alignment (Lluis Pamies-Juarez)
  • rest-api: fix for ‘rx’ commands (Ailing Zhang)
  • rgw: calc user manifest etag and fix check (#8169, #8436, Yehuda Sadeh)
  • rgw: fetch attrs on multipart completion (#8452, Yehuda Sadeh, Sylvain Munaut)
  • rgw: fix buffer overflow for long instance ids (#8608, Yehuda Sadeh)
  • rgw: fix entity permission check on metadata put (#8428, Yehuda Sadeh)
  • rgw: fix multipart retry race (#8269, Yehuda Sadeh)
  • rpm: split ceph into ceph and ceph-common RPMs (Sandon Van Ness, Dan Mick)
  • sysvinit: continue startin daemons after failure doing mount (#8554, Sage Weil)

For more detailed information, see the complete changelog.


v0.82 released

This is the second post-firefly development release. It includes a range of bug fixes and some usability improvements. There are some MDS debugging and diagnostic tools, an improved ‘ceph df’, and some OSD backend refactoring and cleanup.


  • ceph-brag: add tox tests (Alfredo Deza)
  • common: perfcounters now use atomics and go faster (Sage Weil)
  • doc: CRUSH updates (John Wilkins)
  • doc: osd primary affinity (John Wilkins)
  • doc: pool quotas (John Wilkins)
  • doc: pre-flight doc improvements (Kevin Dalley)
  • doc: switch to an unencumbered font (Ross Turk)
  • doc: update openstack docs (Josh Durgin)
  • fix hppa arch build (Dmitry Smirnov)
  • init-ceph: continue starting other daemons on crush or mount failure (#8343, Sage Weil)
  • keyvaluestore: fix hint crash (#8381, Haomai Wang)
  • libcephfs-java: build against older JNI headers (Greg Farnum)
  • librados: fix rados_pool_list bounds checks (Sage Weil)
  • mds: cephfs-journal-tool (John Spray)
  • mds: improve Journaler on-disk format (John Spray)
  • mds, libcephfs: use client timestamp for mtime/ctime (Sage Weil)
  • mds: misc encoding improvements (John Spray)
  • mds: misc fixes for multi-mds (Yan, Zheng)
  • mds: OPTracker integration, dump_ops_in_flight (Greg Farnum)
  • misc cleanup (Christophe Courtaut)
  • mon: fix default replication pool ruleset choice (#8373, John Spray)
  • mon: fix set cache_target_full_ratio (#8440, Geoffrey Hartz)
  • mon: include per-pool ‘max avail’ in df output (Sage Weil)
  • mon: prevent EC pools from being used with cephfs (Joao Eduardo Luis)
  • mon: restore original weight when auto-marked out OSDs restart (Sage Weil)
  • mon: use msg header tid for MMonGetVersionReply (Ilya Dryomov)
  • osd: fix bogus assert during OSD shutdown (Sage Weil)
  • osd: fix clone deletion case (#8334, Sam Just)
  • osd: fix filestore removal corner case (#8332, Sam Just)
  • osd: fix hang waiting for osdmap (#8338, Greg Farnum)
  • osd: fix interval check corner case during peering (#8104, Sam Just)
  • osd: fix journal-less operation (Sage Weil)
  • osd: include backend information in metadata reported to mon (Sage Weil)
  • rest-api: fix help (Ailing Zhang)
  • rgw: check entity permission for put_metadata (#8428, Yehuda Sadeh)


Wow, the last few weeks have been very busy for the Ceph team! While it may have been a few weeks ago, many of us are still feeling the excitement of the most recent OpenStack Summit, and many of us weren’t even there! With the Firefly release still cooling in the packaging repos, there certainly was a lot for the Ceph Community to be excited about. However it was the voices of the users that once again made us the most excited. The results from another OpenStack Foundation OpenStack User Survey was made public at the conference and it has extremely encouraging results for Ceph open source and software-defined storage alike.

The OpenStack Foundation survey results showed that among OpenStack users surveyed, Ceph is a leader across the board for block storage in clouds of all stages. Ceph is cited as one of the leading distributed block storage technology in all three categories.

read more…

v0.81 released

This is the first development release since Firefly. It includes a lot of work that we delayed merging while stabilizing things. Lots of new functionality, as well as several fixes that are baking a bit before getting backported.


  • CephFS support for the legacy anchor table has finally been removed. Users with file systems created before firefly should ensure that inodes with multiple hard links are modifiedprior to the upgrade to ensure that the backtraces are written properly. For example:
    sudo find /mnt/cephfs -type f -links +1 -exec touch \{\} \;
  • Disallow nonsensical ‘tier cache-mode’ transitions. From this point onward, ‘writeback’ can only transition to ‘forward’ and ‘forward’ can transition to 1) ‘writeback’ if there are dirty objects, or 2) any if there are no dirty objects.

read more…

Page 1 of 1312345...10...Last »
© 2013, Inktank Storage, Inc.. All rights reserved.