The Ceph Blog

We’re pushing out v0.30. Highlights include:

• librbd: Fixed race/crash
• mds: misc clustered mds fixes
• mds: misc rename journaling/replay fixes
• mds: fixed flock deadlock when processes die during lock wait
• osd: snaptrimmer fixes, misc races, recovery bugs
• auth: fixed cephx race/crash
• librados: rados bench fix
• librados: flush
• radosgw: multipart uploads
• debian: gceph moved to separate package
• lots of g_conf refactoring, removing of globals, and related cleanup
• qa: lots

The focus this time around continues to be with QA, bug fixes, and cleanup.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.30.tar.gz
• Debian/Ubuntu packages: see http://ceph.newdream.net/wiki/Debian

We’ve released 0.29.1 with a few fixes. The main thing is a fix for a
race condition in librbd that was biting people using rbd with qemu/kvm.

• librbd: fix for race/crash
• osd: fix memory leak
• osd: fix clone size accounting
• mkcephfs: fix ceph.conf reference

Relevant URLs:

• Direct download at: http://ceph.newdream.net/downloads/ceph-0.29.1.tar.gz
• Debian/Ubuntu packages: see http://ceph.newdream.net/wiki/Debian

Ceph v0.29 is ready.  Notable changes since v0.28.2 include

• mds: some fixes for multiple clients accessing the same directory
• obsync: supports rados/rgw backend
• osd: fix bug causing recovering objects to be excluded from object listing
• rados: import/export support for xattrs, incremental updates
• radosgw: misc fixes
• libceph: readdir bug fixes
• osd: fix for various heartbeat failures

Mainly we saw continued stabilization of the OSD peering code, which is now working quite well for us.  For v0.30 we’re continuing to clean up a few OSD corner cases and working on clustered MDS problems.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/downloads/ceph-0.29.tar.gz
• Debian and Ubuntu packages: http://ceph.newdream.net/wiki/Debian

We’ve tagged v0.28.2, which includes several bugs fixed over the past week.  These include:

• mds: crash on unlink (uninitialized variable)
• mds: fix for multiclient writer vs stat hang
• mds: tolerate ENOENT on journal prezeroing
• osd: disable automatic marking of objects as lost (paranoia)
• osd: fix race on pg activation (crash)
• mkcephfs: fix for mon setup

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.28.2.tar.gz
• For Debian and Ubuntu packages, see http://ceph.newdream.net/wiki/Debian

I tagged and uploaded v0.28 a week ago but forgot to blog and email about it. That may be just as well, as there were a number of issues that got fixed last week. I’ll separate out the main items.

Changes for v0.28:

• osd: peering code refactor
• osd: long object name support
• osd: removed fragile class distribution mechanism (now left to admin to distribute .so’s)
• mds: many clustered mds fixes
• libceph: xattr operations, api update
• mon: ‘health’ command more robust
• obsync: many improvements (ACL translation, content-type, attrs, packages)
• rbd: map/unmap kernel devices via ‘rbd map’ and ‘rbd unmap’ commands
• crush: allow – and _ in names, fixed whitespace during compile
• radosgw: many many fixes

Changes for v0.28.1:

• osd: heartbeat exchange fixes (for many ‘wrongly marked down’)
• osd: peering fixes
• daemons: reopen log file on SIGHUP fixed
• mds: fix journal expire crash (after mds restart)
• libceph/cfuse: fix sync write crash

The focus for v0.29 is mainly stability.  The only new features are for obsync (so it can sync directly to rados) and some usability improvements for programatically managing CRUSH maps for larger clusters.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.28.1.tar.gz
• For Debian and Ubuntu packages, see http://ceph.newdream.net/wiki/Debian

We’ve released v0.27.1. This includes a few bugfixes for v0.27, including

• mds: misc journaling and replay bugs
• mds: fixed –reset-journal
• cfuse: fixed device major/minor values
• conf: less noise to stdout
• osd: fixed file descriptor leak from fiemap operation

In short, nothing terribly earth shattering.  v0.28 is about a week away, and will include significant changes in the OSD peering code and more MDS clustering fixes.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.27.1.tar.gz
• For Debian and Ubuntu packages, see http://ceph.newdream.net/wiki/Debian

v0.27 is done!  This mostly bugfixes, cleanups, and incremental improvements.  Notably:

• lots of cleanups in config file loading, handling, to make library behavior sane, warn on config file errors, etc.
• osd: fix out of order ack bug
• mount.ceph: uses kernel keys interface (when available) to pass secrets
• osd, mon: use new syncfs() syscall where available
• librados: compound object operation support
• librbd: snapshot images no longer writeable
• librbd: rollback to snapshot and other misc fixes
• mds: journal replay cleanups, performance, bug fixes
• mds: many clustered mds fixes (mostly with rename and recovery)
• mds: standby-replay mode fixes
• mds: robust lookuphash for better nfs reexport support
• mon: bugfixes with mds takeover
• obsync: synchronize object buckets between s3, directory, swift, rados
• osd: misc recovery fixes
• radosgw: dup bucket creation fixes
• radosgw: many small protocol fixes

As part of the radosgw work we’ve created s3-tests.git, which includes a  bunch of simple tests to verify implementations of the s3 protocol.  See

• git://ceph.newdream.net/git/s3-tests.git

For v0.28 we’re focusing on the OSD cluster, radosgw, and continuing with the MDS clustering fixes.  Sam and Josh are working on a refactor in the  OSD peering code that will make peering more understandable, verifiable, and (we hope) less buggy.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.27.tar.gz
• For Debian and Ubuntu packages, see http://ceph.newdream.net/wiki/Debian

We tagged v0.26 a few days ago.  Changes since the last release include:

• misc build, configure, rpm build fixes
• crypto: support for libnss (which exists in RHEL environments)
• osd: improved throttling
• osd: scrub no longer blocks requests
• osd: vastly improved map update performance
• osd: recovery fixes
• librados, osd: support for object locator strings
• librados: API fixes, extensions
• mds: recovery fix for large directories
• mds: journaling fixes
• mds: rstats fixes
• radosgw: Swift API support.  many fixes

For v0.27 we’re continuing to focus on stabilizing the OSD and radosgw.  There have also been a flurry of bugs found (and fixed!) in the MDS with fsstress from LTP (which, BTW, is a pretty great tool).  As part of this we’re chipping away at the clustered MDS problems as well.  See the current roadmap for the next few intermediate releases and current set of desired 1.0 features.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.26.tar.gz
• For Debian and Ubuntu packages, see http://ceph.newdream.net/wiki/Debian

Are you a Linux kernel programmer interested in a job working on the Ceph distributed file system or btrfs?  Are you a C/C++ developer interested in building scalable, fault tolerant and high performance distributed systems?   Are you an experienced QA engineer with a storage or distributed system background?  DreamHost is hiring Ceph engineers for our Los Angeles and San Francisco offices!

This a bugfix release.  If you’re using librados or librbd, please upgrade, as there are some small API fixes.  If not, there isn’t anything too critical here aside from some OSD recovery corner cases.

• librados: some size_t -> uint64_t type conversions to support large  objects
• librbd: lots of size_t -> uint64_t conversions for 32-bit systems
• cfuse: daemonize fixed
• debian: rbd udev rules
• time conversions fixed for some cases (off by factor of 10)
• osd: pool creation faster
• osd: some recovery fixes
• mds: fixed a mds failover bug (corrupted journal)
• misc small fixes

We’re trying to do a quicker major release cycle as we move toward 1.0, and are aiming for v0.26 in another week and a half.

As always, the thing we are most interested in is help with testing. Please send bug reports to the list or stick them directly in the tracker at http://tracker.newdream.net.

Relevant URLs:

• Direct download at: http://ceph.newdream.net/download/ceph-0.25.1.tar.gz
• For Debian and Ubuntu packages, see http://ceph.newdream.net/wiki/Debian