Ceph Days Southern California

Covering our history with Ceph for the past several years. What we have learned from the experience and what to expect when creating a new platform.

On a long enough timeline, every system will eventually go down. This then creates a situation where an organization experiences an incident. In this session, we will discuss a Ceph incident we experienced, the timeline of events, how it was triggered, what we've learned, and how we've made decisions to improve.

ceph-osd's threading model makes heavy use of thread pools and work queues resulting in latency and cpu overhead which makes it very hard to exploit high performance storage devices. Crimson is a re-implementation of the ceph-osd daemon which replaces those mechanisms with a per-core user space scheduler to keep processing for an IO local to a core in order to minimize context switching overhead. This talk will discuss the current state of crimson in the Reef release.

‘Ceph’, and basically any OpenSource product, is still a “maybe or nah” in LATAM. It’s easier to pay millions for commercial products, where the vendor takes full responsibility from the delivery to the support. It’s also easier to blame the success or failure on the vendor and the quality of the product. LATAM still has a cultural gap regarding OpenSource. A lack of ownership, taking the risk of developing and retaining the required talent in-house to make things work. A challenge that can be tackled with a more specific business analysis of companies in LATAM, including an organizational cultural transition, to demonstrate the benefits of Opensource in the mid-term, putting them more in control and accelerating their digital transformation.

Ceph is a complex piece of software, written in multiple programming languages with dozens of dependencies, many of them bundled in the source tarball.

This talk will go over some of the challenges presented in packaging Ceph. Much of the talk will be applicable to many distributions, as many have similar policies related to bundling of dependencies and internet access at build time. There are some challenges that are specific to Gentoo, such as supporting multiple Python releases at once.

Working together with a major American sports league, we built a multi-site 40 PB active archive housing over 100 years of game video and audio assets by using Ceph as the foundational storage technology. Along the way, we learned many lessons about architecting, deploying, and operationalizing Ceph from the vantage point of a large, modern, and rapidly growing media company. We would like to share our experience and learnings with the community to help others traveling a similar road.

Venue TBA