There are a number of products that Ceph plugs into, and many of them are built on top of open source projects. One of the great parts about Ceph is that a single cluster can service all of your data storage needs, especially as it relates to the Citrix portfolio. Much like Linux in the datacenter, it’s only a matter of time before Open Source becomes the dominant force in this last bastion of proprietary-driven infrastructure.

Let’s take a look at how Ceph can work with:

• Apache CloudStack / Citrix CloudPlatform
• Citrix CloudPortal
• Citrix XenDesktop
• Citrix Sharefile
• Xen / XenServer

CloudStack/CloudPlatform

As of last year, Citrix handed over CloudStack to the Apache community and it came out of incubation a few months ago. There is a really vibrant community around it and Citrix is a major part of that community. Citrix CloudPlatform is a commercial version of Apache Cloudstack, fully tested and supported by Citrix.

Rather than separate storage into APIs by type as in Openstack (Cinder for Block and Swift for Object), CloudStack/CloudPlatform have a concept of Primary and Secondary Storage which aggregate the different storage options based on purpose:

• Primary Storage: is the storage used by a running virtual machine for its live filesystem. It can be delivered by either local,NFS, iSCSI, Fiber Channel or RBD storage.
• Secondary Storage: is used for storing snapshots, ISOs, templates and can be delivered via local, NFS, iSCSI or, as of 4.1, S3 or Swift compatible object storage.

With great work being done by the Ceph community (thanks Wido!), version 4.0 saw the Ceph Block Device (RBD) enabled as a Primary Storage option. This gives you most of Ceph’s powerful features like copy-on-write cloning and thin provisioning with snapshotting and cloning coming in 4.2, which is due out in June. Also, as of 4.2,  Secondary Storage via the RADOS Gateway (RGW) using the generic S3/Swift API but *without* using a NFS server will also be possible.

CloudPortal

Citrix CloudPortal is a web UI that allows end users to request services from underlying IT resources. It is designed to hide the underlying technology from users who merely request the service or product they need and makes the interaction between users and IT departments feel like the interaction between a customer and a service provider.

An administrator configures what services are available, which are then presented as a catalog to end-users to choose from. It is extensible, so can be integrated with any type of underlying software, however it’s initial connection to Ceph is via its being a method of interacting with CloudPlatform.

CloudPortal interacts with CloudPlatform to discover what resources are available so it can present them to end-users via the web UI. As such, the fact that Ceph is storage in CloudPlatform is transparent to the end-user of CloudPortal and they merely make a request for storage capacity.

CloudPortal is currently able to request Primary storage (delivered by Ceph) in CloudPlatform. Pending development work, CloudPortal will also be able to request Secondary Storage (delivered as object storage) which is delivered by Ceph.

XenDesktop (VDI)

XenDesktop is one of Citrix’s flagship products and provides a Virtual Desktop Infrastructure (VDI). VDI is a means to simplify the administration of hundreds or thousands of desktops by an IT department. Rather than IT managing each individual desktop or laptop, the desktops are hosted on a server and delivered over the network to a locked-down end-user device.

In this way, the end-user device requires almost no software management. The desktop, its applications, and content are stored remotely where it can be upgraded, patched and backed-up by the IT department in a single location.

As part of Project Avalon, Citrix has been working to enable XenDesktop on top of cloud infrastructure, including their CloudPlatform product. This means it is now possible to use Ceph as the storage system for desktop virtual machines. Given that storage is one of the biggest administrative pain points for VDI, Ceph’s scale-out nature and its copy-on-write and thin-provisioning features, which lead to fast boot times, are a huge advantage.

Sharefile

Sharefile offers an on-premise/behind the firewall enterprise Dropbox-style user-interface for IT departments. The administrator can configure a number of backends to store the data both on-premise or in the cloud (backed by Amazon S3).

Currently the easiest way to take advantage of Ceph with Sharefile is via CephFS using a Samba gateway (for CIFS). However, since CephFS hasn’t been given the requisite QA love to receive an Inktank blessing, it is worth noting that Citrix is working on modifying Sharefile to use any S3-compatible object storage. This would allow RGW to plug directly in to Sharefile.

Xen/XenServer

As of the latest versions of the open-source Xen code, there is direct use of the upstream RBD-enabled Qemu code. This means that the Xen hypervisor can now natively access RBD images similar to the integration with KVM. Other members of the Ceph community (thanks Sylvain!) have also been working on integrating RBD into the blktap driver that the Xen ecosystem uses, including XenServer (Citrix’s free & commercial product). At the moment, there is no date for which version of XenServer these changes will appear in, but if you are interested be sure to let us or the folks at Citrix know!

Conclusion

With OpenStack, CloudStack, and raw storage Ceph really is becoming the new black, ready to go with any shade of integration you might want to throw at it. It’s great to see open source projects like Ceph redefining the storage space and providing unified answers to what used to be very proprietary questions. The shift towards software-defined storage and a more horizontally scalable approach is clear, however, and is even starting to show in some fancy marketing retreads as the big proprietary folks work to get caught up.

Of course, there are more integrations happening every day. Other cloud flavors like OpenNebula, Ganeti, and Proxmox have Ceph support as well as any number of applications available from hypervisor- and kernel-level integrations. If you have integration work that you would like to share, be sure to hit up our community team. We look forward to seeing how Open Source, and the associated communities, will shape the storage industry going forward.

scuttlemonkey out

Summit Date: 07 May @ 8a – 2p PDT (GMT -8)

The community has already been hard at work generating blueprints for evaluation and discussion. The next step is to discuss those blueprints and figure out how it will all fit together for the “Dumpling” release. Read on for details on how we plan on running the summit, and how you can be prepared to participate.

Blueprints

Even though this was our first time going through the blueprint process (and the process still is being defined as we go), we still managed to generate 16 blueprints! There is a great variety of features and changes the community would like to see, and a number of different stakeholders who are offering to contribute.

For the full list of blueprints, see the Blueprint page on the wiki.

The next step is to decide which of these changes will make it in to the “Dumpling” release, and how it all fits together. This is where the summit comes in!

Summit

Sage and team have spent some time reviewing the available blueprints and have come up with a discussion schedule for the Summit. These discussions should accomplish the following:

• Identify a blueprint owner, if necessary
• Discuss the possible ways the blueprint might be implemented
• Identify the necessary work items and record them in the session's etherpad
• Determine whether the work can be completed before the Cuttlefish feature freeze
• Identify interested parties and discuss ownership of work items

If you submitted a blueprint that made it into the schedule, please be prepared to lead the discussion. If you can't make it to the summit, please let us know. We'll be reaching out to individual blueprint owners to confirm.

Tools

We chose to make this a small, online event so that more people could attend and contribute in a meaningful way without having to disrupt their lives with travel time and cost.  In order to facilitate these virtual discussions, we will be using a few tools.

Google Hangouts – The video conferencing setup we are using is a broadcasted Google Hangout. This means that up to 10 "speakers" can participate, while any number of community members may spectate from the public URL. The talks will also be captured automatically as YouTube videos and published to our channel for later consumption.  If you want to be one of the 10 "speakers" in the hangout, make sure to add your name to the Interested Parties blueprint field.

Etherpad – Each session will have an associated etherpad for collaborative note taking during the session. This should allow us to capture architectural decisions, issues, work items, and other notes during the sessions.

IRC – Each track will have its own logged IRC channel for discussion. This will allow asyncronous discussion, offsite-links, and questions to the "speakers" in a format that can be logged and posted for later consumption.

All of these tools will go live on the day of the summit so as not to have spam content or general indexing done before then.  To find them, click the links in the schedule grid.

Conclusion

Everything is now set to go live next week for the summit. The tools should all go live on the day of the summit, so keep an eye on that summit page for the appropriate participation links and make sure you join the IRC channel a little before it gets started for last minute notes or changes. Thanks to everyone who has been working hard building blueprints and volunteering their time and energy to make Ceph the greatest storage platform available. See you next week!

scuttlemonkey out

Three different non-Inktank groups have stepped up and volunteered to be resident “super-geeks” and help the community. Since we had such a great response we are relaunching this effort as “Geek on Duty” and tweaking our help page a bit to make it easier for people to get the type of assistance that is most appropriate to their needs.

What it is

While our community is quite active at all hours of the day and night, we know that it is often difficult to gauge when various people may be available due to timezone constraints. Our hope was that by providing office hours people would be able to plan when they would allocate time to wrapping their brains around Ceph.

A number of people have expressed that while they had questions, they often didn’t know how to encapsulate them into an asyncronous communication medium like email and wanted to “talk it out.” Knowing that someone would be available for a conversation has made this a much better experience.

What it isn’t

While having access to a “Geek on Duty” may be very helpful in gaining a rapid understanding of Ceph and standing up your first cluster, it’s important to remember that this is not support. Community help should be considered more along the lines of helpful water cooler chatter since it doesn’t come with guarantees, SLAs, or things of that nature.

Once you have moved into the realm of production deployment (or a pre-production setup getting ready for production) it’s probably worth exploring professional support offerings from places like Inktank. Professional support offers you the peace-of-mind of 24/7 support with SLA assurances and ensures that someone will be at the other end of the phone when you pick up.

Other Resources

While the people involved in helping you to adopt Ceph are absolutely invaluable it’s important not to forget that there are many other resources available to you. The Ceph documentation is becoming quite robust, we just relaunched the Ceph wiki, and Inktank has a wide range of resources available including whitepapers, case studies, presentations, and other Ceph-related information.

Conclusion

I think the bottom line here is that the Ceph community is awesome! There are many different places that you can get help, support, or consulting to ensure your deployment of Ceph is smooth and painless. If you have questions or would like to get more involved with Ceph please don’t hesitate to drop us a line at community@ceph.com. Most importantly we want to thank our community for being awesome, keep up the great work.

scuttlemonkey out

At the end of February, one of the OpenNebula developers reached out to let us know that their integration work with Ceph was nearly complete. Below you can find a brief overview of how Ceph behaves in an OpenNebula environment, as well as a link to how to get it set up. Read on for details!

scuttlemonkey: It’s worth noting that the 4.0 release is still in beta and there was a bug discovered in the Ceph driver. Thankfully this has a workaround noted in the doc as well as a fix already committed for the release. If you have any issues feel free to stop by the #opennebula channel and let them know.

OpenNebula continues with its growing support of new storage technologies. OpenNebula 4.0 comes with a fresh integration with Ceph, an impressive distributed object store and file system.

OpenNebula provides an interface for Ceph RBDs (RADOS Block Device), which allows registering images directly in a Ceph pool, and running VMs using that backend.

There is an extensive Ceph for OpenNebula guide, but it can be summarized as follows:

• OpenNebula worker nodes should be part of a working Ceph cluster.
• The ”one” Ceph pool should be available (the name is configurable).
• Use Libvirt/KVM as the hypervisor. Xen is not yet supported.

Once we have that up and running using it is extremely simple!

1. Make sure we have the ”one” Ceph pool

   $ ceph osd lspools
   0 data,1 metadata,2 rbd,3 one,
   

2. Create a Ceph datastore

   $ cat ceph.one
   NAME      = ceph
   DS_MAD    = ceph
   TM_MAD    = ceph
   
   DISK_TYPE = RBD
   
   HOST = ceph0
   POOL_NAME = one
   SAFE_DIRS="/"
   
   $ onedatastore create ceph.one
   ID: 101
   

3. Register an image

   $ oneimage create --name centos-6.4 --path /tmp/centos-6.4.img -d ceph
   ID: 4
   

4. Run your VM

   $ onevm create --name centos-6.4 --cpu 1 --memory 768 --nic 1 --disk centos-6.4 --nic net_172
   ID: 10
   

What happens behind the scenes is that OpenNebula interacts with the Ceph cluster and clones the base image. The Libvirt/KVM deployment file uses that clone image as the OS:

    <disk type='network' device='disk'>
        <source protocol='rbd' name='one/one-4-10-0' />
        <target dev='hdb' bus='ide'/>
    </disk>

All the image handling and manipulation (cloning, renaming, removing, etc…) is performed in a specific server defined in the datastore template, in our case ”HOST = ceph0”, using the ”rbd” capabilities of ”qemu-img” the
registration of new images.

Thanks to the OpenNebula team for hammering out this integration, we love to see new use cases for Ceph!

If you have your own use case story, we would love to hear about it (and share it with the world). Feel free to drop us your thoughts in the form of a suggested blog title, raw data that you wish us to write some prose about, or a fully formed blog post that we can push out to the community. Thanks, and happy Ceph-ing.

(cross-posted on the OpenNebula blog as well)

scuttlemonkey out

New Apache projects start in the Incubator, where they work to build a strong, multi-firm community and adopt the governance practices and philosophies common to all Apache projects. Anyone familiar with Apache knows that this is no small feat; while the end result is worth the struggle, adopting the Apache Way can be a long road.

We salute the Apache CloudStack project for this accomplishment, and we look forward to building amazing things together in the years to come. On behalf of the entire Ceph team, congratulations!

Hi!

As the Ceph community grows, it’s important that help is available to new
users. Up to this point, Inktank engineers have been monitoring the channel
and mailing list as time permits. That made sense when there wasn’t much foot
traffic, but we have a lot now! We’d like to introduce something a bit more
predictable.

Inktank is establishing “office hours” for IRC and the ceph-users mailing list.
During these hours, an Inktank engineer will be focused on answering questions
and providing guidance to community users of Ceph. Our first office hours will
take place tomorrow [14 MAR 2013].

Inktank office hours are posted on the new Getting Help page:
http://ceph.com/resources/getting-help

Naturally, folks from Inktank will still be around outside of those hours,
although they may not be dedicated to the task of helping others. It’s
important to keep in mind that this isn’t tech support and it doesn’t come with
any of the guarantees you would expect from a professional support contract.

You may have noticed that the Getting Help page has our Inktank logo on it. I
would like to do the same for any organization or individual who is interested
in establishing office hours of their own. There are plenty of hours left –
drop a line to community@ceph.com if you’re interested!

Cheers,
Ross

–
Ross Turk
Community, Inktank

@rossturk @inktank @ceph

As the message states, if you are interested in taking part in these office hours (either as an individual, or an organization) please drop our community team a line and we’ll make sure your hours are added to the getting help page. Thanks!

scuttlemonkey out

The release schedule can be divided into four different flavors: point releases, development (interim) releases, stable releases, and long term support (LTS) releases. You can expect these releases to appears on the following periodic schedule:

• Point Releases — ad hoc
• Interim Releases — every 2 weeks
• Stable Releases — every 3 months
• LTS Releases — coming soon!

Point Releases

Point releases are generated on an ad hoc basis for the purposes of critical bug or security fixes. We currently only generate point releases on stable releases for a period of 3-6 months, though this may become longer for certain versions included in downstream distros. Since they are only generated on an as-needed basis they will be commicated via all blog, mailing list, and social media channels as appropriate.

Interim Releases

Every two weeks (approximately) you can expect the latest development work to be aggregated and made available to the community. We wanted a very aggressive schedule here to make sure that everyone building things with, or associated with, Ceph had access to the latest and greatest work by the core Ceph developers. Primarily what you will see in these releases will be bug fixes, patches, and smaller feature additions. Larger features may appear in an interim release, however that feature should not be considered production ready until the next stable release.

These announcements will appear on the blog, and the ceph-devel mailing list with a release number and a summary of changes. Keep your eyes peeled!

Stable Releases

Given the amount of work that has gone into the testing and QA procedures, stable releases can be done on a much more reliable (and frequent!) basis. Much of this work has been directed at our primary testing tool, Teuthology. Originally written as a “test my nightlies” kind of tool, it has been updated to run installations as a task, run longer term testing (instead of just compile-type tests), include more distro-specific testing, and allow for a wider range of testing parameters.

Given the amount of crossover between Inktank and Ceph development it’s also worth noting that Inktank’s testing lab has grown from 160-ish machines to around 286 machines. That plus the amount of rigorous testing we see from the Ceph community means that releases will be put through a lot more paces to be considered “stable” and will have a more traditional feature freeze, QA period, and release candidate lifecycle. The dates of this lifecycle should be available soon via a new Ceph wiki to give upstream users more visibility.

Stable releases will include larger feature changes, bug fixes, and stability from much more exhaustive testing. These releases will function as a “roll up” of all interim releases since the last stable version and where new features that have been added can be considered production-ready. These releases will be featured in our blog, mailing lists, as well as being featured on Ceph.com in our Downloads resource section. Each stable release will have a name associated with a different type of cephalopod corresponding to the appropriate letter of release (“stable” releases thusfar: argonaut, bobtail).

LTS Releases

Given the pace of development, the incredibly wide-range of use cases Ceph is being thrown at, and especially the amount of change in the CephFS portion of Ceph, we don’t feel that having LTS releases quite makes sense yet. However, all of the work above is laying a very solid groundwork for us to make LTS guarantees in the very near future. Of course we’ll be able to get there even faster with your help!

Getting Involved

The more testing and bug reports we see from the community the more sure we’ll be that all angles possible are being considered and tested for, and the sooner we’ll be able to start making even more guarantees than we do now. A great example of this would be relatively new committer, Loic Dachary who has done a bunch of great work around building out unit tests. We love it when new users show up and become prolific, and would love to help anyone interested in contributing, to succeed.

There are a number of different ways that users can get involved, from writing code to just running a test cluster on the latest-and-greatest code and everything in between! If you are interested in lending a hand, feel free to drop us a note via our mailing lists or IRC.

Conclusion

While this may seem like minor news, we’re quite excited by the ramifications of what this means for Ceph. We are very happy with how production clusters have been performing and want to ensure that all of our users, from vast enterprise deployments down to a single 5-minute quickstart, are able to expect a high degree of stability and reliability going forward. The level of community support for testing and development has been amazing, so we also would like to thank all of our contributors, testers, and users for their insight and bug reports! Happy Ceph-ing.

scuttlemonkey out

From February 13-18, we ran our very first Ceph Census. The purpose of this survey was to get a sense of how many Ceph clusters are out there, what they are used for, and which technologies they are used alongside.

The Census was announced on the ceph-devel and ceph-users mailing lists and a link was placed in the topic of the #ceph IRC channel. There were 10 questions in total. The survey was anonymous by default, although people could provide their email address if they chose. In total, we received data from 81 respondents.

Raw responses (without email addresses and other personally-identifiable information) are available at the bottom of this post. Between here and there is my attempt to summarize the most important data. Some questions were optional and some allowed for multiple answers, so the number of responses for each question was often more or less than 81. Enjoy!

• It’s close to an even three-way split between those who are assessing Ceph, those with concrete production plans, and those in production.

  The community reported 21 production clusters with a combined raw storage of 1,154TB. Apparently the team at DreamHost didn’t participate in the Census; DreamObjects alone is over 3PB!

  Pre-production clusters represent a total raw storage of 2,466TB (excluding a reported 20PB cluster.)

• Answer	#
  Assessment / Investigation	36
  Pre-Production	24
  Production	21

• The total amount of storage reported was 5,635TB, and most of it is in clusters with less than 50TB. The average cluster size is just over 72TB.

  Since this question allowed for a free-form text response, I converted each response into TB. If a range was specified, I chose the lower number.

  Of the two largest responses, one was the mysterious 20PB pre-production cluster I mentioned above that provides storage for an OpenStack deployment. The other was a 1PB cluster in pre-production at GRNET SA.

• Answer	#
  <= 10TB	28
  11-50TB	26
  51-100TB	11
  101-200TB	6
  201-500TB	6
  > 500TB	2

• It’s no surprise that half of the reported Ceph clusters are being used to provide storage for cloud deployments. It is interesting, though, that private cloud deployments represent a far larger set of clusters than public ones.

  I didn’t anticipate so much interest in Ceph for backup and archival, but I should have – Ceph’s low cost per gig and ease of expansion make it great for that.

  I am pleased to see big data as a popular use case as well. Many open source distributed filesystems can be used to replace HDFS, and Ceph is no exception.

• Answer	#
  Private Cloud	53
  Backup / Archival	29
  Public Cloud	23
  Big Data	20
  HPC	13
  Legacy	6

• This was kind of an odd question because the “client” OS only matters for some use cases. It doesn’t really matter what OS a REST client is running, for example, but it matters a lot for clients of Ceph’s block and file interfaces.

  Even so, Ubuntu has a substantial lead. Top among the “Other” responses were Gentoo and SLES.

• Answer	#
  Ubuntu	45
  Debian	25
  CentOS	15
  RHEL	12
  Other	21

• On the server-side, Ubuntu is king. Over half of those polled said they were currently running, or planning to run, their clusters on Ubuntu.

  We worked hard early last year to make sure that the Ceph experience on Ubuntu was great, and similar efforts are currently being put into the other major distributions.

  Ubuntu and Debian combined (the apt-get cabal!) account for all but two of the production clusters reflected in this Census.

• Answer	#
  Ubuntu	43
  Debian	15
  CentOS	7
  RHEL	3
  Other	12

• This was kind of a surprise! Actually, two surprises.

  First, OpenStack is the most dominant cloud stack. Integrations with Apache CloudStack, ProxMox, and others have been generating interest – I wouldn’t be surprised to see a more even distribution in the next Census.

  The second surprise is that most respondents use no cloud stack at all…even though the #1 and #3 use cases were cloud deployments!

  The top responses under “Other” were Ganeti and VMWare vCloud.

• Answer	#
  None	35
  OpenStack	17
  ProxMox	6
  OpenNebula	5
  CloudStack	4
  Other	11

So! That concludes our first Ceph Census. I think it was incredibly worthwhile, and I appreciate the participation of all those involved! Full results can be downloaded in CSV here.

We hope to repeat this Census regularly, and we’ll continue to publish results. Until next time!

Cheers,

Ross

As always, if any of you have interesting things that you have done with Ceph we always want to hear about it. Feel free to send a link to @Ceph or email it to our Community alias. Now enjoy this week’s slice of deployment goodness.

Effortless deployment and scaling of a Ceph cluster

In this blog post, we explain how to deploy and scale a cluster hosting a distributed object store and file system called Ceph in the Cloud using ComodIT’s orchestration possibilities. Orchestration was already illustrated in a previous post with a web cluster hosting a WordPress blog. We successfully tested the deployment of Ceph cluster on Amazon EC2 and Openstack, however it should be possible to use any platform supported by ComodIT (Eucalyptus, Rackspace, etc.) or even deploy the cluster on a bare-metal infrastructure.

scuttlemonkey: I followed the install guide below and have a few notes that may help others in their testing. Keep in mind that this demo was put together as a proof-of-concept, and not a robust deployment script. ComodIT plans on improving the script over time, but I think this shows a nice bit of early functionality for evaluation.

The ceph orchestration scripts are available in the ComodIT repository.

If you have other orchestration use cases, we’d be happy to help you out. Contact us and let’s discuss it.

Ceph clusters

A typical Ceph cluster is composed of several instances of each of these services:

• monitors (MONs) which manage the cluster,
• meta-data services (MDSs) which manage the file system namespace,
• object storage services (OSDs) which actually store the data.

In particular, OSDs should be particularly numerous to obtain a large storage. Multiple MONs and MDSs allow to achieve scalability and reliability. Finally, data are replicated among OSDs for reliability.

A Ceph cluster can be scaled in 2 ways:

• increase the number of MONs,
• increase the number of OSDs.

scuttlemonkey: three really, because you can scale (or exclude, if you aren’t using CephFS) the MDSs independently. In this example ComodIT is deploying MON/MDS together though. Also keep in mind CephFS is still a little rough around the edges for a production deployment.

Requirements

1. A ComodIT account with a valid platform and distribution (you won’t be able to deploy this example on the demo platform provided by ComodIT).
2. the ComodIT Python library (bundled with command-line interface, see this tutorial for information about how to install it).

scuttlemonkey: it’s also worth noting here that while I deployed from an Ubuntu machine with minimal effort, the ComodIT guys default to CentOS 6.3 for most of their stuff (that’s the image you’ll need to use for your distribution later on as well) so it might be best to use that for testing.

Deployment

1. Clone the demos public repository and enter Ceph cluster’s folder:

    git clone https://github.com/comodit/demos.git
    cd demos/ceph-cluster

2. Create a config.py file with the following content:

   scuttlemonkey: I noticed that there was a config.py.sample here, so I just did a ‘cp config.py.sample config.py’ and edited the values required.

    endpoint = "https://my.comodit.com/api"
   
    username = "<user>"
    password = "<password>"
    organization = "<org_name>"
   
    time_out = 60 * 30  # seconds
   
    admin_key = "AQAEKwlRgBqsDhAA7cwN/JtEyCym6vYN/ixHqA=="
   
    platform = {"name" : "<plat_name>",
                "settings" : { ... }
               }
   
    distribution = {"name" : "<dist_name>",
                    "settings" : { ... }
                   }

   where <user> and <password> are your ComodIT credentials, <org_name> the name of your organization, <plat_name> the name of a platform in your organization and <dist_name> the name of a distribution in your organization. You should also fill the settings for both platform and distribution.

   scuttlemonkey: Platform and Distribution are things you define in your ComodIT web GUI. For the purposes of this demo I would suggest creating and using an ‘ec2′ paltform and just using ComodIT’s ‘Default Distribution’ (a CentOS image with the user data stuff already configured).

   For instance, you may use an Amazon EC2 platform and store’s CentOS 6.3 AMI. In this case, platform settings look like:

    "settings" : {
                 "ec2.instanceType": "t1.micro",
                 "ec2.securityGroups": "default",
                 "ec2.zone": "eu-west-1a",
                 "ec2.keyPair": "<key name>"
                 }

   scuttlemonkey: I chose to leave ec2.zone blank in this case as Amazon was pitching a fit about us-east-1a at the time I was testing this.

   where <key name> is a key pair name and distribution takes no setting:

    "settings" : {}

3. Setup your ComodIT account i.e. create all required applications and create an environment that will contain cluster’s hosts:

   ./setup.py

4. Actually deploy the cluster:

   ./deploy.py

   A simple Ceph cluster composed of 1 MON, 1 MDS and 2 OSDs hosted by 3 hosts is deployed: the MON and the MDS are hosted by the same host, the OSDs have their own host. Of course, this is not an architecture to use in production, you should always have several MONs. The complete deployment takes a few minutes on Amazon EC2.

scuttlemonkey: it’s worth noting here that if you have an error, or need to ^C out to fix or tweak something, you’ll want to run the ./teardown.py script to reset the stored variables on mon/osd number. If you don’t it may just sit there waiting for a machine to deploy that never will (this is an early prototype afterall).

5. Deployment script prints the public address of what we call the master node i.e. the computer hosting the monitor and MDS. You can connect to this host using SSH and check cluster’s health using the following command (executed as super-user or root):

   ceph -s

   See Ceph’s documentation for more details.

scuttlemonkey: In case you are used to using Ubuntu hosts like I am you’ll need to use ‘ec2-user’ and whatever key pair you specified at ComodIT setup for logging in to your CentOS box.

Scaling up (OSDs)

Add an OSD to deployed cluster:

./scale_osds.py -c 1

-c option is the number of OSDs to add.

scuttlemonkey: in my case, the ‘-c 1′ portion was causing the script to choke (perhaps some issue w/ arg parsing from Cent vs Ubuntu?). I didn’t really poke around to find out, the ./scale_osds.py script defaults to 1 so I just ran it without args and it worked fine.

Scaling up (MONs)

Add a monitor to deployed cluster:

./scale_mons.py -c 1

-c option is the number of monitors to add.

scuttlemonkey: it’s worth noting here that the best setup for a Ceph cluster is to have an odd number of mons for quorum. I mentioned this to the ComodIT guys and they will add a warning in upcoming versions of their deployment. For now it’s up to you to remember (although it will work just fine with an even number for the purposes of seeing it deploy).

Shutting down cluster

You can delete all hosts created during deployment and scaling operations:

./teardown.py

If you also want to clean-up your organization i.e. delete the applications and environment created by setup script:

./cleanup.py

scuttlemonkey: There you have it, another easy way to deploy a Ceph cluster for testing or prototyping. It’s always great to see folks fitting Ceph into their own workflows and environments. Hope to see lots more projects that showcase Ceph! Thanks to the ComodIT guys, especially Gérard (bottom-left below), for putting this demo together and sharing it with all of us.

scuttlemonkey out.
——————–

REPOSTED FROM: http://www.comodit.com/2013/02/12/effortless-deployment-of-a-ceph-cluster/

If any of you have interesting things that you have done with Ceph we always want to hear about it. Feel free to send a link to @Ceph or email it to our Community alias. Now, on to the goods!

Synnefo + RADOS = < 3

We are happy to announce that Synnefo now supports completely unified storage (Files/Images/VM disks) over commodity hardware using RADOS. And since it’s passed the testing phase, it is now heading to our production environment (~okeanos). And it scales!

But what does “completely unified storage” mean and why RADOS?

Let’s take things from the beginning.

Problem #1

Trying to scale a public cloud service over commodity hardware is not a trivial task. At first (mid 2011), we had all our VMs running over DRBD with Ganeti, and our File/Object Storage service (Pithos) backed by a big NAS. DRBD is great, production-quality software, enabling live migrations with no shared storage, and aggregate bandwidth that scales with the number of compute nodes. We wanted to keep all that. On the other hand, we knew that if we wanted the Storage service to scale, we had to get rid of the NAS eventually. We were also eager to explore new paths of having the same backing storage for VM disks and files.

An obvious choice would be to use a distributed filesystem running over commodity hardware, such as Lustre, GPFS or GlusterFS. Ganeti already supported VM disks over shared files, so the only overhead would be to write a shared file driver for our Storage service, which was trivial. However, we quickly realized that we didn’t really need filesystem semantics to store VM volumes, and we could certainly avoid the burden of having to take care of a distributed filesystem. Object/block semantics was what we were looking for.

So, we decided to test RADOS, since we had already been following the progress of Ceph from mid 2010. For the VM disks part, we implemented RBD support inside Ganeti and merged it into the official upstream. Starting with version 2.6, Ganeti supports instance disks on RADOS out of the box. It uses the RBD kernel driver and the RBD tools to do that. Furthermore, we implemented a RADOS backend driver for our Storage service. We chose not to go with RadosGW since we already had in production a storage frontend that implemented the OpenStack Object Storage API and also allowed for more advanced features such as deduplication via content hashing, file sharing among users etc.

By late 2011, we had it all up and running for testing. The architecture looked like this:

The above made two things possible: a) it enabled selection of the storage type for VMs, either RBD or DRBD, as an option for Synnefo end users, and b) it enabled Synnefo administrators to choose between a RADOS cluster or NFS / NAS as a backend for the Storage service. With this approach, we continued to do live migrations of VMs with no physically shared storage, this time over RBD to any physical node and not just DRBD’s secondary node. And we experimented with having RBD-based VM disks in the same RADOS storage cluster as files. So far, so good.

Problem #2

This seemed quite a success at the time, but still didn’t allow us to do magic. And by magic we mean the type of Image handling we were envisioning. We wanted to achieve three things at the same time:

• From the perspective of the Storage service, Images being treated as common files, with full support for remote syncing and sharing among users.
• From the perspective of the Compute service, Images cloned and snapshotted repetitively, with zero data copying from service to service.
• And finally, snapshots appearing as new Image files, again with zero data movement.

So, what could we do? We liked the architecture so far, with Synnefo, Ganeti and RADOS. RADOS seemed a good choice for consolidating storage of VM disks and files in a single place. We decided to design and implement a thin, distributed, custom storage layer completely independent from Synnefo, Ganeti or RADOS that could be plugged in the middle and do the job. If this worked, we could get to keep the goodies of all three base technologies, and work with well-defined abstractions among them. And that’s what we did.

By mid 2012 we had the prototype ready. We called it Archipelago. Then, we needed to integrate it with the rest of the infrastructure and start stress-testing. The integration happened in two directions: with Ganeti, on one side, and with RADOS, on the other.

To integrate with Ganeti, we stopped using Ganeti’s native RBD support to talk to RADOS, since we now had Archipelago in between. We exploited Ganeti’s ExtStorage Interface and wrote the corresponding Archipelago driver (ExtStorage provider) for Ganeti.

To integrate with RADOS, we implemented a RADOS driver for Archipelago. Finally, by late 2012 we had the code completed and have been testing it ever since. The architecture now looks like this:

After 3 months of stress testing, we are now in the process of moving everything into ~okeanos production, which is running more than 2700 active VMs for more than 1900 users, at the time of this writing.

For the impatient, the new feature is already up and running on our free trial demo infrastructure at:

http://www.okeanos.io

So, create a free trial account and choose the “Archipelago” disk option when creating your new VM. You will see it coming up in seconds, thinly! Backed by RADOS

Enjoy!

—————-

REPOSTED FROM: http://www.synnefo-software.blogspot.com/2013/02/we-are-happy-to-announce-that-synnefo_11.html