Live Demo : OpenStack and Ceph

openstack and ceph live demonstration

Hey Guys , with this blog post i would like to share live demonstration screen cast that i have prepared using my openstack and Ceph environment.

What’s expected from this Demo

  • Creating OpenStack glance image stored on Ceph RBD
  • Creating OpenStack instance with Ephermal disk on Ceph RBD ( storing instance on Ceph )
  • Creating OpenStack Cinder volume on Ceph RBD
    • Attaching this volume to OpenStack instance
    • Mounting this volume and generating some load
  • OpenStack Instance LIVE MIGRATION across hypervisors

Environment Details

  • OpenStack RDO ( Kilo Release )
    • 2 Nodes deployment
      • Node1 – Controller + Network + Compute
      • Node2 – Compute
  • Ceph ( Fujitsu Eternus CD10000 Ceph storage appliance )
    • Release : Firefly ( 0.80.7 )
    • 5 node cluster
      • Node 1 : Dedicated management node ( ceph admin node )
      • Node 2 , 3 , 4 : Ceph monitor + OSD nodes
      • Node 5 : Ceph OSD node
  • OpenStack Glance, Cinder & Nova are configured to use Ceph as a storage backend


openstack and ceph live demonstration

Demo Time !!!

Commands used during demo

## Task 1 : Creating OpenStack glance image and making sure its stored on Ceph RBD

openstack image list
glance image-create --name='Fedora-22' --is-public='true' --disk-format='raw' --container-format='bare' < Fedora-Cloud-Base-22-20150521.x86_64.raw
openstack image list
rbd ls -p glance --id glance

## Task 2 : Creating OpenStack instance with Ephermal disk on Ceph RBD

openstack network list
openstack image list
openstack flavor list
openstack keypair list
openstack server list
openstack server create --image <glance_image_id> --flavor 2 --key-name <keypair_name> --nic net-id=<network_id> <Instance_name>
nova floating-ip-pool-list
nova add-floating-ip <instance_id> <floating_ip_address>
nova list
ssh fedora@<floating_ip_address>

## Task 3 : Creating OpenStack Cinder volume on Ceph RBD

openstack volume create --size 10 --description "volume description" --type "ceph_storage" <volume_name>
openstack volume list
rbd ls -p cinder --id cinder
nova volume-attach  <instance_id> <cinder_volume_id> auto
openstack volume list
ssh fedora@<floating_ip_address>
fdisk -l
mkfs.ext4 /dev/vdb
mount /dev/vdb /mnt
df -h

## Task 4 : OpenStack Instance LIVE MIGRATION across hypervisors

nova list
nova hypervisor-servers <hypervisor_1_name>
nova hypervisor-servers <hypervisor_2_name>
virsh list
ssh node2 virsh list
tail -f /var/log/nova-compute.log

## On openstack instance generate load
dd if=/dev/zero of=file1 bs=1M count=9000

## From hypervisor initiate live migration

nova live-migration <instance_id> <new_hypervisor_name>

## Once migration completes, verify instance new location

nova list
nova hypervisor-servers <hypervisor_1_name>
nova hypervisor-servers <hypervisor_2_name>
virsh list
ssh node2 virsh list

Live migration support for OpenStack instances is one of the showstopper feature of Ceph. That’s why Ceph is the most popular storage backend for OpenStack.

In a few hours the voting period for the next OpenStack Summit in Tokyo ends at 30. July, 11:59 PDT (31. July  6:59 UTC / 08:59 CEST).
Here a some interesting talks regarding Ceph you may would like to vote for (this list is not intended to be exhaustive):

Sorting Ceph backport branches

When there are many backports in flight, they are more likely to overlap and conflict with each other. When a conflict can be trivially resolved because it comes from the context of a hunk, it’s often enough to just swap the two commits to avoid the conflict entirely. For instance let say a commit on

void foo() { }
void bar() {}

adds an argument to the foo function:

void foo(int a) { }
void bar() {}

and the second commit adds an argument to the bar function:

void foo(int a) { }
void bar(bool b) {}

If the second commit is backported before the first, it will conflict because it will find that the context of the bar function has the foo function without an argument.

When there are dozens of backport branches, they can be sorted so that the first to merge is the one that cherry picks the oldest ancestor in the master branch. In other words given the example above, a cherry-pick of the first commit be merged before the second commit because it is older in the commit history.

Sorting the branches also gracefully handles interdependent backports. For instance let say the first branch contains a few backported commits and a second branch contains a backported commit that can’t be applied unless the first branch is merged. Since it is required for each Ceph branch proposed for backports to pass make check, the most commonly used strategy is to include all the commits from the first branch in the second branch. This second branch is not intended to be merged and the title is usually prefixed with DNM (Do Not Merge). When the first branch is merged, the second is rebased against the target and the redundant commits disapear from the second branch.

Here is a three lines shell script that implements the sorting:

# Make a file with the hash of all commits found in master
# but discard those that already are in the hammer release.
git log --no-merges \
  --pretty='%H' ceph/hammer..ceph/master \
  > /tmp/master-commits
# Match each pull request with the commit from which it was
# cherry-picked. Just use the first commit: we expect the other to be
# immediate ancestors. If that's not the case we don't know how to
# use that information so we just ignore it.
for pr in $PRS ; do
  git log -1 --pretty=%b ceph/pull/$pr/merge^1..ceph/pull/$pr/merge^2 | \
   perl -ne 'print "$1 '$pr'\n" if(/cherry picked from commit (\w+)/)'
done > /tmp/pr-and-first-commit
# For each pull request, grep the cherry-picked commit and display its
# line number. Sort the result in reverse order to get the pull
# request sorted in the same way the cherry-picked commits are found
# in the master history.
SORTED_PRS=$(while read commit pr ; do
  grep --line-number $commit < /tmp/master-commits | \
  sed -e "s/\$/ $pr/" ; done  < /tmp/pr-and-first-commit | \
  sort -rn | \
  perl -p -e 's/.* (.*)\n/$1 /')

If an OpenStack tenant (account in the OpenStack parlance) is available, the Ceph integration tests can be run with the teuthology-openstack command , which will create the necessary virtual machines automatically (see the detailed instructions to get started). To do its work, it uses the teuthology OpenStack backend behind the scenes so the user does not need to know about it.
The teuthology-openstack command has the same options as teuthology-suite and can be run as follows:

$ teuthology-openstack \
  --simultaneous-jobs 70 --key-name myself \
  --subset 10/18 --suite rados \
  --suite-branch next --ceph next
Scheduling rados/thrash/{0-size-min-size-overrides/...
Suite rados in suites/rados scheduled 248 jobs.

web interface:
ssh access   : ssh ubuntu@ # logs in /usr/share/nginx/html

As the suite progresses, its status can be monitored by visiting the web interface::

And the horizon OpenStack dashboard shows resource usage for the run:

If something goes wrong, the easiest way to free all resources is to run

ssh ubuntu@ sudo /etc/init.d/teuthology restart

where the IP address is the one listed as a reminder (“ssh access: …”) in the output of each teuthology-openstack command (see example above)..
When the run terminates, the virtual machine hosting the web interface and hosting the test results is not destroyed (that would be inconvenient for forensic analysis). Instead, it will be re-used by the next teuthology-openstack run.
When the cluster is no longer needed (and the results have been analyzed) it can be destroyed entirely with

teuthology-openstack --teardown

Special thanks to Zack Cerza, Andrew Schoen, Nathan Cutler and Kefu Chai for testing, patching, advising, proofreading and moral support over the past two months ;-)

As mentioned before: then next OpenStack Summit will take place in Tokyo, Japan from 27-30 October 2015. The “Call for Speakers” ended some days ago and now the vote for presentation period started and will end 30. July, 11:59 PDT (31. July  6:59 UTC / 08:59 CEST). 
I’ve submitted for Toky two talks:

  • 99.999% available OpenStack Cloud – A builder’s guide” – This talk will provide insight in how to build a highly available OpenStack cloud together with Ceph in a classical highly available data centers with e.g. two fire compartments and redundant infrastructure. The focus will be on failure scenarios and their impact on OpenStack and Ceph components, quorum devices you have to consider, strategies on how to ensure quorums and keep the cloud running, pitfalls and solutions.
  • NFV orchestration powered by OpenStack” – This talk would be about existing OpenStack orchestration projects and how they can be used to manage a NFV Cloud, how they integrate, what is missing, and how to map these projects on the ETSI-NFV model.
You can vote, if you are interested to see my talks [1][2] at the summit, every vote is highly welcome. You can find a full abstract at the voting page.
My colleague Ralf Trezeciak submitted also a NFV related talk. He will appreciate your votes for sure, if you are interested in the presentation.
There is a long list of interesting Ceph related talks, simply use the search field on the voting page. I may provide a list in an additional post later.

In the context of the teuthology (the integration test framework for Ceph, there needs to be a PostgreSQL available, locally only, with a single user dedicated to teuthology. It can be setup from a new Ubuntu 14.04 install with:

    sudo apt-get -qq install -y postgresql postgresql-contrib 

    if ! sudo /etc/init.d/postgresql status ; then
        sudo mkdir -p /etc/postgresql
        sudo chown postgres /etc/postgresql
        sudo -u postgres pg_createcluster 9.3 paddles
        sudo /etc/init.d/postgresql start
    if ! psql --command 'select 1' \
          'postgresql://paddles:paddles@localhost/paddles' > /dev/null
        sudo -u postgres psql \
            -c "CREATE USER paddles with PASSWORD 'paddles';"
        sudo -u postgres createdb -O paddles paddles

If anyone knows of a simpler way to do the same thing, I’d be very interested to know about it.

v0.80.10 Firefly released

This is a bugfix release for Firefly.

We recommend that all Firefly users upgrade.

For more detailed information, see the complete changelog.


  • build/ops: package mkcephfs on EL6 (issue#11955pr#4924, Ken Dreyer)
  • build/ops: debian: ceph-test and rest-bench debug packages should require their respective binary packages (issue#11673pr#4766, Ken Dreyer)
  • build/ops: run RGW as root (issue#11453pr#4638, Ken Dreyer)
  • common: messages/MWatchNotify: include an error code in the message (issue#9193pr#3944, Sage Weil)
  • common: Rados.shutdown() dies with Illegal instruction (core dumped) (issue#10153pr#3963, Federico Simoncelli)
  • common: SimpleMessenger: allow RESETSESSION whenever we forget an endpoint (issue#10080pr#3915, Greg Farnum)
  • common: WorkQueue: make wait timeout on empty queue configurable (issue#10817pr#3941, Samuel Just)
  • crush: set_choose_tries = 100 for erasure code rulesets (issue#10353pr#3824, Loic Dachary)
  • doc: backport ceph-disk man page to Firefly (issue#10724pr#3936, Nilamdyuti Goswami)
  • doc: Fix ceph command manpage to match ceph -h (issue#10676pr#3996, David Zafman)
  • fs: mount.ceph: avoid spurious error message (issue#10351pr#3927, Yan, Zheng)
  • librados: Fix memory leak in python rados bindings (issue#10723pr#3935, Josh Durgin)
  • librados: fix resources leakage in RadosClient::connect() (issue#10425pr#3828, Radoslaw Zarzynski)
  • librados: Translate operation flags from C APIs (issue#10497pr#3930, Matt Richards)
  • librbd: acquire cache_lock before refreshing parent (issue#5488pr#4206, Jason Dillaman)
  • librbd: snap_remove should ignore -ENOENT errors (issue#11113pr#4245, Jason Dillaman)
  • mds: fix assertion caused by system clock backwards (issue#11053pr#3970, Yan, Zheng)
  • mon: ignore osd failures from before up_from (issue#10762pr#3937, Sage Weil)
  • mon: MonCap: take EntityName instead when expanding profiles (issue#10844pr#3942, Joao Eduardo Luis)
  • mon: Monitor: fix timecheck rounds period (issue#10546pr#3932, Joao Eduardo Luis)
  • mon: OSDMonitor: do not trust small values in osd epoch cache (issue#10787pr#3823, Sage Weil)
  • mon: OSDMonitor: fallback to json-pretty in case of invalid formatter (issue#9538pr#4475, Loic Dachary)
  • mon: PGMonitor: several stats output error fixes (issue#10257pr#3826, Joao Eduardo Luis)
  • objecter: fix map skipping (issue#9986pr#3952, Ding Dinghua)
  • osd: cache tiering: fix the atime logic of the eviction (issue#9915pr#3949, Zhiqiang Wang)
  • osd: cancel_pull: requeue waiters (issue#11244pr#4415, Samuel Just)
  • osd: check that source OSD is valid for MOSDRepScrub (issue#9555pr#3947, Sage Weil)
  • osd: DBObjectMap: lock header_lock on sync() (issue#9891pr#3948, Samuel Just)
  • osd: do not ignore deleted pgs on startup (issue#10617pr#3933, Sage Weil)
  • osd: ENOENT on clone (issue#11199pr#4385, Samuel Just)
  • osd: erasure-code-profile set races with erasure-code-profile rm (issue#11144pr#4383, Loic Dachary)
  • osd: FAILED assert(soid < scrubber.start || soid >= scrubber.end) (issue#11156pr#4185, Samuel Just)
  • osd: FileJournal: fix journalq population in do_read_entry() (issue#6003pr#3960, Samuel Just)
  • osd: fix negative degraded objects during backfilling (issue#7737pr#4021, Guang Yang)
  • osd: get the currently atime of the object in cache pool for eviction (issue#9985pr#3950, Sage Weil)
  • osd: load_pgs: we need to handle the case where an upgrade from earlier versions which ignored non-existent pgs resurrects a pg with a prehistoric osdmap (issue#11429pr#4556, Samuel Just)
  • osd: ObjectStore: Don’t use largest_data_off to calc data_align. (issue#10014pr#3954, Jianpeng Ma)
  • osd: osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o… (issue#10259pr#3827, Samuel Just)
  • osd: PG::actingset should be used when checking the number of acting OSDs for… (issue#11454pr#4453, Guang Yang)
  • osd: PG::all_unfound_are_queried_or_lost for non-existent osds (issue#10976pr#4416, Mykola Golub)
  • osd: PG: always clear_primary_state (issue#10059pr#3955, Samuel Just)
  • osd: PGLog.h: 279: FAILED assert(log.log.size() == log_keys_debug.size()) (issue#10718pr#4382, Samuel Just)
  • osd: PGLog: include rollback_info_trimmed_to in (read|write)_log (issue#10157pr#3964, Samuel Just)
  • osd: pg stuck stale after create with activation delay (issue#11197pr#4384, Samuel Just)
  • osd: ReplicatedPG: fail a non-blocking flush if the object is being scrubbed (issue#8011pr#3943, Samuel Just)
  • osd: ReplicatedPG::on_change: clean up callbacks_for_degraded_object (issue#8753pr#3940, Samuel Just)
  • osd: ReplicatedPG::scan_range: an object can disappear between the list and t… (issue#10150pr#3962, Samuel Just)
  • osd: requeue blocked op before flush it was blocked on (issue#10512pr#3931, Sage Weil)
  • rgw: check for timestamp for s3 keystone auth (issue#10062pr#3958, Abhishek Lekshmanan)
  • rgw: civetweb should use unique request id (issue#11720pr#4780, Orit Wasserman)
  • rgw: don’t allow negative / invalid content length (issue#11890pr#4829, Yehuda Sadeh)
  • rgw: fail s3 POST auth if keystone not configured (issue#10698pr#3966, Yehuda Sadeh)
  • rgw: flush xml header on get acl request (issue#10106pr#3961, Yehuda Sadeh)
  • rgw: generate new tag for object when setting object attrs (issue#11256pr#4571, Yehuda Sadeh)
  • rgw: generate the “Date” HTTP header for civetweb. (issue#11871,11891pr#4851, Radoslaw Zarzynski)
  • rgw: keystone token cache does not work correctly (issue#11125pr#4414, Yehuda Sadeh)
  • rgw: merge manifests correctly when there’s prefix override (issue#11622pr#4697, Yehuda Sadeh)
  • rgw: send appropriate op to cancel bucket index pending operation (issue#10770pr#3938, Yehuda Sadeh)
  • rgw: shouldn’t need to disable rgw_socket_path if frontend is configured (issue#11160pr#4275, Yehuda Sadeh)
  • rgw: Swift API. Dump container’s custom metadata. (issue#10665pr#3934, Dmytro Iurchenko)
  • rgw: Swift API. Support for X-Remove-Container-Meta-{key} header. (issue#10475pr#3929, Dmytro Iurchenko)
  • rgw: use correct objv_tracker for bucket instance (issue#11416pr#4379, Yehuda Sadeh)
  • tests: force checkout of submodules (issue#11157pr#4079, Loic Dachary)
  • tools: Backport ceph-objectstore-tool changes to firefly (issue#12327pr#3866, David Zafman)
  • tools: ceph-objectstore-tool: Output only unsupported features when incomatible (issue#11176pr#4126, David Zafman)
  • tools: ceph-objectstore-tool: Use exit status 11 for incompatible import attemp… (issue#11139pr#4129, David Zafman)
  • tools: Fix so that -L is allowed (issue#11303pr#4247, Alfredo Deza)


Un-installing OpenStack RDO

Uninstalling OpenStack

RedHat Distribution of OpenStack ( RDO ) is one of the most popular openstack distribution for test, dev and production environments. Deploying OpenStack infrastructure using RDO is fairly easy.

However there is no automated uninstall process for RDO (or OpenStack in general).

With the help of this blog I will share my secret sauce that I personally use to uninstall / remove openstack from my development & test environments. This saves me from reinstalling the entire OS and over the OS configurations + it saves time.

Uninstalling RDO

  • Take necessary backup of your configuration files ( if you like )
  • Its advisable to re-read this script before execution ( no blame game )
  • Be extra cautious for production environments.

Get the script and make it executable

git clone
cd 7a9f1f62c06318a17dad
chmod +x

Link to script


  • Read the script; change it if you want to
  • Execute this script on all of your openstack nodes, whether its compute, controller or network node (one by one)
  • OpenStack RDO should have removed at this point and you are good to re-install it.

YAY…. That was easy 😉

© 2015, Red Hat, Inc. All rights reserved.